Spring, ESBs, and File Suckers

In the enterprise, it's unusual that any application stands alone. Quite often, two or more applications must collaborate, sharing data between each other. A workforce scheduling application may need to share data with the payroll system. Likewise, the payroll system may need to share information with the financials system. And they all may need to interact with the HR application.

There are several tricks, schemes, and hacks to share data between systems. Most of these tricks are one-off integration solutions that are only useful to tie two specific systems together. When you need to throw another system into the mix, you'll probably need to start over and come up with another integration scheme.

A primitive, yet very common integration solution is file-sharing (known as the File Transfer pattern as described in Enterprise Integration Patterns). Using file-sharing a payroll system may write information to a file to be picked up and processed later by the financials system. Often, file-sharing involves some form of scheduling (e.g., cron) to trigger the receiving application to pick up the data file.

The problem with scheduled file-sharing is that the sending application has a finite window of opportunity to write the file before the receiving application comes along and picks it up. In the best case, the sending application finishes writing the file well in advance of the schedule (resulting in wasted time that the receiving application could use to process the data). In the worst case, the receiving application wakes up before the sending application has completed its work (resulting in missing or incomplete data).

In a perfect world, the financials system would pick up the file at precisely the moment that the payroll system has finished writing it. No sooner...no later. The file will always be complete and never have to sit idle waiting for financials to pick it up.

This sounds like a job for a File Sucker.

ESBs as File Suckers

"File sucker" is the name that I use to describe any mechanism that watches a directory for new files. Upon seeing a new file, it immediately sucks the file in and passes it on to any other applications that need it.

File sucking is hardly a new idea. In fact, it is a common feature in most Enterprise Service Bus (ESB) implementations. Although ESBs provide a rich array of integration options that are more robust than file sucking, most ESBs still provide support for monitoring directories and reading files into the bus. Support for file sucking in ESBs enable incremental adoption of the ESB into environments where file-sharing is already a common integration mechanism.

One such ESB that supports file sucking is Mule ESB. In Mule, creating a file sucker involves configuring a component whose inbound endpoint is a file directory. When a file appears in the watched directory it triggers an event in Mule, prompting Mule to read the file into the bus.

There are several ways to setup a file sucker in Mule. But since I'm a Spring fanatic and this is a Spring-centric blog, let me show you a very easy way to use Mule and Spring together to create a file sucker.

What you'll need

Here are the JAR files that I'm using to write the file-sucker example:

• spring.jar (version 1.2.3)
• commons-logging.jar
• mule-1.1-rc1.jar
• mule-extras-spring-1.1-rc1.jar
• geronimo-spec-j2ee-connector-1.5-rc4.jar
• jug-1.0.3.jar
• concurrent-1.3.4.jar
• commons-discovery.jar
• commons-beanutils.jar
• commons-collections.jar
• xstream-1.0.2.jar
• commons-pool.jar

Most of these files (or their equivalents) come with the Mule download. But you'll need to provide your own spring.jar file. For your convenience, I've collected all of the required JAR files and have uploaded them along with my example code for you to download here.

Configuring a multicaster

Mule comes with (in Mule extras) MuleEventMulticaster. This clever class subscribes to one or more service endpoints and, when an event arrives on that endpoint, it multicasts it out to any bean who is interested.

For the purposes of our file sucker example, we'll configure the MuleEventMulticaster as follows:

<beans>
  <bean id="applicationEventMulticaster"
      class="org.mule.extras.spring.events.MuleEventMulticaster">
    <property name="subscriptions">
      <list>
        <value>file:///inbox</value>
      </list> 
    </property>
  </bean>
</beans>

As you may have guessed, this particular MuleEventMulticaster keeps an eye on the "/inbox" directory, pulling in any files it finds there. But what does it do with the files that it finds?

To try this out, let's write a simple class with a main() method to load this bean into a Spring container. Perhaps something like this:

public class FileSucker {
  public static void main(String[] args) {
    new FileSystemXmlApplicationContext("sucker.xml");
  }
}

After running this code, copy some files into the inbox directory. If you take a look at the directory, you'll notice the files mysteriously disappearing almost as soon as they arrive. This proves that Mule is pulling them in. But where do they go?

At this point, the MuleEventMulticaster is sucking in the files, but since there aren't any event listeners, there's nobody to receive them. They're all sucked up with nowhere to go. Let's give them some place to go by adding an event listener.

Building an event listener

Mule's documentation implies that any implementation of Spring's ApplicationListener interface will receive Mule events. But as I've found in my own experiments (and as evidenced in the source code for MuleEventMulticaster) your listener classes must implement MuleEventListener. The good news is that MuleEventListener extends Spring's ApplicationListener and has no additional methods beyond onApplicationEvent()...so creating a Mule listener is not much different than creating any other Spring ApplicationListener implementation.

Here's a simple implementation of MuleEventListener:

package com.habuma.sucker.mule;

import org.mule.extras.spring.events.MuleEventListener;
import org.springframework.context.ApplicationEvent;

public class EventListener implements MuleEventListener {
  public void onApplicationEvent(ApplicationEvent event) {
      System.out.println("GOT AN EVENT: ");
      System.out.println(event.getSource());
  }
}

In the "real world", this event listener would probably be more exciting, reacting to the event by performing some sort of business logic. For simplicity's sake, this event listener simply sends the contents of its event to stdout.

Aside from implementing MuleEventListener, this class is no different than any other Spring ApplicationListener. But by implementing MuleEventListener you're indicating that you want this class to receive Mule events.

Now let's configure EventListener as a bean in the Spring context...

<beans>
...
  <bean id="myListener"
      class="com.habuma.sucker.mule.EventListener" />
...
</beans>

To see the event listener in action, startup the application and copy some files into inbox. You should see the EventListener react to every file that is copied into inbox.

There's only one gotcha: MuleEventMulticaster will send the file events to all MuleEventListeners. What if your application has several event listeners and you want each to list for specific events. For example, what if you have two instances of EventListener, one to receive files from "inbox" and another to receive files from "otherbox"? As configured above, both instances of EventListener will receive all events from MuleEventMulticaster regardless of where they came from. This means that both EventListener instances would receive files from both directories.

To remedy that situation, you'll need to implement MuleSubscriptionEventListener instead of MuleEventListener:

package com.habuma.sucker.mule;

import org.mule.extras.spring.events.MuleSubscriptionEventListener;
import org.springframework.context.ApplicationEvent;

public class SubscriptionListener implements MuleSubscriptionEventListener {
 
  public void onApplicationEvent(ApplicationEvent event) {
      System.out.println("GOT AN EVENT ("+this.hashCode()+") : ");
      System.out.println(event.getSource());
  }
  
  private String[] subs;
  
  public void setSubscriptions(String[] arg0) {
    this.subs = arg0;
    
  }

  public String[] getSubscriptions() {
    return subs;
  }
}

MuleSubscriptionEventListener requires that you implement a getSubscriptions() method in addition to onApplicationEvent(). getSubscriptions() should return an array of String that contains all of the endpoints that the listener is interested in. Although it's not required by MuleSubscriptionEventListener, I've also added a setSubscriptions() method so that I can use Spring's IoC to wire in my endpoint definitions.

One more thing you'll need to do is reconfigure your application context such that each SubscriptionListener specifies their own list of endpoints:

<beans>
  <bean id="applicationEventMulticaster"
      class="org.mule.extras.spring.events.MuleEventMulticaster"/>

  <bean id="myListener"
      class="com.habuma.sucker.mule.SubscriptionListener">
    <property name="subscriptions">
      <list>
        <value>file:///inbox</value>
      </list> 
    </property>
  </bean>

  <bean id="myOtherListener"
      class="com.habuma.sucker.mule.SubscriptionListener">
    <property name="subscriptions">
      <list>
        <value>file:///otherbox</value>
      </list> 
    </property>
  </bean>
</beans>

Here we have two instances of SubscriptionListener, each with their own selection of endpoint subscriptions. Also, notice that MuleEventMulticaster no longer keeps its own list of subscriptions.

Now if you fire up the application, you'll be able to copy files into either "inbox" or "otherbox"...each instance of SubscriptionListener receiving its own set of files.

So what?

As I've said, a real world application would likely do something more interesting than simply print out the contents of the files that are sucked in. It may simply write it to a database or process it and then fire the results out as another event in the ESB. As an exercise for the reader, try to imagine the possibilities of how MuleEventMulticaster can be used in your real world applications.

Before you get too carried away, you should know that there's really more here than meets the eye. Although I've been talking about creating a file sucker, the only thing that makes this a file sucker is that the listeners are subscribed to a "file://" endpoint. The event listener classes themselves have no idea that their content is coming from a file. What if instead of reacting to files being written to a directory, you'd like to respond to an e-mail, a JMS message, or perhaps a message sent via TCP?

No problem...just swap out the "file://" endpoint with any one of Mule's other endpoints. For example, to listen on TCP, port 9999:

  <bean id="myListener"
      class="com.habuma.sucker.mule.SubscriptionListener">
    <property name="subscriptions">
      <list>
        <value>tcp://localhost:9999</value>
      </list> 
    </property>
  </bean>

Or maybe you want to receive messages from a JMS topic or queue...

  <bean id="myListener"
      class="com.habuma.sucker.mule.SubscriptionListener">
    <property name="subscriptions">
      <list>
        <value>jms://my.queue</value>
      </list> 
    </property>
  </bean>

  <!--
     To use JMS you need some additional setup
  -->
  <bean id="jmsConnector" class="org.mule.providers.jms.JmsConnector">
    <property name="specification">
        <value>1.1</value>
    </property>
    <property name="connectionFactory" ref="connectionFactory" />
  </bean>

  <bean id="connectionFactory"
      class="org.activemq.ActiveMQConnectionFactory">
    <property name="brokerURL" value="tcp://localhost:61616" />
  </bean>    

But why choose? Notice that the "subscriptions" property is an array. If you want, you can use listen on several endpoints at the same time:

  <bean id="myListener"
      class="com.habuma.sucker.mule.SubscriptionListener">
    <property name="subscriptions">
      <list>
        <value>file:///inbox</value>
        <value>tcp://localhost:9999</value>
        <value>jms://my.queue</value>
      </list> 
    </property>
  </bean>

Files, TCP, and JMS are just the beginning. Mule has several providers for a variety of different endpoints, including:

• AS400 DataQueue
• E-mail
• File
• FTP
• HTTP
• JDBC
• JMS
• Quartz (for scheduled events)
• RMI
• Servlets
• SOAP (via Axis or Glue)
• SSL
• TCP
• UDP
• VM (in-process or persistent queues)
• XMPP (Jabber)

Use your imagination...Instead of a file sucker, you can use this same technique to have a bean listen for e-mails to arrive at a certain POP3 address. Or maybe you'd like a bean that processes XML messages that arrive via HTTP.

This article has been very Mule-specific. ServiceMix is another up-and-coming ESB in CodeHaus that I've been tinkering with. In some future installment, I'll show you how to do have some similar fun using Spring and ServiceMix. Stay tuned.

There's much more to ESBs and Mule than simply sucking in files and multicasting them to interested listeners. But that's another discussion for another day. In the meantime, if you want to learn more about ESBs, take a look at both Mule and ServiceMix and start tinkering. I also recommend that you check out David Chappell's Enterprise Service Bus book. This book cleared up a lot of ESB concepts for me.

Another book I've enjoyed on application integration in general is Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf. Mule's architecture is based heavily on the patterns described in this book.

I'll be talking more about using Spring with Mule and ServiceMix next weekend at the LoneStar Software Symposium in Austin (and again later this year in Dallas) and at The Spring Experience in December. Hope to see you there!