January 6, 2014

The demise of the ESB in a world of webservices

Abstract: In a world where no single enterprise can develop the software needed to run the enterprise, it no longer makes sense to connect everything to the Enterprise Service Bus, but instead everything should be connected to the internet. This is exactly what is happening. And this trend is causing the dimise of the ESB as there is no validity to the ESB being the backbone of Enterprise Application Integration, something that was promised by the major players in the world of messaging. Promises never truly delivered on.

Remember what the ESB was good for?
  1. Data Exchange
  2. Data Transformation
  3. Data Routing
A role that has been assumed by web-services, both in the form of SOAP based web-services (predominantly conforming to the WS* standards) and in the form of REST-based web-services.

Other posts in this trilogie:
ESB's don't scale, which shouldn't be a problem because that is by design
The ESB addresses a niche, not mainstream

In this third post in a series of three on the ESB of Arc-E-Tect I will delve deeper in the inevitable dimise of the ESB in today's world of connected systems.

Let's face it, the ESB isn't dead, it was never alive in the first place. Not in the incarnation the big vendors would like us to believe. It was and is merely a renamed messaging system.

That being said we can concentrate on whether or not we should still consider the ESB. The Enterprise Service Bus.

This is a post in a series about the Enterprise Service Bus, ESB, an acronym so easily to love or to hate. Parts of this post are also found in another post on the dimise of the ESB in a world of webservices.

First of all, let's not call it Enterprise Service Bus, but stick with ESB. Just like SOAP, which originally was the Simple Object Access Protocol and now just is SOAP. Probably because there was nothing Simple with SOAP. Just like there's nothing Enterprise about ESB. Unless of course you consider the budgets needed to actually get something running.

So, the ESB became obsolete. First of all, we should identify what the ESB was supposed to deliver before we can say it became obsolete.

Basically there are just a few functionalities an ESB actually was supposed to provide.
  1. Data exchange. Now I'm carefully choosing my words here and keeping it as abstract as can be. But in fact the ESB's data exchange capabilities didn't extend much further as messaging. Messaging in the sense of what products like IBM's MQ Series and Microsoft's ActiveMQ provide. You have a limited amount of data, and you send it between two points.
  2. Data transformation. Actually this should be message transformation, but it applies to all data. The ESB is particularly good at transforming one data format to another. This is actually some legacy capability from the time there was no ESB and only MOM (Message Oriented Middleware). Because the promise of being able to loosely couple two systems by utilizing MOM justified the creation of functionality that would transform one data format to another. This to further decouple both points.
  3. Data routing. Again, the premise here was to decouple sender and receiver. As the ESB already decouples sender and receiver by implementing an asynchronous communication protocol, by allowing for transformation from the sender's format to the receivers format, the only coupling aspect still to be resolved is the addressing. The ESB removes the necessity of the sender knowing where the receiver is located by allowing for logical names that are resolved and data is routed to the 'correct' recipient.
In 1997 I started working on an ESB using IBM MQ Series as the underpinning messaging technology, we implemented all three core capabilities of the ESB. The likes of Microsoft, IBM, Lotus and Neon came to visit us as they wanted to see what we were doing. This all at a one of the Dutch leading banks.
As we implemented the usage of the ESB and later on as the Middleware architect and Enterprise Application Integration Architect (EAI) for this customer I found that the premise of the ESB, which at the time was called MOM (Message Oriented Middleware) was awesome.
We connected a front-end developed in Visual Basic with a backend developed in COBOL. On the client side, we developed an ActiveX control in C++ and a whole series of modules that allowed for on the fly creation of message formats, stored in a meta-data format that was retrieved from a meta-data server. The initial format for the messages was actually the format the meta-data server understood and we called it meta-meta data. Pretty deep, huh :)
On the server side, a COBOL program running on the AS/400 we developed some components using C as the programming language of choice. Here as well we developed something similar to the meta-data server. As I recall, we were using copy-books.
It was all pretty advanced stuff and we were able to decouple the server and the client, develop both independently from each other and the queueing provided by MQ Series allowed also for asynchronous communication.
Shortly thereafter products came on the market that did a lot of the functionality we implemented and on the part of MQ Series, NEON was a marketleader.

This is the time around which SOA (Service Oriented Archtecture) started its emergence and everybody and their mother who thought of themselves as a messaging vendor jumped on this bandwagon. With not much more than the good old MOM with a new label attached to it.

Interestingly enough, the messaging format problems was solved by the introduction of XML. XML already existed but as it was both text oriented and self describing and on top of that, like a cherry, it was human readable... more or less. The problem of course still being that the actual message has to ad here to a defined format. DTD's (Document Type Definition) and later XSD's (eXensible Schema Definition) were invented to ensure this and both standards are doing a pretty good job at it. When combining this with XSLT (eXtensible Stylesheet Language Transformation) you have a format that is self describing, a means to define the format and a means to transform from this format to pretty much every other format.
Don't get me wrong, I really love the fact that with XML I can do all this stuff. More importantly, I don't need a message tranformation engine to do so. Of course, this awesomeness only works when you start off with XML.

From a decoupling perspective this is understandably pretty neat because it allows the sender of the message to determine the message format and the recipient can parse the message in a standard way, verify that it complies to the definition and because of that transform it to a format it can understand. Granted, when the sender doesn't use XML you're stuck with being able to handle that format. And in those cases a message transformation engine comes in pretty handy.
The question is what happens when a lot of senders send messages in XML but not the format the recipient understands. Who will do the transformation to the right format? You could do this at the recipient end, but that is bad practice. Why? Because you're coupling sender and receiver again. That was something we were tryint to avoid. Doing this in a message transformation engine makes perfect sense and is the right way to handle this.

So XML is excellent for messaging. Messaging is awesome for asynchronous communciation. And it is great for communications where the sequence of messages is important. Many communications are not asynchronous at all. They're request/reply style interactions. But interestingly enough, these typically don't require ordering... and when the communication si asynchronous, we typically notice that ordering is important.
First about the request/reply not being ordered. What you should realize is that in these cases the client, the requestor takes care of only proceeding with the processing once the reply is received. So the handling from a server (recipient) of multiple requests is regarind multiple requests of multiple clients. These are independent and therefore by nature unordered.
The communications that are asynchronous but need to be ordered are usually feeds of data that need to be processed in sequence. Data feeds from a single client are sequence by using a... yes... a queue. It's the perfect mechanism for sequencing. When dealing with multiple clients feeding the server, we need to resolve to unique sequence numbers that are universally unique or using timestamps... which is always tricky.

I'm probably not the first to notice this as SOAP and WS* standards are addressing this exactly. By this I'm meaning the benefits of XML and the characteristics of communications between systems.
The XML part of SOAP based webservices is that the XML is very well defined. Arguably that is, but from the perspective of the freeformat XML actually is, it is darn well defined. Also, the webservice is a request/reply protocol. In case you want to use it for feeds you just ignore the reply.
There's another nice feature of SOAP and that is that it can use various communication protocols like HTTP(s), JMS (Java Message Service), SMTP (Simple Mail Transfer Protocol, or email) or anything else. Basically, you can send a SOAP message via any means as long as the recipient can get to the message. I've seen SOAP used via HTTP(s) in about 99% of the cases and JMS used in those cases where queueing was necessary.

Now we're getting at the dimise of the ESB.

Remember what the ESB was good for?
  1. Data Exchange
  2. Data Transformation
  3. Data Routing
Well with SOAP being XML based and arguably very wel defined, the Data Transformation is not that relevant anymore. The Data Exchange with SOAP is handled as well, you use HTTP(s) in pretty much all the cases you come across it. Leaves us the Data Routing.
So what was the routing about? Well, it was all about having a logical name resolve to a physical location, where the ESB would determine where to send the message based on an address that one can remember, or that is not tied to a physical location. In the world of HTTP(s), which is dominated by the world of TCP/IP, this is actually handled by the DNS (Domain Naming Service). The DNS can perfectly handle this translation and do the routing. It's what is was designed for. Almost its raison d'etre. Maybe not even almost.
So in a world where we do SOAP based communications, there really is no reason to have an ESB.
So what else can an ESB do? Or rather; what else do we use an ESB for?

Many ESB vendors herald the fact that their ESB is perfect to handle webservices. I never understand what they mean by this. But I think it can mean only two things:
  1. The ESB can provide a WS* interface to a legacy service that is not providing a WS* interface. Basically it exposes a SOAP interface and translates this into a format and a protocol that the legacy service understands.
  2. The ESB can provide a legacy interface to a WS* service. In this case a client doesn't speak SOAP, can't invoke a WS* interface and instad it invokes another interface on the ESB which changes this call into a WS* interface and than returns the result in the correct format.
In the first case, the ESB is basically an adapter from WS* to a legacy format. Which in the old world of EAI makes perfect sense. But with WS* being such a well defined interface standard, it makes just as much sense to develop a WS* adapter in your prefered development environment and deploy it in your prefered application server. Typically this will save a lot of expensive licenses and you'll be able to scale way better. More on that later. The benefit here is that you can leverage the programming skills of your developers to develop in Java or .Net or PHP or any other programming language the WS* frontend and deploy it in any of the relevant application servers. Mind that more and more 'enterprise applications' (like Siebel, Dynamics, SAP, PeopleSoft, etc) provide a WS* interface. because everybody has been jumping on the WebServices bandwagon since the term was invented.
In the second case, the ESB is basically a transformation engine from some protocol to another. Something it has been very well suited. But again, since WS* and SOAP are so well established it makes as much sense if not more, to just develop this transformation at the client side instead of through the ESB. Again, license costs will be reduced and scalability will be enhanced.

Let's address that scalability issue for a second or two. First of all, an ESB doesn't scale. That's by design. It was never intended to scale, it was intended to be the pivot in enterprise communications. ESB's are not buses at all. They are the hub in a hub-and-spoke model. They have to be, because they are the middleman handling all the Babylonical speak impediments within an enterprise. Concurrency is handled by threading. An important aspect here is state. ESB's have the crucial design flaw of being able to keep state. 
When you talk to an ESB vendor about what the ESB can do for you, the vendor will tell you the three features I've listed before and an important 4th one. Well, the vendor thinks it's important, I believe it's a design flaw. This feature is "Data Enrichment". What they mean is that you can send a half-baked message to an ESB and before the ESB is delivering the message to the server, the recipient, it will call all kinds of other 'services' to enrich the original message with more information which is needed to deliver the message. This means that the ESB needs to keep state, while enriching the message. It also means that the ESB is no longer a mere intelligent data transport that routes and transforms on the fly, but it has become an application, a business application.
Because the ESB is designed to be able to handle this, the ESB is designed to be able to keep state. And thus is doesn't scale. It scales as far as the box it is running on can scale. Scalability is all vertically.
There's another problem with the scalability and that is the dependency on physical resources. The queues of an ESB are physical, they're filesystems, databases or something else that's physical and allows for some persistence, or even complete persistence. This means again, that it doesn't scale because it needs coordination of accessing these resources because they guarantee sequence.
When scaling an ESB, it needs to be setup across nodes and there needs to be a vivid dialogue between the nodes about what each node is doing and has been doing, this is pure overhead. The busier the ESB is, the more chatter, ie the more overhead. This will require more nodes which requires more chatter. The result is a very disappointing performance curve.

Don't worry, this is all by design. The design here is that the ESB should be doing more than just be an intelligent data transport, because were it just an intelligent data transport, it would not have any added value in a SOAP ruled WS* compliant world, which is today's world.
The ESB is designed to have added functionality to warrant the purchase of its licenses. This added functionality is allowing the sender (or client, or consumer) to not care about the sequence of its messages, because the ESB handles this. But that's a moo point since the client will either do a fire and forget (a feed) and not worry about sequence or it will wait for the response to a request before continuing processing. No added value at all. But the ESB, by queueing also ensures the sequence of requests or messages from several clients. So the recipient (or producer, or server) gets the requests or feed in the order the ESB receives them. Which means nothing because this may not at all be the same sequence the clients did in fact send them. Think about latency of different clients causing the ordering of messages going boinkers all over. Meanwhile this desparate need to sequence requires that there is a single point in the ESB doing all the sequencing. Which means dropping the desire to be scalable.

The ESB really has no place in an environment where webservices are dominating the communication between systems. And mind that what goes for SOAP based WS* compliant webservices also holds true for RESTful webservices that use XML or JSON as the message format language. Actually even more so, because RESTful webservices are predominantly HTTP(s) based.

Going back to the scalability for a second. In a webservices based environment, scalability is achieved at the recipient end, where the service producer (server or recipient) is designed such that it handles one request (or message) at a time. Such that ordering is either irrelevant or part of the interface specification.

Ask the ESB vendor again why you should get yourself an ESB after reciting the previous, and your vendor will likely start talking about monitoring. How convenient it is to have the ESB doing all the monitoring for you, keeping track of an audit trail or at least a message trail because it is the central hub through which all the messages flow. Emphasizing the ESB's main flaw again, it being a central hub.
But in this reason there's another flaw; The ESB can only monitor, or log for that matter, that what it knows. So it can log the receipt of a message by the ESB, not that a message was send to the ESB. Consequently, the ESB only knows about the sending of the message to the recipient, not about the recipient having received it. And I'm not really convinced that the information the ESB has about the message flow is the most interesting information. Not at all, hence you need to log the sending by the sender and reception by the recipient still. You could use the ESB to be a collector of log messages, have all systems send their log messages to the ESB and have it deal with them, store the messages in a database or in a log file or in both or forward them to a dashboard or do it all. And this is exactly why you should keep an ESB around. There's a full post on this in my blog.

Concluding, the ESB has no right to be a mainstream technology in today's and tomorrow's IT environment where webservices will dominate the interfaces. It's perfect for it's particular niche.

As always, I'm really interested in your views and ideas. More perspectives make for a richer understanding of the topic. So please share your thoughts, stay respectful and be argumentative.