API Management in Azure using Microsoft's solution - API's and Architecture, match made in heaven

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

Summary

The architect is arguably the one most concerned with API's as it is determined by the architecture where the API's reside, what should be API'afied. This has nothing to do with application architecture or integration architecture. The architectural structure of software has no relevance as to when and what to expose through an API. It is the business or domain architect that decides what will be exposed through an API, when and only when the Product Owner wants to reap the benefits of an API to his product. API development may not be controlled by rules, instead it should be governed by principles and guidelines. Although why something should be exposed as an API is a business (architect) call, the how it is exposed is a technical (architect) call.

API's need empowered teams to develop them and more importantly to maintain them. This is not an small thing. API's being extremely hard to develop, they are also extremely strainuous to an organisation because they typically upset the whole structure of the organisation. Even when that organisation is already a product organisation with product teams and product owners.

API's and Architecture

API's are among the hardest things to develop. Why this is, you can read in my previous post here. I think that the role of the architect is extremely important when it comes to API's because they provide access to functionality and information without compromising the underlying systems. And it's not access to just some information, but to information that has meaning on a business level. It's about disclosing your business to the unknown consumer. Well, it's the consumer of your API. So this means that your API's need to fit into the grand scheme of all things IT in your organisation. This is where the architect comes in, because the architect is the city-planner of your IT landscape. See my post on the role of the architect in your organisation, and how that architect may fail miserably here. 

Basically it would be the architect that has a strong say in what the boundaries of your IT landscape are and therefore where the API's are to be found. Mind that this is the IT component of your business architecture.

Say what? Well, it's not that difficult to understand. Consider your business architecture, in a world where IT is considered in terms of products that the organisation is making money off. (As opposite to seeing IT as an enabler or unique selling point.) So basically the organisation defines its business in terms of products. These products and the relationships between them are a way to see the business architecture. It's a very complex exercise to do this and on top of that, many organisations don't even want to make money perse. GO's and NGO's are examples. Still, consider that you do stuff that is of benefit to others, than what you do and what you'll do it with are your products.

These products are at times realised using IT. Maybe the full product is IT or parts of it are. These IT products and their relationships are the IT components of the business architecture I was referring to.

Every connection between any two products is a prime candidate of becoming an API, or becoming API'afied, if that were a word.

Did you notice that I am referring to the business architecture of the organisation and not to an application architecture that is adopted by an organisation. For example, I'm not referring to the 4-tier architecture by Forrester, or the traditional 3-tier (presentation, business, data) architecture, or an SOA or anything like that. Like I discussed in the API-first design post, these architectures have no relationship to your decision what should and what should not be API's. Your business architecture is. Sometimes, especially in larger organisations, that business architecture is decomposed into multiple domain specific architectures, again on a business level. Here you might argue that the relations between domains are even more prime candidates to become API's. And again, how your application architecture is defined has no effect on that.

The Platform

So this is where the architect has value in the API driven universe. Consider this; API's thrive in a platform economy, who decides whether or not there will be a platform and how the platform is shaped? It's the architect and no one else. And this is because the architect is the only one that views things beyond product boundaries, that's the architect's scope. The Product Owner and his Product Team are only concerned with the product. A true Product Owner will also understand the value of API's, because it means more users of his product and more users means more value, but it's the architect that decides a platform is to be developed, raising the value of API's several levels upwards. You should take a look at 'The Age of the Platform' by Phil Simon to understand the value of a platform and why it is such a key to success for many fast growing organisations. Mind that platforms are not only valuable in relation to external customers but also for internal customers.

So basically, the architect is the key role when it comes to API's. The architect defines the business architectures, models the IT products in it, considers where interfacing between domains and products is done through API's. The architect also decides whether or not to work on a platform and that kind of implies going for API's as well.

Organisational Impact

It should also be the architect to determine whether or not the organisation is fit for developing API's. Like I stated in API-first design, developing API's is one of the hardest things to do in software development. Which not only implies, but makes it rather explicit that your organisation needs to be ready in various areas to be able to develop and maintain API's. Continuous Delivery, and preferably Continuous Deployment needs to be in place. You would like to strive for a DevOps IT organisation and you need a very capable Product Owner. On top of that the developers working on API's need to be able to understand the relevant business concepts and need to have extensive experience in the pitfalls of software development. The architect's responsibility is to inform management, including HR management, the impact of adopting a platform strategy, or an API strategy on the organisation. Since architects need to be able to quantify the impact of a change to the IT landscape across all aspects of the enterprise, the architect will need to be able to quantify the impact on staff, staffing and processes alike.

Think about the opportunity to start small, have a single team mature enough to develop and maintain API's. That team should be mandated and thus empowered to develop API's and put them in production, with full autonomy. If the team needs to deploy in production, it should be their call. They should be held accountable and they're responsible for the success of the API. Once that works take all lessons learned and scale your organisation to more teams to work on API's for their products. The architect will know what domain and what product will be best to provide API's to interface with in order to create the most business value.

Governance vs. Control

And then there's the other thing, which is governance. There's no room for control in an API economy. A Product Team that develops API's to access the product's information and functionality will need to have full autonomy. And that means that it can't be controlled. Control means that there's a committee that decides whether or not the team has been abiding by the restrictive rules defined by some other committee. This assumes that it is possible to define rules that API's need to follow without knowing the API and therefore the context within the API is being used. Utterly impossible because when you start developing an API you have no clue about your customer and by monitoring very closely you get an understanding of who your users are and why they want to use your API.

So should there be no control at all? Of course there needs to be control, but it's of the scalable kind. Scalable control is called governance. Governance is done by first define a series of principles. A principle is a rule or belief that governs ones personal behaviour. It's intention is that because one believes that the principle makes sense and one is better off by applying the principle, life is better.

A principle is a rule or belief that governs ones personal behaviour.

By their very nature, principles are used for governance and not for control. It's is one of the most formidable ways to get self organising and autonomous teams work in concert and by the same playbook. The interesting part here is also that you can choose not to stick with a principle in a certain situation. A clear motivation is available implicitly because one abides by the principles, unless there's a good reason. It also allows in your governance structure room for classifying principles in those that can be dropped when the team chooses so and which need an explicit approval from some authoritative body, like an Architecture Board.

Besides the principles, which are set by the architect, the architect needs to define a set of guidelines that inform the Product Team the best course of action to develop an API. These can be related to processes, like how to publish an API. But also technical, for example naming conventions, versioning strategies, etc. Good guidelines are no-brainers but converge various alternative solutions to a problem into a single solution that is the default selectable solution to the problem. It's different from a principle because it's not appealing to a developer's common belief system but to the developer's common sense.

Governance is an architect's dominion. It's not related to a specific team, but it's there to ensure the integrity of the complete IT landscape. This is where the architect shines. The Product Team is responsible for its own principles and guidelines and they should never conflict with the overall, architect driven principles and guidelines. One could argue that the team only makes architecture decisions, based on the principles and guidelines.

Mind that principles are hard to define, almost as hard as API's. Always think of an API as something that sounds like a rule, that has a clear rationale and has an impact which on the one hand details the impact on the existing landscape, the external impact. But also has an internal impact, which details limitations, prerequisites etc on how the software subjected to the principle is developed.

Quite often I see principles that hint towards solutions. For example: High available systems are deployed on a cluster. The rationale is clearly a technical one and the impact restricts the technical aspects of the solution. But an architecture principle's rationale should be a business rationale instead. The impact should allow for as much freedom to the team developing the software without compromising the principle. For API's one needs to be extra careful when defining principles. Stating that API's are always exposed as SOAP webservices seems to be a good principle. SOAP is a great way to do b-2-b integration and it allows for a standardised approach to security, validation, etc. But what if the better way to expose the API is as a RESTful service using JSON? Well in that case the Product Team can motivate why the principle is dropped. But a principle like API's are always exposed based on internet standards and web-based protocols achieves the same, it also allows the product team to select the best solution for their situation and still hold on to the principle. Furthermore, the architect could define a guideline, or guiding principle, stating that for b-2-b connections SOAP is preferred and for b-2-c REST/JSON is preferred.

Empowerment

I've stated before, the Product Team should have a high degree of autonomy when it comes to API development. You want governance in place to make sure that anarchy won't reign. This means that the Product Team should be empowered to decide for itself when principles are selling them short, when a new not yet anticipated situation surfaces or the API's 'market' changes over night due to new insights. In addition, you want the team to be empowered to decide how they develop API's in the most efficient way. Meaning that thinking of an API and having consumers calling the API should be an efficient as possible process. This is an important statement, because it means that the team can't just say 'screw the process or the organisation for that matter' because they have autonomy. Because doing so, would not benefit the complete process from idea to used product.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.
But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan

The demise of the ESB in a world of webservices

Abstract: In a world where no single enterprise can develop the software needed to run the enterprise, it no longer makes sense to connect everything to the Enterprise Service Bus, but instead everything should be connected to the internet. This is exactly what is happening. And this trend is causing the dimise of the ESB as there is no validity to the ESB being the backbone of Enterprise Application Integration, something that was promised by the major players in the world of messaging. Promises never truly delivered on.

Remember what the ESB was good for?

1. Data Exchange
2. Data Transformation
3. Data Routing

A role that has been assumed by web-services, both in the form of SOAP based web-services (predominantly conforming to the WS* standards) and in the form of REST-based web-services.

Other posts in this trilogie:
- ESB's don't scale, which shouldn't be a problem because that is by design
- The ESB addresses a niche, not mainstream

In this third post in a series of three on the ESB of Arc-E-Tect I will delve deeper in the inevitable dimise of the ESB in today's world of connected systems.

Let's face it, the ESB isn't dead, it was never alive in the first place. Not in the incarnation the big vendors would like us to believe. It was and is merely a renamed messaging system.

That being said we can concentrate on whether or not we should still consider the ESB. The Enterprise Service Bus.

This is a post in a series about the Enterprise Service Bus, ESB, an acronym so easily to love or to hate. Parts of this post are also found in another post on the dimise of the ESB in a world of webservices.

First of all, let's not call it Enterprise Service Bus, but stick with ESB. Just like SOAP, which originally was the Simple Object Access Protocol and now just is SOAP. Probably because there was nothing Simple with SOAP. Just like there's nothing Enterprise about ESB. Unless of course you consider the budgets needed to actually get something running.

So, the ESB became obsolete. First of all, we should identify what the ESB was supposed to deliver before we can say it became obsolete.

Basically there are just a few functionalities an ESB actually was supposed to provide.

1. Data exchange. Now I'm carefully choosing my words here and keeping it as abstract as can be. But in fact the ESB's data exchange capabilities didn't extend much further as messaging. Messaging in the sense of what products like IBM's MQ Series and Microsoft's ActiveMQ provide. You have a limited amount of data, and you send it between two points.
2. Data transformation. Actually this should be message transformation, but it applies to all data. The ESB is particularly good at transforming one data format to another. This is actually some legacy capability from the time there was no ESB and only MOM (Message Oriented Middleware). Because the promise of being able to loosely couple two systems by utilizing MOM justified the creation of functionality that would transform one data format to another. This to further decouple both points.
3. Data routing. Again, the premise here was to decouple sender and receiver. As the ESB already decouples sender and receiver by implementing an asynchronous communication protocol, by allowing for transformation from the sender's format to the receivers format, the only coupling aspect still to be resolved is the addressing. The ESB removes the necessity of the sender knowing where the receiver is located by allowing for logical names that are resolved and data is routed to the 'correct' recipient.

In 1997 I started working on an ESB using IBM MQ Series as the underpinning messaging technology, we implemented all three core capabilities of the ESB. The likes of Microsoft, IBM, Lotus and Neon came to visit us as they wanted to see what we were doing. This all at a one of the Dutch leading banks.

As we implemented the usage of the ESB and later on as the Middleware architect and Enterprise Application Integration Architect (EAI) for this customer I found that the premise of the ESB, which at the time was called MOM (Message Oriented Middleware) was awesome.
We connected a front-end developed in Visual Basic with a backend developed in COBOL. On the client side, we developed an ActiveX control in C++ and a whole series of modules that allowed for on the fly creation of message formats, stored in a meta-data format that was retrieved from a meta-data server. The initial format for the messages was actually the format the meta-data server understood and we called it meta-meta data. Pretty deep, huh :)
On the server side, a COBOL program running on the AS/400 we developed some components using C as the programming language of choice. Here as well we developed something similar to the meta-data server. As I recall, we were using copy-books.
It was all pretty advanced stuff and we were able to decouple the server and the client, develop both independently from each other and the queueing provided by MQ Series allowed also for asynchronous communication.
Shortly thereafter products came on the market that did a lot of the functionality we implemented and on the part of MQ Series, NEON was a marketleader.

This is the time around which SOA (Service Oriented Archtecture) started its emergence and everybody and their mother who thought of themselves as a messaging vendor jumped on this bandwagon. With not much more than the good old MOM with a new label attached to it.

Interestingly enough, the messaging format problems was solved by the introduction of XML. XML already existed but as it was both text oriented and self describing and on top of that, like a cherry, it was human readable... more or less. The problem of course still being that the actual message has to ad here to a defined format. DTD's (Document Type Definition) and later XSD's (eXensible Schema Definition) were invented to ensure this and both standards are doing a pretty good job at it. When combining this with XSLT (eXtensible Stylesheet Language Transformation) you have a format that is self describing, a means to define the format and a means to transform from this format to pretty much every other format.
Don't get me wrong, I really love the fact that with XML I can do all this stuff. More importantly, I don't need a message tranformation engine to do so. Of course, this awesomeness only works when you start off with XML.

From a decoupling perspective this is understandably pretty neat because it allows the sender of the message to determine the message format and the recipient can parse the message in a standard way, verify that it complies to the definition and because of that transform it to a format it can understand. Granted, when the sender doesn't use XML you're stuck with being able to handle that format. And in those cases a message transformation engine comes in pretty handy.
The question is what happens when a lot of senders send messages in XML but not the format the recipient understands. Who will do the transformation to the right format? You could do this at the recipient end, but that is bad practice. Why? Because you're coupling sender and receiver again. That was something we were tryint to avoid. Doing this in a message transformation engine makes perfect sense and is the right way to handle this.

So XML is excellent for messaging. Messaging is awesome for asynchronous communciation. And it is great for communications where the sequence of messages is important. Many communications are not asynchronous at all. They're request/reply style interactions. But interestingly enough, these typically don't require ordering... and when the communication si asynchronous, we typically notice that ordering is important.
First about the request/reply not being ordered. What you should realize is that in these cases the client, the requestor takes care of only proceeding with the processing once the reply is received. So the handling from a server (recipient) of multiple requests is regarind multiple requests of multiple clients. These are independent and therefore by nature unordered.
The communications that are asynchronous but need to be ordered are usually feeds of data that need to be processed in sequence. Data feeds from a single client are sequence by using a... yes... a queue. It's the perfect mechanism for sequencing. When dealing with multiple clients feeding the server, we need to resolve to unique sequence numbers that are universally unique or using timestamps... which is always tricky.

I'm probably not the first to notice this as SOAP and WS* standards are addressing this exactly. By this I'm meaning the benefits of XML and the characteristics of communications between systems.
The XML part of SOAP based webservices is that the XML is very well defined. Arguably that is, but from the perspective of the freeformat XML actually is, it is darn well defined. Also, the webservice is a request/reply protocol. In case you want to use it for feeds you just ignore the reply.
There's another nice feature of SOAP and that is that it can use various communication protocols like HTTP(s), JMS (Java Message Service), SMTP (Simple Mail Transfer Protocol, or email) or anything else. Basically, you can send a SOAP message via any means as long as the recipient can get to the message. I've seen SOAP used via HTTP(s) in about 99% of the cases and JMS used in those cases where queueing was necessary.

Now we're getting at the dimise of the ESB.

Remember what the ESB was good for?

1. Data Exchange
2. Data Transformation
3. Data Routing

Well with SOAP being XML based and arguably very wel defined, the Data Transformation is not that relevant anymore. The Data Exchange with SOAP is handled as well, you use HTTP(s) in pretty much all the cases you come across it. Leaves us the Data Routing.
So what was the routing about? Well, it was all about having a logical name resolve to a physical location, where the ESB would determine where to send the message based on an address that one can remember, or that is not tied to a physical location. In the world of HTTP(s), which is dominated by the world of TCP/IP, this is actually handled by the DNS (Domain Naming Service). The DNS can perfectly handle this translation and do the routing. It's what is was designed for. Almost its raison d'etre. Maybe not even almost.
So in a world where we do SOAP based communications, there really is no reason to have an ESB.
So what else can an ESB do? Or rather; what else do we use an ESB for?

Many ESB vendors herald the fact that their ESB is perfect to handle webservices. I never understand what they mean by this. But I think it can mean only two things:

1. The ESB can provide a WS* interface to a legacy service that is not providing a WS* interface. Basically it exposes a SOAP interface and translates this into a format and a protocol that the legacy service understands.
2. The ESB can provide a legacy interface to a WS* service. In this case a client doesn't speak SOAP, can't invoke a WS* interface and instad it invokes another interface on the ESB which changes this call into a WS* interface and than returns the result in the correct format.

In the first case, the ESB is basically an adapter from WS* to a legacy format. Which in the old world of EAI makes perfect sense. But with WS* being such a well defined interface standard, it makes just as much sense to develop a WS* adapter in your prefered development environment and deploy it in your prefered application server. Typically this will save a lot of expensive licenses and you'll be able to scale way better. More on that later. The benefit here is that you can leverage the programming skills of your developers to develop in Java or .Net or PHP or any other programming language the WS* frontend and deploy it in any of the relevant application servers. Mind that more and more 'enterprise applications' (like Siebel, Dynamics, SAP, PeopleSoft, etc) provide a WS* interface. because everybody has been jumping on the WebServices bandwagon since the term was invented.

In the second case, the ESB is basically a transformation engine from some protocol to another. Something it has been very well suited. But again, since WS* and SOAP are so well established it makes as much sense if not more, to just develop this transformation at the client side instead of through the ESB. Again, license costs will be reduced and scalability will be enhanced.

Let's address that scalability issue for a second or two. First of all, an ESB doesn't scale. That's by design. It was never intended to scale, it was intended to be the pivot in enterprise communications. ESB's are not buses at all. They are the hub in a hub-and-spoke model. They have to be, because they are the middleman handling all the Babylonical speak impediments within an enterprise. Concurrency is handled by threading. An important aspect here is state. ESB's have the crucial design flaw of being able to keep state. 

When you talk to an ESB vendor about what the ESB can do for you, the vendor will tell you the three features I've listed before and an important 4th one. Well, the vendor thinks it's important, I believe it's a design flaw. This feature is "Data Enrichment". What they mean is that you can send a half-baked message to an ESB and before the ESB is delivering the message to the server, the recipient, it will call all kinds of other 'services' to enrich the original message with more information which is needed to deliver the message. This means that the ESB needs to keep state, while enriching the message. It also means that the ESB is no longer a mere intelligent data transport that routes and transforms on the fly, but it has become an application, a business application.

Because the ESB is designed to be able to handle this, the ESB is designed to be able to keep state. And thus is doesn't scale. It scales as far as the box it is running on can scale. Scalability is all vertically.

There's another problem with the scalability and that is the dependency on physical resources. The queues of an ESB are physical, they're filesystems, databases or something else that's physical and allows for some persistence, or even complete persistence. This means again, that it doesn't scale because it needs coordination of accessing these resources because they guarantee sequence.

When scaling an ESB, it needs to be setup across nodes and there needs to be a vivid dialogue between the nodes about what each node is doing and has been doing, this is pure overhead. The busier the ESB is, the more chatter, ie the more overhead. This will require more nodes which requires more chatter. The result is a very disappointing performance curve.

Don't worry, this is all by design. The design here is that the ESB should be doing more than just be an intelligent data transport, because were it just an intelligent data transport, it would not have any added value in a SOAP ruled WS* compliant world, which is today's world.

The ESB is designed to have added functionality to warrant the purchase of its licenses. This added functionality is allowing the sender (or client, or consumer) to not care about the sequence of its messages, because the ESB handles this. But that's a moo point since the client will either do a fire and forget (a feed) and not worry about sequence or it will wait for the response to a request before continuing processing. No added value at all. But the ESB, by queueing also ensures the sequence of requests or messages from several clients. So the recipient (or producer, or server) gets the requests or feed in the order the ESB receives them. Which means nothing because this may not at all be the same sequence the clients did in fact send them. Think about latency of different clients causing the ordering of messages going boinkers all over. Meanwhile this desparate need to sequence requires that there is a single point in the ESB doing all the sequencing. Which means dropping the desire to be scalable.

The ESB really has no place in an environment where webservices are dominating the communication between systems. And mind that what goes for SOAP based WS* compliant webservices also holds true for RESTful webservices that use XML or JSON as the message format language. Actually even more so, because RESTful webservices are predominantly HTTP(s) based.

Going back to the scalability for a second. In a webservices based environment, scalability is achieved at the recipient end, where the service producer (server or recipient) is designed such that it handles one request (or message) at a time. Such that ordering is either irrelevant or part of the interface specification.

Ask the ESB vendor again why you should get yourself an ESB after reciting the previous, and your vendor will likely start talking about monitoring. How convenient it is to have the ESB doing all the monitoring for you, keeping track of an audit trail or at least a message trail because it is the central hub through which all the messages flow. Emphasizing the ESB's main flaw again, it being a central hub.

But in this reason there's another flaw; The ESB can only monitor, or log for that matter, that what it knows. So it can log the receipt of a message by the ESB, not that a message was send to the ESB. Consequently, the ESB only knows about the sending of the message to the recipient, not about the recipient having received it. And I'm not really convinced that the information the ESB has about the message flow is the most interesting information. Not at all, hence you need to log the sending by the sender and reception by the recipient still. You could use the ESB to be a collector of log messages, have all systems send their log messages to the ESB and have it deal with them, store the messages in a database or in a log file or in both or forward them to a dashboard or do it all. And this is exactly why you should keep an ESB around. There's a full post on this in my blog.

Concluding, the ESB has no right to be a mainstream technology in today's and tomorrow's IT environment where webservices will dominate the interfaces. It's perfect for it's particular niche.

As always, I'm really interested in your views and ideas. More perspectives make for a richer understanding of the topic. So please share your thoughts, stay respectful and be argumentative.

Iwan

The ESB addresses a niche, not mainstream

Other posts in this trilogie:

- ESB's don't scale, which shouldn't be a problem because that is by design
- The demise of the ESB in a world of webservices

There are many reasons not to have an ESB in your environment. The ESB as we know it has less and less a reason to be implemented in today's world of webservices. There's a full post based on the dimise of the ESB in a world full of webservices.
But this post is about why you should have an ESB and why it does make sense to implement it. Or rather, when it makes sense to implement it.

This is a post in a series about the Enterprise Service Bus, ESB, an acronym so easily to love or to hate. Parts of this post are also found in another post on the dimise of the ESB in a world of webservices.

Remember what the ESB was good for?

1. Data Exchange
2. Data Transformation
3. Data Routing

Well with SOAP being XML based and arguably very wel defined, the Data Transformation is not that relevant anymore. The Data Exchange with SOAP is handled as well, you use HTTP(s) in pretty much all the cases you come across it. Leaves us the Data Routing.
So what was the routing about? Well, it was all about having a logical name resolve to a physical location, where the ESB would determine where to send the message based on an address that one can remember, or that is not tied to a physical location. In the world of HTTP(s), which is dominated by the world of TCP/IP, this is actually handled by the DNS (Domain Naming Service). The DNS can perfectly handle this translation and do the routing. It's what is was designed for. Almost its raison d'etre. Maybe not even almost.
So in a world where we do SOAP based communications, there really is no reason to have an ESB.
What do we keep the ESB around for?

You keep the ESB around for the many-to-many messaging. You keep it around for those cases where you have one or more sender of (similar) messages and one or more recipients of these messages. Because the ESB is a great message dispatcher, it's the message routing part that kicks in. It's something the DNS can't do. In those cases where you want to send the same message to an unspecified amount of recipients where the recipients can register or deregister to receive messages at their leisure you'll want the ESB because managing this at the client side, maintaining a list of recipients and sending each message to each of the registered recipients is bad for performance, susceptible to errors and complicated.

The ESB is great at dispatching the same message to multiple recipients, doing transformations of the data on the fly. So duplicating a production load of messages into a test environment through the ESB, where the ESB is transforming the data just in time before sending it to the test systems (e.g. anonymizing the data, translating from one spoken language to another, transposing id's such that they align with keys in a database etc) is a common practice.

I have worked at a financial institution where we fenced the core banking systems with an ESB, catching every transaction send to the corebanking system and duplicating it to a standby system as well as a test system in a slightly altered form to be able at one side to have resilience at an extremely low price and production load with relevant data in a test environment.

Basically this means that you should keep the ESB around for being able to follow the concept of publish/subscribe. In my opinion the only right concept when it comes to asynchronous communciations. And since this can only be done by either using lower level network protocols like UDP (TIBCO's TIB/Rendezvous was built on UDP and the first real pub/sub message bus), or by having a central dispatcher sending a message to the registered recipients, or by handling it all at the sender side it makes most sense from a security as well as a manageability perspective to do publish/subscribe through a centralized dispatcher. Although for the really lean and mean high performance situations, you want to resolve to lower level network implementations.

Understanding pub/sub and implementing it in the enterprise allows for maximumenablement of the business with minimal IT investments. The afore mentioned use-cases of the monitoring systems where all systems publish monitoring messages (on business level) and various systems subscribe to them, for example a dashboard, and you'll get a very rich BAM (Business Activity Monitoring) solution, which without much additional effort can be used as an application level monitoring solution as well.
The other use-case where production feeds were used to implement improved availability as well as the capability of testing a new release with relevant production loads is of course pretty awesome as it didn't need a lot of additional work once the ESB fence was set up.

So dispite popular believe, there is a reason why you should consider implementing an ESB in your enterprise. But don't adopt it as the default means for communications between applications, instead apply the ESB as a niche product and not as mainstream.

As always, I'm really interested in your views and ideas. More perspectives make for a richer understanding of the topic. So please share your thoughts, stay respectful and be argumentative.

Iwan

ESB's don't scale, which shouldn't be a problem because that is by design

Other posts in this trilogie:

• The dimise of the ESB in a world of webservices
• The ESB addresses a niche, not mainstream

Let's address that scalability issue. First of all, an ESB doesn't scale. That's by design. It was never intended to scale, it was intended to be the pivot in enterprise communications.

This is a post in a series about the Enterprise Service Bus, ESB, an acronym so easily to love or to hate. Parts of this post are also found in another post on the dimise of the ESB in a world of webservices.

ESB's are not buses at all, we only draw them as a bus or rather as a tube to which everything is connected. We draw them the same way we draw TCP/IP networks. But they are actually the hub in a hub-and-spoke model. They have to be, because they are the middleman handling all the Babylonical speak impediments within an enterprise.

An important aspect of the problem of not being scalable is state. ESB's have the crucial design flaw of being able to keep state.

Remember what the ESB was good for?

1. Data Exchange
2. Data Transformation
3. Data Routing

When you talk to an ESB vendor about what the ESB can do for you, the vendor will tell you the three features I've listed before and an important 4th one. Well, the vendor thinks it's important, I believe it's a design flaw. This feature is "Data Enrichment". What the vendor means is that you can send a half-baked message to an ESB and before the ESB is delivering the message to its final destination, the recipient, it will call all kinds of other 'services' to enrich the original message with more information which is needed to deliver the message and have it understood, processable, by the recipient. This means that the ESB needs to keep state, while enriching the message. It also means that the ESB is no longer a mere intelligent data transport that routes and transforms on the fly, but it has become an application, a business application.

Because the ESB is designed to be able to do this, the ESB is designed to be able to keep state. And thus it doesn't scale. It scales as far as the box it is running on can scale. Scalability is all vertically.

There's another problem with the scalability and that is the dependency on physical resources. The queues of an ESB are physical, they're filesystems, databases or something else that's physical and allows for some persistence, or even complete persistence. This means again, that it doesn't scale because it needs coordination of accessing these resources because they guarantee sequence.

When scaling an ESB, it needs to be setup across nodes and there needs to be a vivid dialogue between the nodes about what each node is doing and has been doing, this is pure overhead. The busier the ESB is, the more chatter, ie the more overhead. This will require more nodes which requires more chatter. The result is a very disappointing performance curve.

Don't worry, this is all by design. The design here is that the ESB should be doing more than just be an intelligent data transport, because were it just an intelligent data transport, it would not have any added value in a SOAP ruled WS* compliant world, which is today's world. There's a whole blog post on this.

The ESB is designed to have added functionality to warrant the purchase of its licenses. This added functionality is allowing the sender (or client, or consumer) to not care about the sequence of its messages, because the ESB handles this. But that's a moo point since the client will either do a fire and forget (a feed) and not worry about sequence or it will wait for the response to a request before continuing processing. No added value at all. But the ESB, by queueing also ensures the sequence of requests or messages from several clients. So the recipient (or producer, or server) gets the requests or feed in the order the ESB receives them. Which means nothing because this may not at all be the same sequence the clients did in fact send them. Think about latency of different clients causing the ordering of messages going boinkers all over. Meanwhile this desparate need to sequence requires that there is a single point in the ESB doing all the sequencing. Which means dropping the desire to be scalable.

Ask the ESB vendor again why you should get yourself an ESB after reciting the previous, and your vendor will likely start talking about monitoring. How convenient it is to have the ESB doing all the monitoring for you, keeping track of an audit trail or at least a message trail because it is the central hub through which all the messages flow. Emphasizing the ESB's main flaw again, it being a central hub.

The ESB doesn't scale and it shouldn't be applied in an environment where it is meant to scale beyond just a few boxes. This is why the 'Enterprise' in ESB is misleading. It makes far more sense to have several ESB implementations in the enterprise, serving specific needs. There is a place for the ESB in the enterprise, trust me.

As always, I'm really interested in your views and ideas. More perspectives make for a richer understanding of the topic. So please share your thoughts, stay respectful and be argumentative.

Iwan