API Management in Azure using Microsoft's solution - API-first design

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

Summary

API's are about the hardest thing to create in software engineering. Designing software to act as an API before anything else is even harder.

There's a new buzz going around in IT; API Management, and everything associated with it. I think it's even more buzzy than Microservices. Especially with shout-outs like "API First", compared to "Cloud First", "Mobile First" and "Web First" of yesteryear, we need to be cautious about what this means. This post is not about what API's are, why you should develop API's, or why API development is probably the hardest area in software development in this day and age. This post is about how you should look at Microsoft's offering on API Management and how to use it. Not the definitive guide, but a guideline instead.

This post is focusing on the API-first buzzzzzz...

Microsoft Azure

Skip to the next section when you're not interested in a view of mine on Azure, otherwise keep reading.

Whatever you think about Amazon AWS and the incredible pace by which Dr. Werner Vogel and his team is telling the world what The Cloud is and will be, you have to agree that the more amazing player in the Cloud arena at this point in time is arguably Microsoft.
Amazon has a track record of providing an amazing experience to their customers, both at Amazon.com but also on AWS. Clearly considering what the customer wants and doing everything to make the customer want to spend more money with them then anywhere else. They're doing this for over a decade now. I would argue that AWS is for those that understand the Cloud. Microsoft is arguably only since about 5 or 6 years a Cloud player to consider. Although announced in 2008, their implementation of the Cloud was in early 2010 and called Microsoft Azure in 2014. It really matured in 2016 with proper RBAC (Role Based Access Control) and proper infrastructure-in-code capabilities through the Azure Resource Manager. Other than AWS, which is a true 'API first' and 'Automate everything' platform, Azure is only since recently truly catching up in allowing everything to be done through API's and automation. Which is too bad since one of the Cloud's most awesome traits is the virtually infinite amount of resources allowing to provision and deploy a virtually infinite amount of versions of an application. Anyway, Microsoft's lack of API's and automation capabilities in the Cloud is consistent with its desktop platform Windows, which is a point-and-click affair instead of a scripting affair. Even administrators on Windows will use the point-and-click interface instead of the CLI (Command Line Interface) which is so accustomed to their Unix and Linux colleagues. It makes the platform less intimidating and many of the hard work is hidden behind a button and some checkboxes. This is how the Cloud is managed as well, through a Portal. It's a "point-and-click first" philosophy it seems. And many of the customers of Azure are happy with this. Who wants to be intimidated by a computer after seeing 2001 and Terminator anyway?

Setting AWS and Azure side-by-side one might see the following key difference: AWS allows you to do IT like playing with Lego. Amazon started with the basic building blocks. The Lego of 30 years ago and they allow you to build pretty much everything, if you're willing to spend the time. As time passes they come out with more advanced blocks, like the roofs in Lego. And just like Lego came with the little Lego men, gears, axles and so on, Amazon introduced more and more advanced services on AWS. But the idea, it seems, has always been to allow the customer to build pretty much everything she wants. Microsoft on the other hand seems to have followed the PlayMobil path. The fun is in the playing part and not so much in the building part. So open the Azure box and unlike AWS, you can start playing from day one. And as time passes by, there are more and more lower level components available, so you can start playing new games by building new toys.
Neither approach is better than the other, but you should carefully look at where you are, what you can do and what you want in order to be able to select either AWS or Azure. And IMHO, if you're just selected to go with Azure, don't even think about also doing AWS. And the other way around, because you would be most likely dealt lemons and you won't be able to make lemonade out of them.

API Management

First of all, API Management is a concept. I'm taking it you understand what an API is and I won't go into it any further than necessary for understanding this post.
API Management is done by people and some of them are using tools and technologies we call API Managers. Just like an enterprise, it has Management, and Management is made up of Managers.

Second of all, API-first design is not a fluke and it's not a hype either. Instead it is calling doom and gloom over your IT organization, or quite the opposite when you really know what you're doing.

Third of all, there's nothing MicroServices about API's although you see both walking around hand-in-hand more often than not.

Let's start with number 2...

API-first design
With API-first design, what you do is, you think about the functionality or information you want to expose and design it such that you don't have a clue about who will need or want to use that functionality or information. What you need to do next is worry about this. Start worrying a lot, because if you don't have a clue about who will access your functionality or information, let's call it a service, you don't know their intent either. And you don't know how often they'll access your service. You don't know whether or not you can trust them either. Are they taking security as serious as you do? Are they complying with all regulations you need to comply with? etc. Worries, worries, worries.

By not making any assumptions about the consumer of your service, you will need to make sure that it can stand the assault of all kinds of consumers. I do see your mind fluttering into the direction of non-functionals. Things like resilience and availability maybe? And encryption and access management possibly as well. Drop the mother in the cloud and you've got your scalability straight out of the Cloud's elasticity. (Of course you're very intimate with my post on this topic). And when you're in the Cloud, you know pretty much for sure that your Cloud provider will have all kinds of network protection in place without you even knowing it. Making sure that you, nor any other customer of theirs, will suffer from a DDOS attack and the likes.
But when we talk about API-first, we also need to think about the granularity of the service's exposed functionality or information. We need to think about it very, very carefully. In general we can say that less is more when it comes to this. But in fact, you want to hit that sweet-spot, because only then you're API is build to last.

That sweet-spot is something really important and almost impossible to find, let alone to hit. It's like the mythical G-spot. We know it should be there, but once you find it, you don't have the guarantee you will be able to find it all the times you're looking for it.
Granularity of your service defines it's usability and it is extremely business-bound. So first of all, you will need to know exactly the business context of your API and, and this is important, understand the business context of your consumers. And you should, no you must, accept that you'll have this latter one wrong. The most unlikely consumer will be your most valuable customer, so unless you're capable of reacting to her needs very quickly. Able to adapt to a massive change in positioning of your API in the grand scheme of the universe due to a pivot of your business, don't even try to go for API-first design.

By getting down to business, understand that an API is to be considered a product. It's not just access to your productised software solution, no it is your product. Nobody will care and shouldn't care how you do the magic behind the API as long as you're doing it. So you need to treat your API as a product. And that means that your API should have business value and as such, you should be very much aware that your API is valuable to your customers. In short, think about your revenue generated by your API. This is where your business context is showing it's ugly face again, because initially you can only assume how your API can be valuable to your customers. Hence you can only guess, albeit an educated guess, how you can get revenue from your API. Here again you'll need to be able to adapt to changing views and accept that hindsight is 20-20.
What this means is that you'll have to build (business) metrics into your API. Think upfront how you can verify your hypothesis and then do the validation of it as well. And don't be shy to change your product. A good product is one that is used, a great product is one that is valuable.
A key aspect of your API's granularity is about how elementary it is. You don't want to have an API that is only providing basic information, one that is only spewing data. Instead you want an API that is providing information in context. How much context is part of finding that sweet-spot. For example an Address by itself might mean nothing unless you know whose address it is, so a service that spews out addresses is of limited value, but one that provides access to a person and her address might be invaluable. Or maybe access to information is not that much of a thing to consider and instead you provide access to a function, for example to put some random text on API Management on the internet.

A good product is one that is used, a great product is one that is valuable.

The point here is that there's no way to determine what makes sense as an API and what not, unless you understand the context of your API at your end. And this only because it provides you with a starting point to make educated guess about the context of your API's consumer.
And at all times, the quality of your product needs to be such that you won't need to enter dialogue with the customer about the product, explaining why it's not working.

By the way, your architecture, blue-prints or design-patterns are no indication of what should be an API in the sense of API-first. In fact, I would argue that they are not an indication of what should be API's and managed as such and what not.

Considering you're developing according to service oriented concepts, and quite possibly adopting the principles of Forrester's 4-tier architecture. Which is in fact not a bad thing at all and probably a very beneficial view on things in general. So considering this, you might be inclined to consider very service in a specific architectural layer to be an API. For example the Aggregation Tier in Forrester's model or the Systems Tier in that model, could be considered to be an API. Which is total rubbish!
In software engineering we don't just do, we think about it and decide. So calling something a service in a specific part of our architecture an API is nonsense, should be cause of being fired in fact. And in fact, after thinking about it and deciding you have a service that is bound to be reused doesn't mean that will be an API either. So you'll have to do some more thinking... think... think... think. And then worry about calling something an API when you know in what situation the service is going to be used. Worry about calling something an API when you know who's going to call you to yell at you because the service is not abiding the service contract. Because if you know this, you know who your service's consumer is, and most likely there's no need to do any API'afyication on your service. Doesn't mean that you shouldn't treat it as an API, it means you don't have to treat it as an API if that doesn't warrant the effort.

In software engineering we don't just do, we think about it and decide.

API's have all kinds of stuff in them one way or the other, that allows them to be unleashed into the world. It's all the stuff that ensures that they survive the onslaught of calls by initially unknown subjects, the unsubs.
It's the part of API-first design that ensures that none of the wonderful promises of pots with gold by exposing the service's splendour, is compromised. It's the resilience, availability, scalability, security as well as metrics and other business requirements we too often consider IT stuff.
Think about the idea of implementing it over and over again for every service interface. If that means you're starting to sigh, and very deep at that, than only do it for the real API's. The service interfaces for which you don't know who'll call them. But when you get a smile on your face, eyes start to gleam, then you might as well take it from the other end; think about why you shouldn't treat the service as an API. Could be because of money (using standard tooling to API'afying service interfaces cost money) or maybe the operational overhead is not worth it.

So, in order to do API-first design, you need to understand your own business, meaning you need to understand what you want to expose and how you can create business value with your information or capabilities. This is hard enough for most software engineers, so they'll need the support of a business expert. Than you need to think about why others, and you don't know who that might be, would like to use your API, your product, and why it's valuable for them. This requires business knowledge and some understanding of how to see business value in using information flows. This is not the average business person you need to work with.
Than you need a thorough understanding of how to develop a resilient, performant and secure piece of software that can run in the Cloud, i.e. and elastic environment that is to be considered to be crashing all the time. And if you have all this in place, you need to be able to be roll-out new versions of your product on a very regular basis, with an extremely high level of quality. And on top of that, never get frustrated because your business pivots over and over again, because they're not really sure for whom the product is intended.

API's are about the hardest thing to create in software engineering. Designing software to act as an API before anything else is even harder.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.
But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan

API Versioning, another religious war

Summary: When you ask your best friend, e.g. Google, for an answer as to what is really the best way to handle different versions of the same interface, you'll find out that there is no straight answer. Actually it is a religious war and many of the  arguments given for either of the various ways of versioning interfaces are actually just opinions on aesthetics. Below you'll read my reasoning for what my views are on the various options you'll have to version your service interfaces. Probably you'll find other ways of skinning this cat as well.


One of the core aspects that we need to decide upon when working on our web-services is decide on how we will handle different versions of the services.

There are two dimensions to this:

1. Production vs. Non-Production (We're not doing DTAP but PorN) See the relevant principle here: Everything is Production or Not Principle. 
2. Current version of the web service and the next version of the web service.

Although these dimensions seem to be different, in fact we're talking about one and the same challenge we'll have to overcome.

Basically we have a version of a service in production, and one version in development (in case we need to change the service). Once that development version reaches production, we have two versions in production according to this principle Service Interface Compatibility Principle. As long as no customer can access the version that is in development, this version is not in production, according to principle RBAC is everywhere, every service can only be used by a user with the right role. Somebody not being a developer cannot use a service that is in development. This is an important notion because it removes the necessity of a separate infrastructure to separate production from non-production, although it still might be a good idea to have some separation.

As a consequence, we can deploy a service and limit access to the service through Role Based Access Control and since development of a service always results in a new version of that service, we can see the solution to dimension 2 as a solution for dimension 1, essentially allowing us to see both to be the same.

Looking at service versioning, we have to distinguish between service interface and service implementation. Since both have their own life-cycle, we can treat them separately. In addition, when we talk about service versioning we are mainly concerned about the impact of a new version of a service onto the consumer of that service, so we're concerned about the service interface and not so much with the service implementation. So within the scope of this post, we're discussing the version of the interface and not its implementation.
Furthermore, when we talk about services, we talk about RESTful services. And finally, we talk about the version of the published interface which is different from the version of the source code implementing that interface or the service attached to it.

When the interface of a service changes, we either deal with a change that breaks the consumer, or one that doesn't break the consumer. At all times we strive for the latter, since not breaking the consumer means no impact on the consumer. Changes that extend the resource, are changes that have no impact, since consumers of RESTful services are to ignore what they don't know. So any new fields in the resource representation will be ignored. Consumers that ignore these fields will not be able to benefit from the change. These are syntactical changes we want, and versioning has negligible impact on the consumer, the consumer does not need to be aware of the change.
Changes to the syntax of the resource representation we don't want but are inevitable, are those that change existing fields. For example they change the type of a field from an integer to a float. In this case we need to make sure that the consumer is not confronted with the change without it changing as well. Hence either the service should no longer be accessible, resulting in an error at the consumer, limiting damage. Or by being backwards compatible and thus returning the old representation to that consumer.

As a third change, we have semantic changes, which are changes to the actual resources. For example a generic resource turns out to have become more specific and we start limiting representations to just those specific resources. For example we have the concept of Customer which is a Company with which we have a Contract, and then we also have Customers that are not companies but Consumers, although still Customers, they're not the same. As the semantics of the resource has changed, we can't talk about the resource in relation to the changing interface as if it were the same resource. The actual resource has changed, so has its representation and therefore it impacts the consumer, which will break. In this case, again, we need to make the old service to become unavailable or we keep on supporting it.

There are in essence three major philosophies when it comes to API or service versioning:

1. Include a version in the URI that identifies the resource, e.g. https://www.foobar.com/api/<version>/<resource>
2. Include a version in the URI that identifies the resource, using a request parameter, e.g. https://www.foobar.com/api/<resource>?version=<version>
3. Include a version in the Accept-header of the request, thus using http Content Negotiation, Accept: application/vnd.<resource>.<version>+json

Of these three cases of changing interfaces, the last one is the easiest to address. Since the change results in a new resource type, the identifier of the resources, the URI, must be changed. This results in a URI in accordance with: http://www.foobar.com/api/<version>/<resource>. Although the location of the version indicator is arbitrary, the best practice is to have it as close to the domain as possible. Typically right before or right after the resource. The philosophy here is that we're talking about a new resource and not a new version of an existing resource. Hence the version indicator in front of the resources. Alternatively, the version is omitted and a new resource name is used. This is preferred when the resource should have a new name. In the example above the URI could change from http://www.foobar.com/api/customer to http://www.foobar.com/api/customer/consumer and http://www.foobar.com/api/customer/company.
Note that it remains a challenge to decide what the old URI will return.

The first and second change types are actually similar and should indicate a new version of an existing resource. When we encode the version in the URI, it makes most sense to do it after the resource indicator; http://www.foobar.com/api/<resource>/<version>. The problem here becomes immediately clear as it very much resembles the version in the previous scenario. Keeping these the same will be confusing. Alternatively, this would be a good candidate for the Accept-header option, keeping the URI the same and only use the accept-header for content negotiation. The resource stays exactly the same, so the identifier of the resource as well. The version of it's representation changes though, i.e. the indicator in the request depicting what format is accepted by the consumer, is explicit. Accept: application/vnd.<resource>.<version>+json.

There is a rather major drawback with this approach in that it becomes rather complicated to quickly test the solution, since from a browser, the http-header is almost never open for manipulation. So this solution doesn't allow for usage from the browser address-bar. This is typically solved by supporting both versioning strategies simultaneously. Which allows for two major advantages, besides support for programmable consumers and browsers;

1. The URI approach for versioning will return the relevant documentation of the API for that particular version, if that version exists or an http error when it doesn't.
2. The baseline URI can remain and stay usable. Access to the resource is through http://www.foobar.com/api/<resource> and the acceptable version by convention when none is specified in the header is the first version of the resource, or alternatively, the oldest supported version in case versions are at some point dropped.

By following the above, the intentions of services and therefore API's remain.

When you ask your best friend, e.g. Google, for an answer as to what is really the best way to handle different versions of the same interface, you'll find out that there is no straight answer. Actually it is a religious war and many of the  arguments given for either of the various ways of versioning interfaces are actually just opinions on aesthetics.
Above you're reading my reasoning for what my views are on the various options you'll have to version your service interfaces. Probably you'll find other ways of skinning this cat as well.
My advice to you is to closely look at your specific situation, what you want from your interfaces, the need for (backwards) compatibility and then choose wisely.

The disruptive nature of the cloud - The Fallacy of Availability through Reliability

So, as the title of this post suggests, I want to discuss the disruptive nature of the Cloud. This will be a series of 5 posts in total, you're reading the last post.

Read about the cloud and what it means and you're bound to read that the introduction of the Cloud, the real Cloud, the one that meets all criteria for being a Cloud, has been disruptive, but has it?.

I like the NIST definition of Cloud, it is quite comprehensive and less vendor-biased than Gartner's defintion.

The Cloud has been disruptive when it comes to the IT industry, especially the hosting market. But it has also been a force in how we handle IT within the enterprise. There are a few important aspects of IT in the enterprise that we consider to have changed due to the Cloud:

Moving from in-house IT management services to off-site IT management services. 

Moving from CAPEX (Capital Expenses) based IT investments to OPEX (Operating Expenses). 

Moving from on-premise (business) applications to off-premise (hosted) applications. 

Moving from a centralized IT to a democratized IT 

I'm sure you can think of other movements as well in your IT environment, but these are typically considered to be not only happening, but also to be disruptive within the enterprise.

These are in fact not really changes happening due to the Cloud, the Cloud merely gave these movements a boost and fast-tracked the changes in IT. The point in case, as you can read in above articles, is that the cloud hasn't been that disruptive at all. It's more or less all the same all over again.

But there's an actual disruptive nature to the Cloud, it's got everything to do with the many 9s you read about in brochures of Cloud providers. The amount of numbers of 9 relates to the stability or rather the availability of of something. It's the percentage of up-time of something, preferably a service, but in many cases it's just about a server.
The disruptive part is in that traditionally the availability of a service or even a server is depending on the stability of the service or server for that matter. And what is traditionally the case is that the availability of a service, an application, is actually depending on the stability of the infrastructure on which the application is running. The more reliable the infrastructure, the more available the application and therefore the service is.
As enterprises controlled the infrastructure, even in a hosted environment, applications were developed that relied on how the infrastructure was realized.
And that's where the disruptive part comes into play, because in the cloud, the enterprise no longer controls the infrastructure, and by no means one can depend on its reliability.

The bottom line here is that traditionally applications are designed-for-success. The application is relying on the fact that the infrastructure can be depended on and that in essence, the infrastructure will not fail. In the cloud this is not the case, applications need to be designed-for-failure.

Back to why this is the case. It's quite simple. Clouds are consisting of massive amounts of infrastructure. This is because the Cloud provider wants to achieve an economy of scale such that it can offer infrastructure and consequently services on that infrastructure for as many customers as possible for as low a price as possible. In order to do this, it makes more sense to use cheap hardware, so costs are low instead of expensive hardware. Because, well 100,000 servers of US$ 2,500 each is more expensive than those same 100,000 servers costing each US$800. The price per server for a customer can be lower in the latter case... you get the economics.
But as things go with cheap stuff, it breaks down all the time. But spare parts are cheap as well. The Cloud provider just needs to make sure that the fixed or new server is up an running again in no time. Oh, and when you run out of spare parts, you just buy other cheap kit.
With virtualization coming into play, a piece of hardware going belly up means hardly anything as the virtual server can be back up on another piece of cheap kit within literally minutes or even seconds.
Enterprises buy expensive hardware because they want reliable hardware as they're typically too small to get actual economies of scale so the whole Cloud paradigm doesn't work.

Here's the pretty part, when you control the infrastructure, you can determine what part of the availability you want to handle within the infrastructure and what's in the application. You have access to the complete OSI stack, so you decide where what is handled.
Now forget about being in control of the complete stack, you're only controlling layer 7, the application layer. Or when you're lucky, which you won't, you have something to say about the 6th layer, the presentation layer. All of a sudden, all the availability requirements will have to be handled in layer 7. That's right, in the application. Because all you know is that 99.999% of the time (probably more likely is 99.8% of the time) the infrastructure will do what you want it to do, but when it doesn't. You have no clue, you just know that it'll be back up in seconds. Probably not knowing what it was doing before it crashed. In fact, Cloud providers will not tell you how they reach the many 9s from their brochures, you can sue them when they don't come true on their promises.

What's so disruptive about this? Well, ever since businesses started using computers, applications and more importantly the design and programming models could stay the same going from one computing model to another. Always the paradigm has been design-for-success. Granted there was error handling, but this was handling errors that could be predicted. Wrong user input, key-uniqueness violations, race-conditions. But with the Cloud, all of a sudden, the paradigm must be design-for-failure. Assume it won't work and make the application robust enough to handle failure, handling errors that cannot be predicted. Unavailability of resources, session losses at inconvenient times, state of the infrastructure changing all the time without the application being aware of it.
See? The issue here is that applications can't be migrated from one computing model to another by just adapting the model when you go to the Cloud. All of a sudden, the application model will most likely have to change significantly. And with that, your breed of developers need to change as well. They have to re-learn how to program enterprise grade software. And no, just creating stateless horizontally scaling applications doesn't cut it. Because unless you design for failure, your application won't be horizontally scaling across the board. It will always have a bottleneck at the ESB (see my previous post on why the ESB doesn't scale, by design) or the database. In fact, the scaling capabilities of an application has nothing to do with the Cloud, it's a traditional problem that surfaced with the rapid increase of computing utilization. More users, more transactions, more requests, more of everything required different kinds of scaling than the traditional vertical (faster CPU, more memory, more bandwidth) kind. This is not related to the Cloud at all. In fact, the scaling solutions we all know is also heavily relying on the reliability of the infrastructure.

Concluding, the Cloud is disruptive. It's disruptive in that it needs redesign of the applications that are migrating from the on-premise to the Cloud.

Okay, there's another thing that's different from traditional computing models, and I hinted on that already. You have no clue as how things are done at the Cloud provider, you have no say about it, and the Cloud provider will never tell you. And that's something that a lot of enterprises have to get used at. You have to trust your computing resources vendor at face-value that you get what you're paying for, and you pay for the result and not for how it's done. And that's disruptive for a lot of managers and architects, especially because these are typically the control-freak kind of people.

NB: The Cloud needs economies of scale, most enterprise's IT environments are not big enough to reach these economies of scales, thus an on-premise private Cloud makes no sense from a cost perspective. This is not to say that your own Cloud is a no-go.

The reason why IT architects fail, almost consistently, is modeling.

Summary: Because every architect will try to fit her solution in her specific model, the way architectures are created within a project is independent from each other. Thus within a project the application architecture is not really considered when creating the infrastructure architecture, nor is the data architecture or the security architecture. Consequently, creating an architecture for any business application is a waste of time. Or is it?

Probably, when you're like me an architect or like some people I know a project manager, you already know that involving an architect in your project, or acting as an architect in a project guarantees exactly two things:

1. The project will not deliver on time
2. The project will not stay within budget

Of course I'm just kidding, in fact you should be experiencing the exact opposite, involving an architect in your project or acting as an architect on a project, provided the architect is considered to be a project member and not somebody that can get a project out of a dire situation, means that the project is most likely delivered on time or as close as possible to it and all within budget or close to the budget. This is what the architect does and there's more. The architect will ensure that the deliverable will be of a higher quality and therefore be of more use once taken into production. The promised or predicted ROI (Return of Investment) from the business case is more realistic.

The reason why the architect is helping with this is not because she's architecting the deliverable, although she is, that doesn't really help. The reason is that the architect is the single one person in the project that is looking at the problem(s) at hand from a holistic point of view. Approaching from different angles and addressing issues with an open mind. Architects have the talent to think outside of the box, creativity is their key capability.

Still, architects almost all fail all the time in doing architecture, that is IT architects. Because this blog is about IT architects. The key reason here is that they are architecting within the boundaries of their area of expertise.

So what do I mean by all this. It's actually quite simple. Consider yourself in case you're an architect or that person in your project that is the architect. What kind of architect is she? Most likely, we're dealing with an application architect, an infrastructure architect, a security architect, a data architect, a UI architect, an information architect, a business architect or maybe even an enterprise architect. Point here is that within pretty much any serious business application, you're touching all of the areas discussed above. But hardly ever do you see an architecture that is integrating all of these areas into one single model.

Architects are working a model and in the best of cases, are handing their model to other architects to make them aware. The project manager has no clue of what's going on or dares to make any assumptions as to what is going on here. But the case in point is that the application architect creates the application's architecture, hopefully taking the input from the business architect to understand where in the various business processes the application is being used. 

Once the application architecture is done, the security architect is tasked to make it secure. This is when the application architect is not suffering from a god complex and doing all that security design afterwards himself. Obviously the data architect has no clue as to what is going on, because he's never involved. The application architect did all that work instead. Because, data is an integral part of the application, so...

This is where the architect is done and the project's developers take over. Best case scenario, the resulting application is close to what the architecture intended it to be. At least the application will adhere to some of the enterprise's standards, values and principles. Once all that development is over, the infrastructure architect is requested to, oh wait, best case scenario, the application architect is reviewing the application for compliance with the architecture, and this is when the infrastructure architect is to create the infrastructure architecture on which the application is deployed.

Okay almost the same scenario, but now the story starts with the infrastructure architecture and the application architecture is subjected to the infrastructure architecture.

Now do the same with a security architecture to start with and ...

In all these scenario's the different architectures are created as separate products. One is subjected to the other. There's really absolutely nothing holistic about it. There's nothing integral to it either. And it makes no sense when you think of it.

I will not make an analogy with architects of buildings or other physical structures because there is hardly an analogy.

The issue at hand here is, as I stated, that the different architectures are considered separate products, with hardly any coherence between them. For some weird and actually inconceivable reason, we're let to believe that a security architecture can be created separate from an application architecture and infrastructure architecture and data architecture and business architecture. This is of course complete nonsense. There is no way you can create a generic infrastructure architecture, not even a high level reference architecture without understanding the application that will have to run on it. Same for a data architecture. What's the point of having that without knowing the application of the data in a business process? There is none. Don't let anybody fool you, because it's just not there.

An important reason why we are working this way, is because we are let to believe that we need to work with models. Decent architects will at least not try to fit the world into their models, but every architect will work with models. Which makes sense because this allows them, us, to simplify the problem's solution. Understand the model and you understand why the solution works. But have you ever tried to come up with a model that covers all of the architectures mentioned before? A single model? Have you tried to draw a diagram that shows it all? And in case you tried, did you succeed? It's not possible because the perspectives, the points of view, the points of interest are different for each of the architectures. They operate, they concern themselves on different levels of abstractions and with different levels of detail.

Because all these models consist of different elements, different symbols is used on the diagrams and the level of abstraction is seemingly different, the architectures are done separately, in a logical order to the project manager, the enterprise's culture and the individual relationships between the architect. Key point is that the consistency between architectures is coincidental.

Hence, the creation of any architecture in the end is a waste of time, because none will hold when the project manager actually wants to deliver on time and within budget. Because he will sacrifice security over deadlines when it costs too much implementing the infrastructure according to the architecture. The data architecture is thrown out the window as soon as the application is not performing because of the database queries. Etc.

There is a solution to all this and that is to create a truly integrated and holistic architecture of the complete solution. Have all these architects work on a single architecture that encompasses all these different aspects and have a single solution architecture that addresses application composition, data structures, business processes, infrastructure and security consistently.

This requires that dogma is thrown out the window and reference architectures are basically just informed decisions on architecture principles and guidelines with architecture blueprints emanating from them.

But most importantly, it requires a model that covers every single aspect of an application. And I mean every single aspect. The model must and can only be a meta model. A definition of a business application defined in data elements and their inter-relationships (including cardinality, direction of traversion etc) that allows for the definition of every single aspect of any business application. This meta model is a data model.

And there's the key to the issue, the model we're having to deal with in order to make an architecture worth creating, is a data model. The different architectures we're used to having are just different views on that data model. This allows for various levels of abstraction with various level of detail, as however of interest to the reader.

Because the model is capable of capturing every aspect of the business application, the model is capable of capturing every aspect of a set of business applications, hence it allows us to capture the complete application landscape and hence consistency across the board, architecturally speaking that is.

Now why is this so important, what is the significance of having a single model for all architectures to be based on? The significance is in the fact that all of a sudden an architecture cannot be subjected to another architecture, because there is only one architecture. Consistency and coherence are a given, it is impossible to not have a concise architecture that is comprehensive and all encompassing because that would mean that part(s) of the architecture are not finished. Which is fine because it will have to be a conscious decision to omit these parts.

In addition, for application developers it will be counter-productive not to stay with the architecture, because that will mean that deployment in production will be harder, because every single aspect of the application is taken care of in the architecture. Thus allowing the developer to concentrate on quality, and consequently an improved ROI.

I hope you enjoyed this blog post, I for one want to thank you for reading it. I am aware that this post was quite long, but I hope worth your while reading it. Please leave any comments, agreements or disagreements as the more views there are, the better sight we all have,

Iwan

The ESB addresses a niche, not mainstream

Other posts in this trilogie:

- ESB's don't scale, which shouldn't be a problem because that is by design
- The demise of the ESB in a world of webservices

There are many reasons not to have an ESB in your environment. The ESB as we know it has less and less a reason to be implemented in today's world of webservices. There's a full post based on the dimise of the ESB in a world full of webservices.
But this post is about why you should have an ESB and why it does make sense to implement it. Or rather, when it makes sense to implement it.

This is a post in a series about the Enterprise Service Bus, ESB, an acronym so easily to love or to hate. Parts of this post are also found in another post on the dimise of the ESB in a world of webservices.

Remember what the ESB was good for?

1. Data Exchange
2. Data Transformation
3. Data Routing

Well with SOAP being XML based and arguably very wel defined, the Data Transformation is not that relevant anymore. The Data Exchange with SOAP is handled as well, you use HTTP(s) in pretty much all the cases you come across it. Leaves us the Data Routing.
So what was the routing about? Well, it was all about having a logical name resolve to a physical location, where the ESB would determine where to send the message based on an address that one can remember, or that is not tied to a physical location. In the world of HTTP(s), which is dominated by the world of TCP/IP, this is actually handled by the DNS (Domain Naming Service). The DNS can perfectly handle this translation and do the routing. It's what is was designed for. Almost its raison d'etre. Maybe not even almost.
So in a world where we do SOAP based communications, there really is no reason to have an ESB.
What do we keep the ESB around for?

You keep the ESB around for the many-to-many messaging. You keep it around for those cases where you have one or more sender of (similar) messages and one or more recipients of these messages. Because the ESB is a great message dispatcher, it's the message routing part that kicks in. It's something the DNS can't do. In those cases where you want to send the same message to an unspecified amount of recipients where the recipients can register or deregister to receive messages at their leisure you'll want the ESB because managing this at the client side, maintaining a list of recipients and sending each message to each of the registered recipients is bad for performance, susceptible to errors and complicated.

The ESB is great at dispatching the same message to multiple recipients, doing transformations of the data on the fly. So duplicating a production load of messages into a test environment through the ESB, where the ESB is transforming the data just in time before sending it to the test systems (e.g. anonymizing the data, translating from one spoken language to another, transposing id's such that they align with keys in a database etc) is a common practice.

I have worked at a financial institution where we fenced the core banking systems with an ESB, catching every transaction send to the corebanking system and duplicating it to a standby system as well as a test system in a slightly altered form to be able at one side to have resilience at an extremely low price and production load with relevant data in a test environment.

Basically this means that you should keep the ESB around for being able to follow the concept of publish/subscribe. In my opinion the only right concept when it comes to asynchronous communciations. And since this can only be done by either using lower level network protocols like UDP (TIBCO's TIB/Rendezvous was built on UDP and the first real pub/sub message bus), or by having a central dispatcher sending a message to the registered recipients, or by handling it all at the sender side it makes most sense from a security as well as a manageability perspective to do publish/subscribe through a centralized dispatcher. Although for the really lean and mean high performance situations, you want to resolve to lower level network implementations.

Understanding pub/sub and implementing it in the enterprise allows for maximumenablement of the business with minimal IT investments. The afore mentioned use-cases of the monitoring systems where all systems publish monitoring messages (on business level) and various systems subscribe to them, for example a dashboard, and you'll get a very rich BAM (Business Activity Monitoring) solution, which without much additional effort can be used as an application level monitoring solution as well.
The other use-case where production feeds were used to implement improved availability as well as the capability of testing a new release with relevant production loads is of course pretty awesome as it didn't need a lot of additional work once the ESB fence was set up.

So dispite popular believe, there is a reason why you should consider implementing an ESB in your enterprise. But don't adopt it as the default means for communications between applications, instead apply the ESB as a niche product and not as mainstream.

As always, I'm really interested in your views and ideas. More perspectives make for a richer understanding of the topic. So please share your thoughts, stay respectful and be argumentative.

Iwan

Flexibility in the 21st century: Agility

Hi,

I've been a developer for pretty much ever since I got introduced to computers. First I developed for my Sinclair ZX81, then I moved on to MSX, made the transition to DOS in 1989 and moved along with Microsoft to Windows. I've developed on OS2 and various flavors of Unix and I've been using a multitude of programming languages ranging from BASIC to Modula2 and Pascal to C, C++ and Java.

I've been to university studying Computer Science in the 20th century and back then we were thought to design and develop for flexibility. The reason behind it has always been that you should be prepared for the fact that once you're done and you've delivered there will be new releases, new requirements, changed requirements, etc. Possibly even while you were working on the system you would be confronted with new insights, new ideas, new and more important requirements. Re-use was all the rage and you should prepare yourself for changes by being able to re-use 80% of the code, of the features, of the functionality already created and only add an additional 20% of design, code, testing and documentation for the new stuff.
Even in the wonderful world of OO, CBD and Modularity, the beat of the drum was about being flexible. Create a flexible solution, one that can bend and twist all over the place to accommodate for every change in the requirements. Be it functional or non functional. The whole idea was and with many of us currently still is, that when you design or develop something it should be so flexible that a new requirement would be already implemented by 80% of the existing code.

I remember having reviewed a system that was almost entirely written in database tables. An engine would simply read a set of tables and an object model was instantiated and executed. Which is not all that unfamiliar when it concerns for example workflow engines or rules engines. But this little piece of software concerned an actual programming language in database tables. I could write a simple "Hello World" application in the database. The reasoning behind this little piece of ingenuity was that there wouldn't be any deployment stress with testing of the software etc once the application was deployed. Just updating the tables would allow for new functionality. Of course until after the first update of the tables wrecked the system because there was no syntactical or semantical checking of the 'new program'.

Flexibility is something that needs to be provided by frameworks. This is why you would create frameworks. But the thing with frameworks is, is that it is extremely hard to actually write a framework that is actually usable as a frameworks. They tend to be too restricted to a specific purpose or too generic. Most of the popular frameworks started out as a specific solution that was repeatedly implemented, based on a well known enterprise design pattern. After the 3rd time of implementing the pattern the designer or developer recognizes the parts that are consistently changing and those that remain and he'll be able to start developing the framework. This even holds true for design-time frameworks.

With the advent of eXtreme Programming and Agile project management (any flavor) the world of software development also got an overhaul. No more lengthy analysis and design phases prior to development and instead short cycles to get the product out. In the old days (which in IT means 'not that long ago') analysis and design was considered extremely important because one wouldn't want to miss a requirements, be able to prepare for the next version, build in hooks and nooks to facilitate additions without corrupting the overall architecture. But the longer the time between conceiving an idea and realizing, the further apart the idea and the realization of the idea are. Hence the short cycles of today, which still have an analysis and design phase, but shorter. And no, one can't do without analysis, design in agile or even without architecture, but that in another post.

Now, what is so dramatically different in terms of flexibility in the 21st century compared to the previous millennium? Well, the difference is in mindset of the developer. The developer and even the designer creates for replacement not for changing. By this I mean that when designing or developing nowadays, one needs to keep in mind that pieces can be ripped out and replaced by something completely else. Without heart feelings. And yes, letting go of your baby while still in its infancy is a change in mindset. Today you create something that is not supposed to last forever, but something that lasts until the next big thing. No more stone carved COBOL systems that will survive humanity.

The big difference here is that every application is a business functional framework, one that targets a specific area of business functionality but caters only for that functionality that is required. New functionality is added in a consistent manner, existing functionality is removed when no longer valid or replaced by a new piece of code when required to be changed. Mind that the key here is 'replaced'. Code is not amended to implement the change. Because it is way to cumbersome to change the code, it is better, faster and more convenient to build the new functionality from scratch and add it to this business framework. This is flexibility through agility allows IT to build business applications that are consistently 'correct' in design and easy to support and maintain.
The whole premise of agile development is to scope an iteration purely to those requirements that are immediately necessary and developable within the constraints of the iteration (timelines, budget, resourcing, etc), while not taking (too much) into consideration what might be required in the next iteration. Which basically means that whatever you create may be thrown away in the next iteration because it conflicts with the requirements of that next iteration.
The idea that in agile development there is no place for analysis and design and certainly not architecture that many agile adepts have embraced is of course a consequence of architects, designers and analysts not being able to analyze, design and architect keeping in mind that every single component in the solution should be throwawayable. They're quite often still create flexible systems instead of agile systems, hence the conflicts with the developers.

Agile systems are designed by keeping in mind that the components in the systems are decoupled, on a functional level. On a requirement level that is, which boils down to the fact that the analyst needs to define the requirements and their impact on the technology such that the requirements themselves are decoupled from each other. Dependencies between requirements result in tightly coupled implementations. When two requirements depend on each other, they should be treated as one. Consequently the designer should design the solution for each requirement independent of the other requirement. Re-use of functionality already designed for one requirement to be met should not be in the design... or at least not in the first iteration of the design. Preferably, re-use should only be taken into account in the technical design of the solution if at all in the design. Why not leave this decision to the developers?
This will allow for the assumption that what is designed today will not be there tomorrow.

So where does this leave the architect. It is clear that every iteration will have its analysis and its design phase. You cannot get around it. It is the rule of problem solving: understand the problem (analysis), think of what the solution(s) might be (design), implement the solution that is most feasible if feasible at all (development). You just cannot solve a problem you don't understand.
But where is the architect in all this. Surely you can't architect a system for which the requirements are not clear at all, they're made up and prioritized as the project goes. Or rather as time moves on. The architect defines the architectural principles that the solution with every iterations needs to comply to, to safeguard the consistency of the solution, the project's deliverables, in the grand scheme of all systems and solutions in the environment. The architect will define that at all times user interface and business logic need to be separated in design and in implementation to run on different (logical) servers. The architect will define that users of the system that are external to the company need to explicitly be authenticated using a secure token, where as users that are accessing the system through the local LAN can be authenticated using userid/password only. The architect will define that the allowed technologies for implementing the solution should be platform agnostic, allowing the solution to be deployed on Windows, Unix, Linux and OSX systems. And the architect will verify compliancy with the architectural principles prior to deployment in production.
The architect is or heads the design authority.

Without architectural principles that guarantee the integrity and consistency of the IT landscape yet support or at least allow agility in this same landscape, agile methodologies will create chaos and therefore significantly increase the TCO of the landscape.

As always, I welcome any comments. Whether you agree or disagree or just find this post informative. Please let me know where you think I am wrong or right and moreover why you think this. It will be very educational for me as well as the other readers of this post.

Thank you for reading this far.

Iwan