Principles of the API-First paradigm

API's are weird puppies. They've been around since forever and whole battles have been fought in courts to gain access to API's. This was in the old days when having a platform with an API for 3rd parties to develop for your platform meant that you had a lot of leverage over those parties. Not providing early access to them meant that they would be late to the party and all the sweets where already shared between the early guests. Having an API meant having power. Not providing access to your API resulted in lawsuits because you were a monopolist, using your position as the platform owner to bully your competition into obliteration.

Nowadays the situation is completely different. Having an API means having a future. Nowadays to survive as a business you need to have a platform, which in turn means you need to have an API. Unless you have an API, one that suits the needs of 3rd parties and that is easy to use, you're doomed. You'll be just an Application Vendor, and toast before you can say "I wish I had built a platform". An excellent book on the topic is "The Age of the Platform" by Phil Simon.

As you can read in my previous post, API's are the hardest thing to develop. It always helps to have some guidance when you're up for a difficult task, and often principles help you getting this guidance. There're quite a few other guidelines out there on the interweb that are more than helpful. Below are 5 principles of the API-First paradigm. Followed by 5 derived principles and at the end of the post, I'm listing a couple of more than interesting resources on API design.

One final note before I go into the principles. An API is not much more than an interface into the depths of your platform. It discloses your platform and makes it possible for others to leverage it for their needs. It is therefore critical to have comprehensive documentation for your API. Now comprehensive doesn't mean that you need a big fat document describing everything you can think of that might be remotely related to your API. Typically you will need to document the technical interface of the API, i.e. the call signature of the API, including the input and output parameters of the API.

In addition you will want to document it's behavior exhaustively, i.e. only develop the behavior your documentation mentions. It makes perfect sense to use BDD (Behavior Driven Development, see my post on the topic) for this purpose. By doing so, your users know exactly what to expect from the API and it allows them to create stubs that behave the same as your API without the need to have access to your API's implementation. Finally it helps to have a simple example on how to use the API without making any assumptions as to why one would use the API. This could be through showing the relevant cURL statements.

Enjoy your reading…

Principles of the API-First paradigm

The API is the product.

It is owned by a product owner, and the responsibility of a product team. A team is responsible for it, from cradle to grave. This team is responsible for managing and ensuring its proper operation. Don't consider an API as part of another product, something that's a by-product, because it means that you won't treat it with the proper dignity and soon you'll find yourself alienating from your users.

The API makes no assumptions about its Consumers.

It is robust and resilient enough to thwart of any consumer with bad intentions as well as ignorant ones that have no clue about what they're doing. All this without compromising the service level to all well intending consumers. Mind that in most cases you're working on an API while making assumption as to who and why they'll use your API, only to find yourself in a situation where your users are all but the ones you envisioned, and all using your API for completely different reasons than why the API was created in the first place.

The API is suiting the needs of the Consumer

The API is developed only when there is at least one consumer and it is tailored to suit this consumer, by this the API is creating business value through its consumer. Don't be so arrogant to develop something that's supposed to be an API without having at least one consumer lined up to actually use it and tell you how great it is. Apart from the fact that working on something that is not being used is a complete waste of time, it also means that you're not really understanding the fact that you should not make any assumptions about the API's consumers. No assumptions at all.

The API is built to last

An API is never breaking compatibility with its consumers, although it may rely on the paradigm of 'Ignore what is unknown' to allow for extending the result. API's, other than the humans that build them, never break a contract. Period! This is to say that they will respect contracts, will have addendums to the existing contract, or in those cases when the API, which is after all a product, is no longer adding value, they will void the contract, but obviously respecting the contractual notice period. On a side note, it is fair to assume that its consumers abide by the convention that they'll ignore what they don't understand or know, hence the fact that you can add addendums to existing contracts without actually breaking the contract.

The API is idempotent.

At all times, in any situation, always. It doesn't matter what you do, how you do it or when you do it, when you call an API it always behaves exactly the same. Meaning that its result is always the same and therefore perfectly predictable. The order in which API's are called doesn't matter, as long as you put in the same values, you will get the same results.

Architecture principles for the API

The API is clear in its error reporting

Errors are either caused by the API implementation, the API consumer or something unexpected. It makes perfect sense to stick with the conventions of HTTP errors, why invent something new, when the existing is fitting your purposes perfectly. But whatever you do, make a clear distinction between these three causes for errors.

The API is always managed.

An API can be managed independently from;
Technical level, i.e. it can be accessed securely and consistently throughout its lifespan -> Scalability, Availability, etc

Application level, i.e. consumers can use the API based on a published interface that is behaving per this interface -> Interfaces, authentication/authorization, etc.

Functional level, i.e. users of the consumer can be provided access to the API based on the agreements made -> Throttling, metering, monitoring and reporting, etc.

The API can be implemented independently of its interface;

They are truly decoupled. Multiple implementations can co-exist and determining which implementation to choose for each individual call can be context driven. Mind that the interface should be technology agnostic, or at least as much as possible, and in no way should the interface disclose anything about the implementation other than that it should explain how it behaves.

The API is technology agnostic.

The interface can be called by a consumer of any technology without any assumptions of the technology used in the API implementation, and vice versa. This is pretty much an extension of the previous principle, but important enough to mention explicitly.

API's are like dogfood.

The API is used by the team responsible for it in the same use cases as the consumers for which the API was intended; "Eat your own dogfood" paradigm. Always make sure that in case you need functionality that one of your API's is providing, you're calling that API yourself and making yourself dependent on the API. Only this way you understand your API, and more importantly you'll be very cautious about how you treat your API and therefore how you treat its users. After all, you'll be one of them.

Interesting Reading

• Google's API design guidelines
• 18F API guidelines
• W3C Web API Design Cookbook
• CISCO API design guidelines
• Haufe API style guide
• Heroku API guidelines
• Paypal API guidelines

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contactlist.But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

API Management in Azure using Microsoft's solution - Resources and Templates [2/2]

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

This post is the last post in of a series on API Management using Microsoft's solution.

API Management is not the same as an API Manager. In short, API Management is what you need to do with your API's, and an API Manager is something with which you can do this. There are a lot of different solutions that allow you to implement API Management, one of these is Microsoft's solution on Azure. The tricky part is that when you start reading their documentation on API Management you'll read little about how to actually implement API Management in your organization using their solution. It's all very conceptual. This shouldn't be a problem since the concept behind API Management are more important than the actual implementation… until you want to actually implement it.

Read the previous posts on the topic to learn more about the concept, continue reading this post to learn more about how to use Microsoft's solution in Azure, their cloud platform.

Resources and Templates

Finally, resources and templates, the bricks and mortar of the cloud. In the cloud you're typically dealing with resources. An infinite amount of resources, or at least that's how it feels. Everything in the cloud is, or should be a resource. Some are very basic like computing power, storage and networking. Others are more comprehensive like database, firewalls and message queues. And then there are a quite a few that are truly complex and very useful on a high level, for example directory services.

The cloud, being what it is, is like a normal infrastructure on which you host something that generates value for your business. Hence everything you need to run business applications in a traditional hosting environment, you also need in a cloud environment. Obviously there are significant differences between traditional hosting platforms and the cloud, but when you don't take too close a look, you're likely not to see these differences.

So in the cloud you also need to define systems, attach storage, put a firewall in front, put connectivity in place etc. You can do this by hand every time you need an application's infrastructure. Typically through a portal by clicking your way around and assemble the infrastructure. But more sophisticated and a way better practice, is to define the infrastructure in a text file, typically JSON for most cloud platforms, and use the cloud vendor's tooling to create the infrastructure based on this file. As such, the file becomes a template for a specific infrastructure setup you need. By providing a parameter-file you can externalize specifics of the infrastructure. For example the URL's to locate a web-service can be defined in this parameter-file to distinguish between an infrastructure intended for testing and the same infrastructure intended for production runs.

The particular template is called a resource template, it defines which resources are needed and how are they specified in order to run a business application.

One of these resources that you can use is an API manager, just like you can specify databases and virtual machines as resources. And here's your challenge.

The challenge is in that an API Manager consists of three parts;

1. Developer portal, used by developers to find your API's and their documentation.
2. Publisher portal, used by API developers and the likes to manage the API's.
3. Gateway, used by applications developed by those mentioned above in 1 to access API's managed by those as mentioned above in 2.

Each of these have their own context and is used by a different group of 'users'. The real interesting part of the API Manager is the API Gateway as it is the component that exposes the API's you've been developing. This is your product. It is the resource that is part of your software. And the thing is; it can be shared or limited in scope to just the product you're developing.

Ideally you would have one gateway per product, because the gateway and particularly the API's it exposes, are part of your product and as your product evolves, the API's that come with it will evolve as well. And of course you would want a consistent life cycle across all components that relate to your product. Since the API gateway is just like any other resource in Azure, the above is perfectly doable. In fact, it is possible to include the API gateway as part of your product's resource template and when you provision the relevant infrastructure and deploy your product on it, the API gateway is provisioned as well.
Pretty awesome, when you're willing to forget that the costs of an API gateway are pretty steep. We're talking about close to €2,5k / month. There's not really a price based on usage. Microsoft is really weird in that when it comes to pricing in the cloud. That whole pay-per-use is not really everywhere in their pricing schemes. I like Amazon better in that regard.

So an API gateway per product is not really an option in most cases I would argue. Instead, I would advise to have a gateway per product suite. In case you have teams that handle multiple products, scope the gateway to such teams, or otherwise scope the gateways to department. Use it as a rule of thumb though and not as the law.

The point here is that you want to be able to have your API's evolve with your products and that you want teams to be as independent of each other as possible. But in addition you want your API's to be operated independent of each other. And this is important. In Azure, API's don't scale, it's the gateway that scales. And you want to be able to be smart about that. Especially when it comes to policies and usage tracking or rather generating value from API's being used. When a team is responsible for the success of its products and therefore the value that is being generated, it becomes obvious that that team would want to be able in control of what is affecting their success.

The alternative would be to work with an SRE approach, where you have a team that's responsible for your platform, including your cloud platform(s). This team would then realize the API gateway for the other teams as a service. The catch here is that this platform team decides where your API's are 'hosted', or rather whether or not you share the API gateway between teams or not. Unless your platform team is really committed and more importantly has a thorough understanding of what API's really are and I mean really understand this, I would advice against this approach. I would oppose it for the sole reason that your API's are the window into the soul of your organization. When the API is not performing well, your organization is not performing well. And especially when you're going API first, and thus build a platform, you're screwed without proper API management.

In case you do decide to go the platform team route, make sure that your processes are completely automated. Including the deployment of new API's as well as new versions of existing API's. My preposition here is that you'll be working agile as can be, deploy to production as soon as you're confident that your software is up to it. Meaning that new software needs most likely new (versions of) API's. Don't make the platform team a bottleneck, so make sure that you're working with them to deploy the changes API's consistently, repeatable and consistently. Better to abide by their rules then put your own in place. Drop the whole platform team approach when they're not providing a 100% automated deployment process for your API's.

Then there's the portals. The developer portal is a tricky one because it provides access to your API's from a developer perspective. You should be really nervous when you're nervous about potential unwanted developers nosing into your API registry. Because it means your security is way, way, way below par. Remember, API's are different from regular services in that they are build such that they make no assumptions as to who accesses them. And unless you've build them that way, you'll be in for some really serious security challenges. That said, there's no reason why not to have different portals for developers within your organisation and developers from outside your organisation. And have API's exposed only to internal teams and publicly exposed API's. Just make sure that this is an exposure aspect and not, I repeat, not an API implementation aspect.

So develop API's as if they're possible accessed by just anybody and their mother. Expose API's to a subset of this group.

Then there's the operational aspect of your API captured in the publisher portal. Here you should take an approach that only the team responsible for an API should have access to the API from a management perspective in an operational environment. In other words, access to an API's policies is for the team that 'owns' the API only. You'll need to take care of that. period.

Mind that Microsoft is rapidly changing their API Management service on Azure. Most likely. as I type this, they're making life easier for you on using the service in your organization. The concepts as I've described still hold though. And as Microsoft will hopefully come to realize that also for API Management a pay-per-use model is the way to go, you'll be able to treat API Management as part of your product instead of as part of your platform.

This concludes my series on API Management using Microsoft's solution. I hope you did enjoy it.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

API Management in Azure using Microsoft's solution - Resources and Templates [1/2]

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

Applications

Applications are pieces of software that when combined they perform a business function. Typically, applications perform various business functions, but ideally each application provides access to a single business function. If you think of it this way, most of the software we traditionally know and see in our IT landscapes, what we call applications, actually has multiple applications. You can use this software in a variety of ways in a variety of situations to perform a variety of business functions. So the name 'Application' is incorrect but understandable.

As always, the cause of this misnomer can be found in history. Deploying software used to be a complex thing in the past as was software development and even more so developing software that communicates with other software. A ton of different concepts, protocols and technologies have emerged over time to facilitate the connectivity between different pieces of software. (Skip the next section when you're not interested in a short but incomplete overview of software interoperability)

Software Interoperability

Microsoft released (D)COM in the past followed by ActiveX. The rest of the world joined into the fray with the incompatible CORBA standard(s), there was RMI for the Java world and RPC for the C(++) world. Then we had other solutions based on message exchange in the form of MOM (Message Oriented Middleware) which was, more, protocol independent. Although MOM as a concept was awesome, the big vendors succeeded in ensuring that their products were not interoperable. And because most of the vendors realised their customers didn't really appreciate this, the ESB was invented. Again, it was there to make sure that pieces of software could interoperate and fix the complexity of our monoliths. ESB's, as we all know, are a big waste of time and money and by far deliver on their promises. I did write a post on the topic, which you can read by clicking here and I'll write a little update on the topic soon as many ESB aficionado's don't see this yet.

So after we started omitting ESB's from our IT landscapes but kept the ESB concept around, we finally found ourselves in a world that actually delivered on promises and was low cost; The Internet.

Web-services

You can skip this part if you already know everything about web-services and are not interested in what I have to say about them. It's relevant, but not such that you can't skip over this section.

So web-services, both SOAP and REST-based, are interesting little beasts in our IT landscape. Web-services are the first real useful concept when realising an SOA. A Service Oriented Architecture. Based on Internet technology and therefore implicitly complying with all the requirements of an SOA.
And then there's the fact that web-services are accessed through a simple interface, the URL. So even in case you don't think about it, you're forced to limit the scope of what a web-service can do because you're limited to a URL. And although behind many URL's there can be a single monolithic structure, or you can use all kinds of different ways of encoding many functions within one URL, there really is no point in doing so, because working with URL's makes it harder to do things the wrong way than to do it the right way.
And the cool part is that this is true for even high level business functions that require a variety of (technical) low level web-services. On every level, working with web-services (almost) requires you to develop software that do only a single (business) function. As such, a web-service can be considered an application.

Web-services make it harder to do it wrong than to do it right.

If you look at it this way, a web-service being an application, you can see that a web-service is a small piece of software that provides just enough functionality for another piece of software or a person, to consider useful.
In effect; Every web-service, by means of its interface exposed as a URL, is exposing a (business) function. And as soon as you aggregate various web-services into a new web-service, meaning you call several web-services in a particular order to implement a high level business function, your new web-service complies to the same rules.

When we look at API's, they're the interface that we put in front of some software to expose specific business functionality. Look at the previous post on the topic by clicking here.

API's are products. There's an intended full stop because in many ways people tend not to think that way. But really, an API is where the boundary of a system is. The rest of the world accesses that software system through the API. The rest of the world might be a user interface.
So if you think about API's being products, or possibly a set of API's being a product, you're on the right track, because you then understand that a web-service is not a product in itself, instead it is a piece in a product's puzzle.
Why this is important? Because the API developer has the responsibility towards the consumers of the API, ensuring that changes to the API do not impact existing consumers of the API and lure new consumers at the same time. A developer of a web-service does not have this responsibility. or rather there's a way out.

Remember that in a previous post I mentioned that an API does not make any assumptions about it's consumers. They're not known in advance and any assumed consumer is likely not the biggest fan. Web-services on the other hand do know their consumers. Or at least, they can be developed such that assumptions are made regarding the consumer. This is also the reason why there's a need for API management in some cases and no need at all in other cases.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.


Arc-E-Tect

The Arc-E-Tect's Predictions for 2017 - API's and Webservices [2/10]

The Arc-E-Tect's Prediction on API's

It's 2017, meaning 2016 is a done deal, and most of my predictions for 2016 (I predicted them about a year ago, but never got around documenting them in any form) have proven to be absolutely bogus. Which leads me to come up with my predictions for 2017 and properly document them. So continue your effort reading this post and hopefully you'll enjoy reading them. Unfortunately we'll have to wait for about a year to find out to what extent I got it all right, but for now, ... API's!

Why API's? Well, API's are all the rage and everybody and their mother is working on platforms, and as we all know; Without API's there is no platform.

API's in, Webservices out

Okay, in 2017 we'll feel ashamed when we talk about web-services and SOA. Instead we'll talk about API's. This is closely related to my first prediction on Microservices, which you can read here.

API's are basically another word for webservice to many people. So there's not much different here. Just as with the Microservices, we'll see mentioning of API's more and more where in the past we talked about webservices. Until people are referring to platforms, something that is really picking traction. Still I'm not talking about platforms, but about API's instead.

The reason for this is their strong relationship with Microservices. Delivery of a platform is a strategic decision that defines the direction in which an organisation is thinking about products. API's, which expose the functionality of a platform, are products. You can read all about this in my series on API Management on Azure, which you can find here. Other than web-services, which are pieces of functionality and/or data exposed via a well defined interface, API's are always targeted at an external consumer of the service. In other words, an API never knows who's calling nor does it make any assumptions about who's calling. Web-services on the other hand might very well be limited to a known set of consumers and make assumptions about their consumers.

The promise of web-services, predominantly a decoupling between functionalities in an IT landscape, is limited to those case where the web-service, or rather it's interface is treated as an API. API's, almost by definition, are bound to deliver on the promise of decoupling. When developed properly, taken care of and fronting independent pieces of software, API's are the closest thing to silver bullets in software we've come across so far. And since we love silver bullets, API's are in, web-services turned out to be just plain old regular bullets, so they're out.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

API Management in Azure using Microsoft's solution - Why API's should be managed and by whom

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

I hope you've read my previous post on the topic of API Management, if not I strongly advise you to click here and read it.

Summary

Although managing API's has some resemblence with Looijen's model for IT management, the typical segregation between business and IT that stems from Looijen's model is disastrous for API management. API's need to be managed by a single team, the Product Team. Any other way of managing an API doesn't make sense. API's are products and should be treated that way.

API Management

API's are interfaces to (business) functionality that are accessed by, in theory, just about anybody. As can be read in my previous post, the idea of having API's and one of the main reasons why developing them is so hard, is the fact that you don't know up front who is going to call the API. The point in fact is that in a world of API's there is no room for the arrogance of the developer knowing exactly who is going to use the software and why. Instead, API's are inherently developed to support the case that the developer was wrong.

Because of this, when you expose an API, you'll need to make sure that you can support the API once it is deployed and used by somebody. Supporting means that you need to make sure the API is running consistently over time, the API must be usable and stay usable over time. Hence the API must be managed.
In order to do so, API's need to be managed on three different levels that to some extent relate to the support levels documented by Maarten Looijen, of which a description in Dutch can be found here. Basically he defines Functional, Application and System Management

The Looijen Model

(Skip this section in case you're not interested in the different kinds of support types.)

The model is not shocking as it states that there're three types of support that is required to support an IT system:

1. Functional Management. [Original text translated by Google:] Functional management is the management form that includes all administrative tasks that are required in the context of the use of information systems. As usage focuses on functionality, such as entering, handling, procurement, transportation and storage of data refers to information management.
2. Application Management. [Original text translated by Google:] Application management involves developing and maintaining applications. In the words of Parts and Looijen: "There may be many situations in which changes should be made in the original application software or database structures (...) Whenever such a situation occurs, application maintenance is performed but also.. when there is no direct reason is to perform application-maintenance, it is because of the continuity may still be required in order to maintain a complete management shape surrounding this executive task area application maintenance, including management and executive tasks. " Application management is like form, as it were about the task area applicate maintenance built around. It is to recognize application is essential that once developed information not evident also delivers the tools to easily support the change process (eg. Documentation). In addition, persons other than the original developers may be involved in the execution of modifications. Therefore it is important to maintain conditions which an application must meet so that they can be maintained 'easy'.
3. Systems Management. [Original text translated by Google:] Technical management includes the management form all tasks necessary to install, accept and operationalize and maintain information systems and technical infrastructures. Under technical management also includes optimizing processes and making changes in the technical infrastructure as a result of errors, extension or replacement. The technical management is focused on the technical platform, consisting of equipment associated with basic software, and implementation of information systems built on it.

Management of API's the Looijen way

Now that you understand what I mean by the Looijen model, it's time to inform you how API's should be managed.

API's are to be managed on the same levels as other software, that is the functional, the application and the system level.

System Level: On the system level, API's need to be managed such that they are made available. At the very minimum there should be an error stating the API cannot be accessed. At system level you manage the infrastructure on which the API is running. This might include some middleware as well. Since API's are to be called a lot, system management also covers scaling, i.e. elasticity, as well as all kinds of resilience aspects. Also security aspects like DDOS prevention, (web application) firewalls, etc. are part of the system management aspects of API management.

Application Level: On the application level, API's need to be managed such that they can be accessed, that the API documentation, (how to call the API?), is available, etc. Basically what it means is that there is an API to be called and that the technical aspects of the API itself are managed. For example the fact that an API accepts both JSON and XML parameters, that there is an API specific certificate defined such that the caller can verify the API call is served by a trusted system etc.

Functional Level: On the functional level, API's need to be managed such that they provide some value to somebody, preferably to both the consumer of the API as the team behind the API. API's are excellent for creating win-win situations, and these are business relevant situations. Basically what managing API's on a functional level means, is that you manage the relationship between the API and the consumer of the API. Think about a consumer having to pay for access to the API and setting different thresholds to control this.

When you look at the above different levels of API management, you may have noticed that on a system and an application level, managing the API is rather technical, but on a functional level it's really a business concern. Switching back to Looijen's model and you see that he makes a clear distinction between the two as well. System and application management are technical concerns, functional management is a business concern.

What if API's are not managed?

So what if you don't manage API's? The answer is actually rather simple. It becomes irrelevant, and in the end will seize to exist. For one thing, an API needs to be kept available. When it's not available, no consumer can access it. So on a system level it needs to be managed. This is no different from any other software application. Not managing an API on a system level results in the same chaos as not managing any other software, with the side note that API's are called by any(kind of) consumer, so the 'blast radius' cannot be predicted. From this, there's no reason not to manage the API on a system level as any software needs to be managed. There's one other point to be made here, and that is that an API needs to be kept available, meaning that it needs to scale, preferably horizontally and it needs to be resilient. Both are aspects that are not traditionally system management aspects as with API's this needs to be addressed on a software level.

On an application level things are getting a little convoluted with the system level management. In fact you might already have gotten the opinion that system level management is very much like application management and in fact you're correct. But system level management is more of a 'keep the shop open' issue, and application level management is more of a 'retain customers and grow your market' issue. When customer are not attended to, they'll go elsewhere. API's need to be managed on an application level as they are called because the consumer wants to. When the consumer needs to, eventually he'll leave for a more convenient API that provides sufficient added value to migrate to it. Hence the API needs to be amended to support other protocols, other data formats, more security etc. All aspects of application level management. See this as a matter of keeping the interface of the API consistent, but the implementation behind it evolving. So the API might be providing a consistent end-point to a consumer but evolves from a low-cost inefficient implementation to a costly but efficient implementation. This is API management on application level; replacing implementations behind an interface. If an API is not managed like this, it will become obsolete at some point. Obsolete API's are useless but cost a lot.

The system level and application level management concerns of an API are clearly required in case an API to some system or information is provided. When the API is not managed on both levels, the API will become irrelevant. But what about the functional level?

Functional management of an API is addressing an API on a business level. Read the post on API first design here to understand why API development is costly and if not anything else, the most difficult thing you can do in software development. Because it's so tricky, it's important that there's a business reason to actually do API development in the first place. And this is where functional management of an API comes to the stage. This is where it is managed that an API is monitored on a business level, for example keeping track of how many different consumers are using the API, how many of the consumers are 'returning customers' etc. Functional management of an API is all about managing the metrics about consumers of the API and act on them. Consumers that do a lot of calls may be willing to pay for each call to the API, consumers from specific industries will be more willing to sacrifice some performance in order to gain security etc.

When API's are not managed on a functional level, there is no point in developing an API. API development is so complex that there needs to be a business justification to do so. And it needs to be validated that the initial justification which is always based on assumptions, is correct. Not managing API's on a functional level will ensure that funding in the broadest sense of the word, will stop eventually.

Who manages API's?

API's are managed by both business and IT. Refer to Looijen again and you see that for software solutions, managing these solutions requires both IT and business. The same goes for API's. In a traditional organisation, one that has a clear separation between business on one side and IT on the other, API's are technically managed by IT and the developer of the API needs to provide an interface to the business to manage the API on a functional level.

Further complicating the scene, refer to organisations where IT is divided into a department that is more concerned about applications and the other department more about infrastructure. In this situation system level management is done by the infrastructure people and application level management is done by the application people.

Followers of the Looijen model are typically of the opinion that Application Support can be done by system managers. The reasoning often is that system managers are technically skilled people that understand applications as they understand the infrastructure on which the applications are running. And since they manage the technical aspects of the application and not the functional parts, they're more than suitable to manage the application as well as the underlying infrastructure. This view is incorrect (click here to read why) and when it comes to API's it's disastrous.

When you have an API based platform, and there really is no other way to have a platform, you need to consider the software implementing the API as well as the infrastructure on which it is running as a single system. It makes hardly any sense to see them as two separate pieces of the API puzzle. The reasoning here is that scaling, resilience and other aspects of the system level management of the API, i.e. managing the availability of the API, is an application or software concern that leverages infrastructure which in fact could be to a great extent be software as well. Therefore API's are preferably managed on both system and application level by the same team. Consider the API as a product and we're talking a Product Team here.

The importance of an API's availability and the complexity of developing an API leave no room for any other approach than a very tight cohesion between the API implementation and the full stack on which the API is running. Both the API and the underlying stack need to be developed in conjunction.

By the way, software developers shouldn't be concerned with system level management of the API and system managers shouldn't be concerned with application level management. Either both are in the same team, or the team consists of members that understand both levels of management, a rare breed indeed.

So what about functional managers of API's? These are business people, people that understand the business behind API's as well as the API they're managing. They're not technical people at all, they understand platforms, but from a business perspective. They can see IT as a product, and they discuss IT in terms of solutions instead of technologies. They're not Product Owners, they might be the most important stakeholders of the PO on some level.

API's have an implementation and they have consumers, but there're also people that make money of the API. It's the holy trinity of the API: Producer, Consumer and Beneficiary.

Because managing an API on functional level will have its repercussions on both the application level and system level of the API, turning the functional dials needs to be done in concert with the application and system level management efforts. Therefore, functional management should be done preferably by the same team as is taking care of the application and system management of the API. Resulting in a true Product Team.

Concluding

API's need to be managed because otherwise they will become irrelevant and obsolete. In this API's are not other than other software solutions. The complex nature of API's are an additional reason to manage them carefully.

API management can be discussed using the model by Looijen on IT management as all key aspects of his model apply and the clear distinction between business and IT exist with API's as well.

On the other hand, Looijen's idea of a clean segregation between business and IT and even within IT is not a logical segregation for API management. In fact, API's need to be managed by a single team on all levels of API management; The Product Team.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.

But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan

API Management in Azure using Microsoft's solution - API's and Architecture, match made in heaven

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

Summary

The architect is arguably the one most concerned with API's as it is determined by the architecture where the API's reside, what should be API'afied. This has nothing to do with application architecture or integration architecture. The architectural structure of software has no relevance as to when and what to expose through an API. It is the business or domain architect that decides what will be exposed through an API, when and only when the Product Owner wants to reap the benefits of an API to his product. API development may not be controlled by rules, instead it should be governed by principles and guidelines. Although why something should be exposed as an API is a business (architect) call, the how it is exposed is a technical (architect) call.

API's need empowered teams to develop them and more importantly to maintain them. This is not an small thing. API's being extremely hard to develop, they are also extremely strainuous to an organisation because they typically upset the whole structure of the organisation. Even when that organisation is already a product organisation with product teams and product owners.

API's and Architecture

API's are among the hardest things to develop. Why this is, you can read in my previous post here. I think that the role of the architect is extremely important when it comes to API's because they provide access to functionality and information without compromising the underlying systems. And it's not access to just some information, but to information that has meaning on a business level. It's about disclosing your business to the unknown consumer. Well, it's the consumer of your API. So this means that your API's need to fit into the grand scheme of all things IT in your organisation. This is where the architect comes in, because the architect is the city-planner of your IT landscape. See my post on the role of the architect in your organisation, and how that architect may fail miserably here. 

Basically it would be the architect that has a strong say in what the boundaries of your IT landscape are and therefore where the API's are to be found. Mind that this is the IT component of your business architecture.

Say what? Well, it's not that difficult to understand. Consider your business architecture, in a world where IT is considered in terms of products that the organisation is making money off. (As opposite to seeing IT as an enabler or unique selling point.) So basically the organisation defines its business in terms of products. These products and the relationships between them are a way to see the business architecture. It's a very complex exercise to do this and on top of that, many organisations don't even want to make money perse. GO's and NGO's are examples. Still, consider that you do stuff that is of benefit to others, than what you do and what you'll do it with are your products.

These products are at times realised using IT. Maybe the full product is IT or parts of it are. These IT products and their relationships are the IT components of the business architecture I was referring to.

Every connection between any two products is a prime candidate of becoming an API, or becoming API'afied, if that were a word.

Did you notice that I am referring to the business architecture of the organisation and not to an application architecture that is adopted by an organisation. For example, I'm not referring to the 4-tier architecture by Forrester, or the traditional 3-tier (presentation, business, data) architecture, or an SOA or anything like that. Like I discussed in the API-first design post, these architectures have no relationship to your decision what should and what should not be API's. Your business architecture is. Sometimes, especially in larger organisations, that business architecture is decomposed into multiple domain specific architectures, again on a business level. Here you might argue that the relations between domains are even more prime candidates to become API's. And again, how your application architecture is defined has no effect on that.

The Platform

So this is where the architect has value in the API driven universe. Consider this; API's thrive in a platform economy, who decides whether or not there will be a platform and how the platform is shaped? It's the architect and no one else. And this is because the architect is the only one that views things beyond product boundaries, that's the architect's scope. The Product Owner and his Product Team are only concerned with the product. A true Product Owner will also understand the value of API's, because it means more users of his product and more users means more value, but it's the architect that decides a platform is to be developed, raising the value of API's several levels upwards. You should take a look at 'The Age of the Platform' by Phil Simon to understand the value of a platform and why it is such a key to success for many fast growing organisations. Mind that platforms are not only valuable in relation to external customers but also for internal customers.

So basically, the architect is the key role when it comes to API's. The architect defines the business architectures, models the IT products in it, considers where interfacing between domains and products is done through API's. The architect also decides whether or not to work on a platform and that kind of implies going for API's as well.

Organisational Impact

It should also be the architect to determine whether or not the organisation is fit for developing API's. Like I stated in API-first design, developing API's is one of the hardest things to do in software development. Which not only implies, but makes it rather explicit that your organisation needs to be ready in various areas to be able to develop and maintain API's. Continuous Delivery, and preferably Continuous Deployment needs to be in place. You would like to strive for a DevOps IT organisation and you need a very capable Product Owner. On top of that the developers working on API's need to be able to understand the relevant business concepts and need to have extensive experience in the pitfalls of software development. The architect's responsibility is to inform management, including HR management, the impact of adopting a platform strategy, or an API strategy on the organisation. Since architects need to be able to quantify the impact of a change to the IT landscape across all aspects of the enterprise, the architect will need to be able to quantify the impact on staff, staffing and processes alike.

Think about the opportunity to start small, have a single team mature enough to develop and maintain API's. That team should be mandated and thus empowered to develop API's and put them in production, with full autonomy. If the team needs to deploy in production, it should be their call. They should be held accountable and they're responsible for the success of the API. Once that works take all lessons learned and scale your organisation to more teams to work on API's for their products. The architect will know what domain and what product will be best to provide API's to interface with in order to create the most business value.

Governance vs. Control

And then there's the other thing, which is governance. There's no room for control in an API economy. A Product Team that develops API's to access the product's information and functionality will need to have full autonomy. And that means that it can't be controlled. Control means that there's a committee that decides whether or not the team has been abiding by the restrictive rules defined by some other committee. This assumes that it is possible to define rules that API's need to follow without knowing the API and therefore the context within the API is being used. Utterly impossible because when you start developing an API you have no clue about your customer and by monitoring very closely you get an understanding of who your users are and why they want to use your API.

So should there be no control at all? Of course there needs to be control, but it's of the scalable kind. Scalable control is called governance. Governance is done by first define a series of principles. A principle is a rule or belief that governs ones personal behaviour. It's intention is that because one believes that the principle makes sense and one is better off by applying the principle, life is better.

A principle is a rule or belief that governs ones personal behaviour.

By their very nature, principles are used for governance and not for control. It's is one of the most formidable ways to get self organising and autonomous teams work in concert and by the same playbook. The interesting part here is also that you can choose not to stick with a principle in a certain situation. A clear motivation is available implicitly because one abides by the principles, unless there's a good reason. It also allows in your governance structure room for classifying principles in those that can be dropped when the team chooses so and which need an explicit approval from some authoritative body, like an Architecture Board.

Besides the principles, which are set by the architect, the architect needs to define a set of guidelines that inform the Product Team the best course of action to develop an API. These can be related to processes, like how to publish an API. But also technical, for example naming conventions, versioning strategies, etc. Good guidelines are no-brainers but converge various alternative solutions to a problem into a single solution that is the default selectable solution to the problem. It's different from a principle because it's not appealing to a developer's common belief system but to the developer's common sense.

Governance is an architect's dominion. It's not related to a specific team, but it's there to ensure the integrity of the complete IT landscape. This is where the architect shines. The Product Team is responsible for its own principles and guidelines and they should never conflict with the overall, architect driven principles and guidelines. One could argue that the team only makes architecture decisions, based on the principles and guidelines.

Mind that principles are hard to define, almost as hard as API's. Always think of an API as something that sounds like a rule, that has a clear rationale and has an impact which on the one hand details the impact on the existing landscape, the external impact. But also has an internal impact, which details limitations, prerequisites etc on how the software subjected to the principle is developed.

Quite often I see principles that hint towards solutions. For example: High available systems are deployed on a cluster. The rationale is clearly a technical one and the impact restricts the technical aspects of the solution. But an architecture principle's rationale should be a business rationale instead. The impact should allow for as much freedom to the team developing the software without compromising the principle. For API's one needs to be extra careful when defining principles. Stating that API's are always exposed as SOAP webservices seems to be a good principle. SOAP is a great way to do b-2-b integration and it allows for a standardised approach to security, validation, etc. But what if the better way to expose the API is as a RESTful service using JSON? Well in that case the Product Team can motivate why the principle is dropped. But a principle like API's are always exposed based on internet standards and web-based protocols achieves the same, it also allows the product team to select the best solution for their situation and still hold on to the principle. Furthermore, the architect could define a guideline, or guiding principle, stating that for b-2-b connections SOAP is preferred and for b-2-c REST/JSON is preferred.

Empowerment

I've stated before, the Product Team should have a high degree of autonomy when it comes to API development. You want governance in place to make sure that anarchy won't reign. This means that the Product Team should be empowered to decide for itself when principles are selling them short, when a new not yet anticipated situation surfaces or the API's 'market' changes over night due to new insights. In addition, you want the team to be empowered to decide how they develop API's in the most efficient way. Meaning that thinking of an API and having consumers calling the API should be an efficient as possible process. This is an important statement, because it means that the team can't just say 'screw the process or the organisation for that matter' because they have autonomy. Because doing so, would not benefit the complete process from idea to used product.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.
But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan