Principles of the API-First paradigm

API's are weird puppies. They've been around since forever and whole battles have been fought in courts to gain access to API's. This was in the old days when having a platform with an API for 3rd parties to develop for your platform meant that you had a lot of leverage over those parties. Not providing early access to them meant that they would be late to the party and all the sweets where already shared between the early guests. Having an API meant having power. Not providing access to your API resulted in lawsuits because you were a monopolist, using your position as the platform owner to bully your competition into obliteration.

Nowadays the situation is completely different. Having an API means having a future. Nowadays to survive as a business you need to have a platform, which in turn means you need to have an API. Unless you have an API, one that suits the needs of 3rd parties and that is easy to use, you're doomed. You'll be just an Application Vendor, and toast before you can say "I wish I had built a platform". An excellent book on the topic is "The Age of the Platform" by Phil Simon.

As you can read in my previous post, API's are the hardest thing to develop. It always helps to have some guidance when you're up for a difficult task, and often principles help you getting this guidance. There're quite a few other guidelines out there on the interweb that are more than helpful. Below are 5 principles of the API-First paradigm. Followed by 5 derived principles and at the end of the post, I'm listing a couple of more than interesting resources on API design.

One final note before I go into the principles. An API is not much more than an interface into the depths of your platform. It discloses your platform and makes it possible for others to leverage it for their needs. It is therefore critical to have comprehensive documentation for your API. Now comprehensive doesn't mean that you need a big fat document describing everything you can think of that might be remotely related to your API. Typically you will need to document the technical interface of the API, i.e. the call signature of the API, including the input and output parameters of the API.

In addition you will want to document it's behavior exhaustively, i.e. only develop the behavior your documentation mentions. It makes perfect sense to use BDD (Behavior Driven Development, see my post on the topic) for this purpose. By doing so, your users know exactly what to expect from the API and it allows them to create stubs that behave the same as your API without the need to have access to your API's implementation. Finally it helps to have a simple example on how to use the API without making any assumptions as to why one would use the API. This could be through showing the relevant cURL statements.

Enjoy your reading…

Principles of the API-First paradigm

The API is the product.

It is owned by a product owner, and the responsibility of a product team. A team is responsible for it, from cradle to grave. This team is responsible for managing and ensuring its proper operation. Don't consider an API as part of another product, something that's a by-product, because it means that you won't treat it with the proper dignity and soon you'll find yourself alienating from your users.

The API makes no assumptions about its Consumers.

It is robust and resilient enough to thwart of any consumer with bad intentions as well as ignorant ones that have no clue about what they're doing. All this without compromising the service level to all well intending consumers. Mind that in most cases you're working on an API while making assumption as to who and why they'll use your API, only to find yourself in a situation where your users are all but the ones you envisioned, and all using your API for completely different reasons than why the API was created in the first place.

The API is suiting the needs of the Consumer

The API is developed only when there is at least one consumer and it is tailored to suit this consumer, by this the API is creating business value through its consumer. Don't be so arrogant to develop something that's supposed to be an API without having at least one consumer lined up to actually use it and tell you how great it is. Apart from the fact that working on something that is not being used is a complete waste of time, it also means that you're not really understanding the fact that you should not make any assumptions about the API's consumers. No assumptions at all.

The API is built to last

An API is never breaking compatibility with its consumers, although it may rely on the paradigm of 'Ignore what is unknown' to allow for extending the result. API's, other than the humans that build them, never break a contract. Period! This is to say that they will respect contracts, will have addendums to the existing contract, or in those cases when the API, which is after all a product, is no longer adding value, they will void the contract, but obviously respecting the contractual notice period. On a side note, it is fair to assume that its consumers abide by the convention that they'll ignore what they don't understand or know, hence the fact that you can add addendums to existing contracts without actually breaking the contract.

The API is idempotent.

At all times, in any situation, always. It doesn't matter what you do, how you do it or when you do it, when you call an API it always behaves exactly the same. Meaning that its result is always the same and therefore perfectly predictable. The order in which API's are called doesn't matter, as long as you put in the same values, you will get the same results.

Architecture principles for the API

The API is clear in its error reporting

Errors are either caused by the API implementation, the API consumer or something unexpected. It makes perfect sense to stick with the conventions of HTTP errors, why invent something new, when the existing is fitting your purposes perfectly. But whatever you do, make a clear distinction between these three causes for errors.

The API is always managed.

An API can be managed independently from;
Technical level, i.e. it can be accessed securely and consistently throughout its lifespan -> Scalability, Availability, etc

Application level, i.e. consumers can use the API based on a published interface that is behaving per this interface -> Interfaces, authentication/authorization, etc.

Functional level, i.e. users of the consumer can be provided access to the API based on the agreements made -> Throttling, metering, monitoring and reporting, etc.

The API can be implemented independently of its interface;

They are truly decoupled. Multiple implementations can co-exist and determining which implementation to choose for each individual call can be context driven. Mind that the interface should be technology agnostic, or at least as much as possible, and in no way should the interface disclose anything about the implementation other than that it should explain how it behaves.

The API is technology agnostic.

The interface can be called by a consumer of any technology without any assumptions of the technology used in the API implementation, and vice versa. This is pretty much an extension of the previous principle, but important enough to mention explicitly.

API's are like dogfood.

The API is used by the team responsible for it in the same use cases as the consumers for which the API was intended; "Eat your own dogfood" paradigm. Always make sure that in case you need functionality that one of your API's is providing, you're calling that API yourself and making yourself dependent on the API. Only this way you understand your API, and more importantly you'll be very cautious about how you treat your API and therefore how you treat its users. After all, you'll be one of them.

Interesting Reading

• Google's API design guidelines
• 18F API guidelines
• W3C Web API Design Cookbook
• CISCO API design guidelines
• Haufe API style guide
• Heroku API guidelines
• Paypal API guidelines

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contactlist.But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

API Management in Azure using Microsoft's solution - Resources and Templates [1/2]

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

Applications

Applications are pieces of software that when combined they perform a business function. Typically, applications perform various business functions, but ideally each application provides access to a single business function. If you think of it this way, most of the software we traditionally know and see in our IT landscapes, what we call applications, actually has multiple applications. You can use this software in a variety of ways in a variety of situations to perform a variety of business functions. So the name 'Application' is incorrect but understandable.

As always, the cause of this misnomer can be found in history. Deploying software used to be a complex thing in the past as was software development and even more so developing software that communicates with other software. A ton of different concepts, protocols and technologies have emerged over time to facilitate the connectivity between different pieces of software. (Skip the next section when you're not interested in a short but incomplete overview of software interoperability)

Software Interoperability

Microsoft released (D)COM in the past followed by ActiveX. The rest of the world joined into the fray with the incompatible CORBA standard(s), there was RMI for the Java world and RPC for the C(++) world. Then we had other solutions based on message exchange in the form of MOM (Message Oriented Middleware) which was, more, protocol independent. Although MOM as a concept was awesome, the big vendors succeeded in ensuring that their products were not interoperable. And because most of the vendors realised their customers didn't really appreciate this, the ESB was invented. Again, it was there to make sure that pieces of software could interoperate and fix the complexity of our monoliths. ESB's, as we all know, are a big waste of time and money and by far deliver on their promises. I did write a post on the topic, which you can read by clicking here and I'll write a little update on the topic soon as many ESB aficionado's don't see this yet.

So after we started omitting ESB's from our IT landscapes but kept the ESB concept around, we finally found ourselves in a world that actually delivered on promises and was low cost; The Internet.

Web-services

You can skip this part if you already know everything about web-services and are not interested in what I have to say about them. It's relevant, but not such that you can't skip over this section.

So web-services, both SOAP and REST-based, are interesting little beasts in our IT landscape. Web-services are the first real useful concept when realising an SOA. A Service Oriented Architecture. Based on Internet technology and therefore implicitly complying with all the requirements of an SOA.
And then there's the fact that web-services are accessed through a simple interface, the URL. So even in case you don't think about it, you're forced to limit the scope of what a web-service can do because you're limited to a URL. And although behind many URL's there can be a single monolithic structure, or you can use all kinds of different ways of encoding many functions within one URL, there really is no point in doing so, because working with URL's makes it harder to do things the wrong way than to do it the right way.
And the cool part is that this is true for even high level business functions that require a variety of (technical) low level web-services. On every level, working with web-services (almost) requires you to develop software that do only a single (business) function. As such, a web-service can be considered an application.

Web-services make it harder to do it wrong than to do it right.

If you look at it this way, a web-service being an application, you can see that a web-service is a small piece of software that provides just enough functionality for another piece of software or a person, to consider useful.
In effect; Every web-service, by means of its interface exposed as a URL, is exposing a (business) function. And as soon as you aggregate various web-services into a new web-service, meaning you call several web-services in a particular order to implement a high level business function, your new web-service complies to the same rules.

When we look at API's, they're the interface that we put in front of some software to expose specific business functionality. Look at the previous post on the topic by clicking here.

API's are products. There's an intended full stop because in many ways people tend not to think that way. But really, an API is where the boundary of a system is. The rest of the world accesses that software system through the API. The rest of the world might be a user interface.
So if you think about API's being products, or possibly a set of API's being a product, you're on the right track, because you then understand that a web-service is not a product in itself, instead it is a piece in a product's puzzle.
Why this is important? Because the API developer has the responsibility towards the consumers of the API, ensuring that changes to the API do not impact existing consumers of the API and lure new consumers at the same time. A developer of a web-service does not have this responsibility. or rather there's a way out.

Remember that in a previous post I mentioned that an API does not make any assumptions about it's consumers. They're not known in advance and any assumed consumer is likely not the biggest fan. Web-services on the other hand do know their consumers. Or at least, they can be developed such that assumptions are made regarding the consumer. This is also the reason why there's a need for API management in some cases and no need at all in other cases.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.


Arc-E-Tect

API Management in Azure using Microsoft's solution - Why API's should be managed and by whom

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

I hope you've read my previous post on the topic of API Management, if not I strongly advise you to click here and read it.

Summary

Although managing API's has some resemblence with Looijen's model for IT management, the typical segregation between business and IT that stems from Looijen's model is disastrous for API management. API's need to be managed by a single team, the Product Team. Any other way of managing an API doesn't make sense. API's are products and should be treated that way.

API Management

API's are interfaces to (business) functionality that are accessed by, in theory, just about anybody. As can be read in my previous post, the idea of having API's and one of the main reasons why developing them is so hard, is the fact that you don't know up front who is going to call the API. The point in fact is that in a world of API's there is no room for the arrogance of the developer knowing exactly who is going to use the software and why. Instead, API's are inherently developed to support the case that the developer was wrong.

Because of this, when you expose an API, you'll need to make sure that you can support the API once it is deployed and used by somebody. Supporting means that you need to make sure the API is running consistently over time, the API must be usable and stay usable over time. Hence the API must be managed.
In order to do so, API's need to be managed on three different levels that to some extent relate to the support levels documented by Maarten Looijen, of which a description in Dutch can be found here. Basically he defines Functional, Application and System Management

The Looijen Model

(Skip this section in case you're not interested in the different kinds of support types.)

The model is not shocking as it states that there're three types of support that is required to support an IT system:

1. Functional Management. [Original text translated by Google:] Functional management is the management form that includes all administrative tasks that are required in the context of the use of information systems. As usage focuses on functionality, such as entering, handling, procurement, transportation and storage of data refers to information management.
2. Application Management. [Original text translated by Google:] Application management involves developing and maintaining applications. In the words of Parts and Looijen: "There may be many situations in which changes should be made in the original application software or database structures (...) Whenever such a situation occurs, application maintenance is performed but also.. when there is no direct reason is to perform application-maintenance, it is because of the continuity may still be required in order to maintain a complete management shape surrounding this executive task area application maintenance, including management and executive tasks. " Application management is like form, as it were about the task area applicate maintenance built around. It is to recognize application is essential that once developed information not evident also delivers the tools to easily support the change process (eg. Documentation). In addition, persons other than the original developers may be involved in the execution of modifications. Therefore it is important to maintain conditions which an application must meet so that they can be maintained 'easy'.
3. Systems Management. [Original text translated by Google:] Technical management includes the management form all tasks necessary to install, accept and operationalize and maintain information systems and technical infrastructures. Under technical management also includes optimizing processes and making changes in the technical infrastructure as a result of errors, extension or replacement. The technical management is focused on the technical platform, consisting of equipment associated with basic software, and implementation of information systems built on it.

Management of API's the Looijen way

Now that you understand what I mean by the Looijen model, it's time to inform you how API's should be managed.

API's are to be managed on the same levels as other software, that is the functional, the application and the system level.

System Level: On the system level, API's need to be managed such that they are made available. At the very minimum there should be an error stating the API cannot be accessed. At system level you manage the infrastructure on which the API is running. This might include some middleware as well. Since API's are to be called a lot, system management also covers scaling, i.e. elasticity, as well as all kinds of resilience aspects. Also security aspects like DDOS prevention, (web application) firewalls, etc. are part of the system management aspects of API management.

Application Level: On the application level, API's need to be managed such that they can be accessed, that the API documentation, (how to call the API?), is available, etc. Basically what it means is that there is an API to be called and that the technical aspects of the API itself are managed. For example the fact that an API accepts both JSON and XML parameters, that there is an API specific certificate defined such that the caller can verify the API call is served by a trusted system etc.

Functional Level: On the functional level, API's need to be managed such that they provide some value to somebody, preferably to both the consumer of the API as the team behind the API. API's are excellent for creating win-win situations, and these are business relevant situations. Basically what managing API's on a functional level means, is that you manage the relationship between the API and the consumer of the API. Think about a consumer having to pay for access to the API and setting different thresholds to control this.

When you look at the above different levels of API management, you may have noticed that on a system and an application level, managing the API is rather technical, but on a functional level it's really a business concern. Switching back to Looijen's model and you see that he makes a clear distinction between the two as well. System and application management are technical concerns, functional management is a business concern.

What if API's are not managed?

So what if you don't manage API's? The answer is actually rather simple. It becomes irrelevant, and in the end will seize to exist. For one thing, an API needs to be kept available. When it's not available, no consumer can access it. So on a system level it needs to be managed. This is no different from any other software application. Not managing an API on a system level results in the same chaos as not managing any other software, with the side note that API's are called by any(kind of) consumer, so the 'blast radius' cannot be predicted. From this, there's no reason not to manage the API on a system level as any software needs to be managed. There's one other point to be made here, and that is that an API needs to be kept available, meaning that it needs to scale, preferably horizontally and it needs to be resilient. Both are aspects that are not traditionally system management aspects as with API's this needs to be addressed on a software level.

On an application level things are getting a little convoluted with the system level management. In fact you might already have gotten the opinion that system level management is very much like application management and in fact you're correct. But system level management is more of a 'keep the shop open' issue, and application level management is more of a 'retain customers and grow your market' issue. When customer are not attended to, they'll go elsewhere. API's need to be managed on an application level as they are called because the consumer wants to. When the consumer needs to, eventually he'll leave for a more convenient API that provides sufficient added value to migrate to it. Hence the API needs to be amended to support other protocols, other data formats, more security etc. All aspects of application level management. See this as a matter of keeping the interface of the API consistent, but the implementation behind it evolving. So the API might be providing a consistent end-point to a consumer but evolves from a low-cost inefficient implementation to a costly but efficient implementation. This is API management on application level; replacing implementations behind an interface. If an API is not managed like this, it will become obsolete at some point. Obsolete API's are useless but cost a lot.

The system level and application level management concerns of an API are clearly required in case an API to some system or information is provided. When the API is not managed on both levels, the API will become irrelevant. But what about the functional level?

Functional management of an API is addressing an API on a business level. Read the post on API first design here to understand why API development is costly and if not anything else, the most difficult thing you can do in software development. Because it's so tricky, it's important that there's a business reason to actually do API development in the first place. And this is where functional management of an API comes to the stage. This is where it is managed that an API is monitored on a business level, for example keeping track of how many different consumers are using the API, how many of the consumers are 'returning customers' etc. Functional management of an API is all about managing the metrics about consumers of the API and act on them. Consumers that do a lot of calls may be willing to pay for each call to the API, consumers from specific industries will be more willing to sacrifice some performance in order to gain security etc.

When API's are not managed on a functional level, there is no point in developing an API. API development is so complex that there needs to be a business justification to do so. And it needs to be validated that the initial justification which is always based on assumptions, is correct. Not managing API's on a functional level will ensure that funding in the broadest sense of the word, will stop eventually.

Who manages API's?

API's are managed by both business and IT. Refer to Looijen again and you see that for software solutions, managing these solutions requires both IT and business. The same goes for API's. In a traditional organisation, one that has a clear separation between business on one side and IT on the other, API's are technically managed by IT and the developer of the API needs to provide an interface to the business to manage the API on a functional level.

Further complicating the scene, refer to organisations where IT is divided into a department that is more concerned about applications and the other department more about infrastructure. In this situation system level management is done by the infrastructure people and application level management is done by the application people.

Followers of the Looijen model are typically of the opinion that Application Support can be done by system managers. The reasoning often is that system managers are technically skilled people that understand applications as they understand the infrastructure on which the applications are running. And since they manage the technical aspects of the application and not the functional parts, they're more than suitable to manage the application as well as the underlying infrastructure. This view is incorrect (click here to read why) and when it comes to API's it's disastrous.

When you have an API based platform, and there really is no other way to have a platform, you need to consider the software implementing the API as well as the infrastructure on which it is running as a single system. It makes hardly any sense to see them as two separate pieces of the API puzzle. The reasoning here is that scaling, resilience and other aspects of the system level management of the API, i.e. managing the availability of the API, is an application or software concern that leverages infrastructure which in fact could be to a great extent be software as well. Therefore API's are preferably managed on both system and application level by the same team. Consider the API as a product and we're talking a Product Team here.

The importance of an API's availability and the complexity of developing an API leave no room for any other approach than a very tight cohesion between the API implementation and the full stack on which the API is running. Both the API and the underlying stack need to be developed in conjunction.

By the way, software developers shouldn't be concerned with system level management of the API and system managers shouldn't be concerned with application level management. Either both are in the same team, or the team consists of members that understand both levels of management, a rare breed indeed.

So what about functional managers of API's? These are business people, people that understand the business behind API's as well as the API they're managing. They're not technical people at all, they understand platforms, but from a business perspective. They can see IT as a product, and they discuss IT in terms of solutions instead of technologies. They're not Product Owners, they might be the most important stakeholders of the PO on some level.

API's have an implementation and they have consumers, but there're also people that make money of the API. It's the holy trinity of the API: Producer, Consumer and Beneficiary.

Because managing an API on functional level will have its repercussions on both the application level and system level of the API, turning the functional dials needs to be done in concert with the application and system level management efforts. Therefore, functional management should be done preferably by the same team as is taking care of the application and system management of the API. Resulting in a true Product Team.

Concluding

API's need to be managed because otherwise they will become irrelevant and obsolete. In this API's are not other than other software solutions. The complex nature of API's are an additional reason to manage them carefully.

API management can be discussed using the model by Looijen on IT management as all key aspects of his model apply and the clear distinction between business and IT exist with API's as well.

On the other hand, Looijen's idea of a clean segregation between business and IT and even within IT is not a logical segregation for API management. In fact, API's need to be managed by a single team on all levels of API management; The Product Team.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.

But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan

API Management in Azure using Microsoft's solution - API's and Architecture, match made in heaven

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

Summary

The architect is arguably the one most concerned with API's as it is determined by the architecture where the API's reside, what should be API'afied. This has nothing to do with application architecture or integration architecture. The architectural structure of software has no relevance as to when and what to expose through an API. It is the business or domain architect that decides what will be exposed through an API, when and only when the Product Owner wants to reap the benefits of an API to his product. API development may not be controlled by rules, instead it should be governed by principles and guidelines. Although why something should be exposed as an API is a business (architect) call, the how it is exposed is a technical (architect) call.

API's need empowered teams to develop them and more importantly to maintain them. This is not an small thing. API's being extremely hard to develop, they are also extremely strainuous to an organisation because they typically upset the whole structure of the organisation. Even when that organisation is already a product organisation with product teams and product owners.

API's and Architecture

API's are among the hardest things to develop. Why this is, you can read in my previous post here. I think that the role of the architect is extremely important when it comes to API's because they provide access to functionality and information without compromising the underlying systems. And it's not access to just some information, but to information that has meaning on a business level. It's about disclosing your business to the unknown consumer. Well, it's the consumer of your API. So this means that your API's need to fit into the grand scheme of all things IT in your organisation. This is where the architect comes in, because the architect is the city-planner of your IT landscape. See my post on the role of the architect in your organisation, and how that architect may fail miserably here. 

Basically it would be the architect that has a strong say in what the boundaries of your IT landscape are and therefore where the API's are to be found. Mind that this is the IT component of your business architecture.

Say what? Well, it's not that difficult to understand. Consider your business architecture, in a world where IT is considered in terms of products that the organisation is making money off. (As opposite to seeing IT as an enabler or unique selling point.) So basically the organisation defines its business in terms of products. These products and the relationships between them are a way to see the business architecture. It's a very complex exercise to do this and on top of that, many organisations don't even want to make money perse. GO's and NGO's are examples. Still, consider that you do stuff that is of benefit to others, than what you do and what you'll do it with are your products.

These products are at times realised using IT. Maybe the full product is IT or parts of it are. These IT products and their relationships are the IT components of the business architecture I was referring to.

Every connection between any two products is a prime candidate of becoming an API, or becoming API'afied, if that were a word.

Did you notice that I am referring to the business architecture of the organisation and not to an application architecture that is adopted by an organisation. For example, I'm not referring to the 4-tier architecture by Forrester, or the traditional 3-tier (presentation, business, data) architecture, or an SOA or anything like that. Like I discussed in the API-first design post, these architectures have no relationship to your decision what should and what should not be API's. Your business architecture is. Sometimes, especially in larger organisations, that business architecture is decomposed into multiple domain specific architectures, again on a business level. Here you might argue that the relations between domains are even more prime candidates to become API's. And again, how your application architecture is defined has no effect on that.

The Platform

So this is where the architect has value in the API driven universe. Consider this; API's thrive in a platform economy, who decides whether or not there will be a platform and how the platform is shaped? It's the architect and no one else. And this is because the architect is the only one that views things beyond product boundaries, that's the architect's scope. The Product Owner and his Product Team are only concerned with the product. A true Product Owner will also understand the value of API's, because it means more users of his product and more users means more value, but it's the architect that decides a platform is to be developed, raising the value of API's several levels upwards. You should take a look at 'The Age of the Platform' by Phil Simon to understand the value of a platform and why it is such a key to success for many fast growing organisations. Mind that platforms are not only valuable in relation to external customers but also for internal customers.

So basically, the architect is the key role when it comes to API's. The architect defines the business architectures, models the IT products in it, considers where interfacing between domains and products is done through API's. The architect also decides whether or not to work on a platform and that kind of implies going for API's as well.

Organisational Impact

It should also be the architect to determine whether or not the organisation is fit for developing API's. Like I stated in API-first design, developing API's is one of the hardest things to do in software development. Which not only implies, but makes it rather explicit that your organisation needs to be ready in various areas to be able to develop and maintain API's. Continuous Delivery, and preferably Continuous Deployment needs to be in place. You would like to strive for a DevOps IT organisation and you need a very capable Product Owner. On top of that the developers working on API's need to be able to understand the relevant business concepts and need to have extensive experience in the pitfalls of software development. The architect's responsibility is to inform management, including HR management, the impact of adopting a platform strategy, or an API strategy on the organisation. Since architects need to be able to quantify the impact of a change to the IT landscape across all aspects of the enterprise, the architect will need to be able to quantify the impact on staff, staffing and processes alike.

Think about the opportunity to start small, have a single team mature enough to develop and maintain API's. That team should be mandated and thus empowered to develop API's and put them in production, with full autonomy. If the team needs to deploy in production, it should be their call. They should be held accountable and they're responsible for the success of the API. Once that works take all lessons learned and scale your organisation to more teams to work on API's for their products. The architect will know what domain and what product will be best to provide API's to interface with in order to create the most business value.

Governance vs. Control

And then there's the other thing, which is governance. There's no room for control in an API economy. A Product Team that develops API's to access the product's information and functionality will need to have full autonomy. And that means that it can't be controlled. Control means that there's a committee that decides whether or not the team has been abiding by the restrictive rules defined by some other committee. This assumes that it is possible to define rules that API's need to follow without knowing the API and therefore the context within the API is being used. Utterly impossible because when you start developing an API you have no clue about your customer and by monitoring very closely you get an understanding of who your users are and why they want to use your API.

So should there be no control at all? Of course there needs to be control, but it's of the scalable kind. Scalable control is called governance. Governance is done by first define a series of principles. A principle is a rule or belief that governs ones personal behaviour. It's intention is that because one believes that the principle makes sense and one is better off by applying the principle, life is better.

A principle is a rule or belief that governs ones personal behaviour.

By their very nature, principles are used for governance and not for control. It's is one of the most formidable ways to get self organising and autonomous teams work in concert and by the same playbook. The interesting part here is also that you can choose not to stick with a principle in a certain situation. A clear motivation is available implicitly because one abides by the principles, unless there's a good reason. It also allows in your governance structure room for classifying principles in those that can be dropped when the team chooses so and which need an explicit approval from some authoritative body, like an Architecture Board.

Besides the principles, which are set by the architect, the architect needs to define a set of guidelines that inform the Product Team the best course of action to develop an API. These can be related to processes, like how to publish an API. But also technical, for example naming conventions, versioning strategies, etc. Good guidelines are no-brainers but converge various alternative solutions to a problem into a single solution that is the default selectable solution to the problem. It's different from a principle because it's not appealing to a developer's common belief system but to the developer's common sense.

Governance is an architect's dominion. It's not related to a specific team, but it's there to ensure the integrity of the complete IT landscape. This is where the architect shines. The Product Team is responsible for its own principles and guidelines and they should never conflict with the overall, architect driven principles and guidelines. One could argue that the team only makes architecture decisions, based on the principles and guidelines.

Mind that principles are hard to define, almost as hard as API's. Always think of an API as something that sounds like a rule, that has a clear rationale and has an impact which on the one hand details the impact on the existing landscape, the external impact. But also has an internal impact, which details limitations, prerequisites etc on how the software subjected to the principle is developed.

Quite often I see principles that hint towards solutions. For example: High available systems are deployed on a cluster. The rationale is clearly a technical one and the impact restricts the technical aspects of the solution. But an architecture principle's rationale should be a business rationale instead. The impact should allow for as much freedom to the team developing the software without compromising the principle. For API's one needs to be extra careful when defining principles. Stating that API's are always exposed as SOAP webservices seems to be a good principle. SOAP is a great way to do b-2-b integration and it allows for a standardised approach to security, validation, etc. But what if the better way to expose the API is as a RESTful service using JSON? Well in that case the Product Team can motivate why the principle is dropped. But a principle like API's are always exposed based on internet standards and web-based protocols achieves the same, it also allows the product team to select the best solution for their situation and still hold on to the principle. Furthermore, the architect could define a guideline, or guiding principle, stating that for b-2-b connections SOAP is preferred and for b-2-c REST/JSON is preferred.

Empowerment

I've stated before, the Product Team should have a high degree of autonomy when it comes to API development. You want governance in place to make sure that anarchy won't reign. This means that the Product Team should be empowered to decide for itself when principles are selling them short, when a new not yet anticipated situation surfaces or the API's 'market' changes over night due to new insights. In addition, you want the team to be empowered to decide how they develop API's in the most efficient way. Meaning that thinking of an API and having consumers calling the API should be an efficient as possible process. This is an important statement, because it means that the team can't just say 'screw the process or the organisation for that matter' because they have autonomy. Because doing so, would not benefit the complete process from idea to used product.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.
But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan

API Management in Azure using Microsoft's solution - API-first design

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

Summary

API's are about the hardest thing to create in software engineering. Designing software to act as an API before anything else is even harder.

There's a new buzz going around in IT; API Management, and everything associated with it. I think it's even more buzzy than Microservices. Especially with shout-outs like "API First", compared to "Cloud First", "Mobile First" and "Web First" of yesteryear, we need to be cautious about what this means. This post is not about what API's are, why you should develop API's, or why API development is probably the hardest area in software development in this day and age. This post is about how you should look at Microsoft's offering on API Management and how to use it. Not the definitive guide, but a guideline instead.

This post is focusing on the API-first buzzzzzz...

Microsoft Azure

Skip to the next section when you're not interested in a view of mine on Azure, otherwise keep reading.

Whatever you think about Amazon AWS and the incredible pace by which Dr. Werner Vogel and his team is telling the world what The Cloud is and will be, you have to agree that the more amazing player in the Cloud arena at this point in time is arguably Microsoft.
Amazon has a track record of providing an amazing experience to their customers, both at Amazon.com but also on AWS. Clearly considering what the customer wants and doing everything to make the customer want to spend more money with them then anywhere else. They're doing this for over a decade now. I would argue that AWS is for those that understand the Cloud. Microsoft is arguably only since about 5 or 6 years a Cloud player to consider. Although announced in 2008, their implementation of the Cloud was in early 2010 and called Microsoft Azure in 2014. It really matured in 2016 with proper RBAC (Role Based Access Control) and proper infrastructure-in-code capabilities through the Azure Resource Manager. Other than AWS, which is a true 'API first' and 'Automate everything' platform, Azure is only since recently truly catching up in allowing everything to be done through API's and automation. Which is too bad since one of the Cloud's most awesome traits is the virtually infinite amount of resources allowing to provision and deploy a virtually infinite amount of versions of an application. Anyway, Microsoft's lack of API's and automation capabilities in the Cloud is consistent with its desktop platform Windows, which is a point-and-click affair instead of a scripting affair. Even administrators on Windows will use the point-and-click interface instead of the CLI (Command Line Interface) which is so accustomed to their Unix and Linux colleagues. It makes the platform less intimidating and many of the hard work is hidden behind a button and some checkboxes. This is how the Cloud is managed as well, through a Portal. It's a "point-and-click first" philosophy it seems. And many of the customers of Azure are happy with this. Who wants to be intimidated by a computer after seeing 2001 and Terminator anyway?

Setting AWS and Azure side-by-side one might see the following key difference: AWS allows you to do IT like playing with Lego. Amazon started with the basic building blocks. The Lego of 30 years ago and they allow you to build pretty much everything, if you're willing to spend the time. As time passes they come out with more advanced blocks, like the roofs in Lego. And just like Lego came with the little Lego men, gears, axles and so on, Amazon introduced more and more advanced services on AWS. But the idea, it seems, has always been to allow the customer to build pretty much everything she wants. Microsoft on the other hand seems to have followed the PlayMobil path. The fun is in the playing part and not so much in the building part. So open the Azure box and unlike AWS, you can start playing from day one. And as time passes by, there are more and more lower level components available, so you can start playing new games by building new toys.
Neither approach is better than the other, but you should carefully look at where you are, what you can do and what you want in order to be able to select either AWS or Azure. And IMHO, if you're just selected to go with Azure, don't even think about also doing AWS. And the other way around, because you would be most likely dealt lemons and you won't be able to make lemonade out of them.

API Management

First of all, API Management is a concept. I'm taking it you understand what an API is and I won't go into it any further than necessary for understanding this post.
API Management is done by people and some of them are using tools and technologies we call API Managers. Just like an enterprise, it has Management, and Management is made up of Managers.

Second of all, API-first design is not a fluke and it's not a hype either. Instead it is calling doom and gloom over your IT organization, or quite the opposite when you really know what you're doing.

Third of all, there's nothing MicroServices about API's although you see both walking around hand-in-hand more often than not.

Let's start with number 2...

API-first design
With API-first design, what you do is, you think about the functionality or information you want to expose and design it such that you don't have a clue about who will need or want to use that functionality or information. What you need to do next is worry about this. Start worrying a lot, because if you don't have a clue about who will access your functionality or information, let's call it a service, you don't know their intent either. And you don't know how often they'll access your service. You don't know whether or not you can trust them either. Are they taking security as serious as you do? Are they complying with all regulations you need to comply with? etc. Worries, worries, worries.

By not making any assumptions about the consumer of your service, you will need to make sure that it can stand the assault of all kinds of consumers. I do see your mind fluttering into the direction of non-functionals. Things like resilience and availability maybe? And encryption and access management possibly as well. Drop the mother in the cloud and you've got your scalability straight out of the Cloud's elasticity. (Of course you're very intimate with my post on this topic). And when you're in the Cloud, you know pretty much for sure that your Cloud provider will have all kinds of network protection in place without you even knowing it. Making sure that you, nor any other customer of theirs, will suffer from a DDOS attack and the likes.
But when we talk about API-first, we also need to think about the granularity of the service's exposed functionality or information. We need to think about it very, very carefully. In general we can say that less is more when it comes to this. But in fact, you want to hit that sweet-spot, because only then you're API is build to last.

That sweet-spot is something really important and almost impossible to find, let alone to hit. It's like the mythical G-spot. We know it should be there, but once you find it, you don't have the guarantee you will be able to find it all the times you're looking for it.
Granularity of your service defines it's usability and it is extremely business-bound. So first of all, you will need to know exactly the business context of your API and, and this is important, understand the business context of your consumers. And you should, no you must, accept that you'll have this latter one wrong. The most unlikely consumer will be your most valuable customer, so unless you're capable of reacting to her needs very quickly. Able to adapt to a massive change in positioning of your API in the grand scheme of the universe due to a pivot of your business, don't even try to go for API-first design.

By getting down to business, understand that an API is to be considered a product. It's not just access to your productised software solution, no it is your product. Nobody will care and shouldn't care how you do the magic behind the API as long as you're doing it. So you need to treat your API as a product. And that means that your API should have business value and as such, you should be very much aware that your API is valuable to your customers. In short, think about your revenue generated by your API. This is where your business context is showing it's ugly face again, because initially you can only assume how your API can be valuable to your customers. Hence you can only guess, albeit an educated guess, how you can get revenue from your API. Here again you'll need to be able to adapt to changing views and accept that hindsight is 20-20.
What this means is that you'll have to build (business) metrics into your API. Think upfront how you can verify your hypothesis and then do the validation of it as well. And don't be shy to change your product. A good product is one that is used, a great product is one that is valuable.
A key aspect of your API's granularity is about how elementary it is. You don't want to have an API that is only providing basic information, one that is only spewing data. Instead you want an API that is providing information in context. How much context is part of finding that sweet-spot. For example an Address by itself might mean nothing unless you know whose address it is, so a service that spews out addresses is of limited value, but one that provides access to a person and her address might be invaluable. Or maybe access to information is not that much of a thing to consider and instead you provide access to a function, for example to put some random text on API Management on the internet.

A good product is one that is used, a great product is one that is valuable.

The point here is that there's no way to determine what makes sense as an API and what not, unless you understand the context of your API at your end. And this only because it provides you with a starting point to make educated guess about the context of your API's consumer.
And at all times, the quality of your product needs to be such that you won't need to enter dialogue with the customer about the product, explaining why it's not working.

By the way, your architecture, blue-prints or design-patterns are no indication of what should be an API in the sense of API-first. In fact, I would argue that they are not an indication of what should be API's and managed as such and what not.

Considering you're developing according to service oriented concepts, and quite possibly adopting the principles of Forrester's 4-tier architecture. Which is in fact not a bad thing at all and probably a very beneficial view on things in general. So considering this, you might be inclined to consider very service in a specific architectural layer to be an API. For example the Aggregation Tier in Forrester's model or the Systems Tier in that model, could be considered to be an API. Which is total rubbish!
In software engineering we don't just do, we think about it and decide. So calling something a service in a specific part of our architecture an API is nonsense, should be cause of being fired in fact. And in fact, after thinking about it and deciding you have a service that is bound to be reused doesn't mean that will be an API either. So you'll have to do some more thinking... think... think... think. And then worry about calling something an API when you know in what situation the service is going to be used. Worry about calling something an API when you know who's going to call you to yell at you because the service is not abiding the service contract. Because if you know this, you know who your service's consumer is, and most likely there's no need to do any API'afyication on your service. Doesn't mean that you shouldn't treat it as an API, it means you don't have to treat it as an API if that doesn't warrant the effort.

In software engineering we don't just do, we think about it and decide.

API's have all kinds of stuff in them one way or the other, that allows them to be unleashed into the world. It's all the stuff that ensures that they survive the onslaught of calls by initially unknown subjects, the unsubs.
It's the part of API-first design that ensures that none of the wonderful promises of pots with gold by exposing the service's splendour, is compromised. It's the resilience, availability, scalability, security as well as metrics and other business requirements we too often consider IT stuff.
Think about the idea of implementing it over and over again for every service interface. If that means you're starting to sigh, and very deep at that, than only do it for the real API's. The service interfaces for which you don't know who'll call them. But when you get a smile on your face, eyes start to gleam, then you might as well take it from the other end; think about why you shouldn't treat the service as an API. Could be because of money (using standard tooling to API'afying service interfaces cost money) or maybe the operational overhead is not worth it.

So, in order to do API-first design, you need to understand your own business, meaning you need to understand what you want to expose and how you can create business value with your information or capabilities. This is hard enough for most software engineers, so they'll need the support of a business expert. Than you need to think about why others, and you don't know who that might be, would like to use your API, your product, and why it's valuable for them. This requires business knowledge and some understanding of how to see business value in using information flows. This is not the average business person you need to work with.
Than you need a thorough understanding of how to develop a resilient, performant and secure piece of software that can run in the Cloud, i.e. and elastic environment that is to be considered to be crashing all the time. And if you have all this in place, you need to be able to be roll-out new versions of your product on a very regular basis, with an extremely high level of quality. And on top of that, never get frustrated because your business pivots over and over again, because they're not really sure for whom the product is intended.

API's are about the hardest thing to create in software engineering. Designing software to act as an API before anything else is even harder.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.
But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Iwan