The Arc-E-Tect predictions for 2017 - In hindsight [2/2]

Last year, like every year, I did some predictions on what would be in and what would be out in 2017. But unlike other years, last year I actually posted those predictions on the internet.
Before I start with my predictions for 2018, I will go back to my predictions for 2017 and see how things turned out.
This is part two, and part one you can find here.

#6: KVI in, KPI out

"Forget about performance. Performance, in the end, means nothing when it comes to an organisation’s bottomline. What matters is value. However you want to cut it, unless value is created, it’s not worth the effort. And by value being created I mean that the difference between cost and benefit increases."

First prediction that I am looking at in this post is a bust. Although more and more teams and organisations are transforming into agile adopters, the value driven aspects of agility is still undervalued by most. I hardly come across organisations, departments or even just teams where success is measured in terms of realised value. Vanity metrics are pretty much still the norm. It's a shame because it also means that the promise of applying agile concepts are still a long way from being realised.

#7: Products in, Projects out

"It shouldn't surprise you, but I'm not a big proponent of projects and instead love to see it when organisations switch to a product focused approach. But in 2017 it will turn out that I'm not the only one."

This is happening big time in a lot of environments I've been in. The main reason why organisations transition from a project perspective towards a product perspective is because of CI/CD (Continuous Integration/Continuous Delivery). With reduced cycle times as a result of automation of the software delivery process, it is almost impossible to not release a product early and keep on working on it. Hence, delivery to production does not result in the end of a team.
My main concern in these situations is the lack of a Product Owner who has mandate over scope. The Project Manager typically does not have that mandate. It is the next step.

#8: Heterogeneous in, Homogeneous out

"In 2017 we’ll truly face the uprising of new and more technologies, concepts, architectures, models, etc. And in order to be able to manage this we will finally understand that we need to embrace the fact that our environments consist of a multitude of everything. In many smaller organisations that are at the forefront of technology and that are working in agile environment it is a given, but now that large organisations have also set out to adopt the ‘Spotify’ concept and thus teams have a huge amount of autonomy, polyglot is key."

Yes! Most organisations have dropped their need for huge standardisation efforts. Instead I see that architecture principles are becoming highly popular. With that and the gradual move towards autonomous teams I do see a shift in mindset where homogeneous environments is no longer considered the answer to all IT problems. This is also a mindset shift from efficient towards effective.

#9: Activities in, Roles out

"The thing is, we’re moving, as an industry, in the direction where we want be able to get feedback as early in the process as possible, which means that every person concerned with creating and delivering a products will be involved in everything needed to create that product and ensure that it works as intended and more importantly as needed. In this setup, everybody is what we in 2016 called a full-stack developer."

In 2017 this didn't happen. The T-Shaped employee and the Full-stack developer are found in small organisations. Large enterprises still have a culture based on decades of functional hierarchies. Contracts are still based on roles where T-Shaped and Full-stack have yet to find their spot. Unless agile transformations are no longer considered to be merely an IT and even just a software development thing, it will become very hard to get into cultures where teams are considered to be the atomic entity in product development and instead of roles and responsibilities, tasked are performed as activities.

#10: Agile in, Waterfall uhm... also in

"Well, agile is finally in and is going to replace waterfall projects in those organisations where there is an active movement towards agile. Which nowadays are the majority of enterprises. These organisations are heavily invested in dropping the traditional practices and adopting new, more business value oriented practices."

In 2017 I saw more and more organisations realising that the typical waterfall projects can actually be done in agile ways. This notion is actually causing the existence of waterfall to be questioned. Do we still need waterfall? No, not at all. But we still need large projects. In 2017 I saw a realisation by many managers as well as architects that large project and waterfall are not different words for the same behemoth, instead there is no a clear tendency to actually do large projects by applying agile practices and waterfall seems to be relegated to only tiny projects. Ironic, but pretty awesome.

This was part two of a two part on a quick glance on my predictions of 2017. Yesterday, I have posted part one of the series and see about how the first 5 predictions turned out. Next week will be about my predictions for 2018.

I hope you enjoyed this post. Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

The Arc-E-Tect predictions for 2017 - In hindsight [1/2]

Last year, like every year, I did some predictions on what would be in and what would be out in 2017. But unlike other years, last year I actually posted those predictions on the internet.
Before I start with my predictions for 2018, I will go back to my predictions for 2017 and see how things turned out.

#1: Microservice in, SOA out

"In 2017 people will start looking at Microservices as something that is useful and way better to have in your architecture than services. So a Microservices Architecture will replace Service Oriented Architectures in 2017."

With a massive transition towards agile practices and organisations embracing scaled agile frameworks, it has been inevitable, the Microservice Architecture (MSA) has been broadly embraced. Or has it?

In 2017 I've seen that those organisations that require true agile concepts in practice in order to be(come) sustainable also embraced MSA as the architecture of choice. The change in mindset that is required for MSA to thrive in an IT landscape and an organisation itself for that matter turns out to be more encompassing than mostly thought. I've seen it fail in those organisations that merely do agile, and succeed in those situations that are agile. Yes, MSA and Agile are going hand in hand.

#2: API's in, Webservices out

"Okay, in 2017 we'll feel ashamed when we talk about web-services and SOA. Instead we'll talk about API's. This is closely related to my first prediction on Microservices, which you can read here."

Here I can be short: There's hardly any talk about web-services anymore. It's all about API's nowadays and that has been the case for the better part of 2017. Over the course of 2017 the notion of API's also shifted from merely glorified web-services towards true business services.

#3: Application Architecture in, Application Model out

"Yes, in 2016 I've been confronted with application models. Again and again I have been slapped with models of applications and yes, I've been on the other end of the slapin' stick as well. Shoving application models into other people's faces. Stuffing it down their throats, making them, no forcing them to understand."

Unfortunately this prediction didn't come true at all. Although it depends on how you look at it. In 2017 I've been in more discussions than before about Application Architectures, although in most cases people were actually talking about models. I guess the terminology is out of vogue, but a lot of architects still have a hard time to use the correct terminology. Still, to me the Application Model isn't out and the Application Architecture isn't in. Just yet. Probably with a more widespread adoption of MSA, we're bound to ditch the model and embrace the architecture.

#4: Internet in, Intranet out

"So the internet will be in, and no longer will we consider the intranet as the context in which our software is running. Talk with any cyber security firm and they will tell you that security has become a real issue since computers got connected. Networks are the root of all evil when it comes to viruses and the likes. The larger the networks, the bigger the problems. And with heterogeneity the number of threats only grew, probably exponentially."

This so turned out to be a correct prediction, and like I envisioned, one of the main drivers has been security. And the lack of it, in many cases.

In most environments I've been working in and with over the course of 2017 there was a real notion that no longer was it affordable to not consider security on an application level and assume that applications could be accessed from the internet. Even when that wasn't supposed to happen. Finally we know that assuming the network to be secure is an assumption that really does make an ass of u and me (assume -> ass-u-me)

The good if not best aspect of this is a security-by-design mindset in most if not all people involved in product development.

#5: DevOps in, Scrum out

"I can be very short about this. Business has finally come to understand that IT is not something that enables them to deliver new products to their customers but instead IT is what they deliver to their customers. IT has become a product, and therefore an immediate business concern."

In 2017 it turned out to be not that short, unfortunately. What I've seen happening is that unless agility is a true business concern, a matter of business sustainability, DevOps is not something organisations want to embrace. Although this is primarily a matter of large enterprises, those with seemingly enough money in the bank to linger a while longer before feeling the need of being able to wart of the threads of start-ups and their agility.

This was part one of a two part on a quick glance on my predictions of 2017. In the next couple of days, possibly tomorrow, I will post part two of the series and see about how the remaining 5 predictions turned out. Next week will be about my predictions for 2018.

I hope you enjoyed this post. Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

Microservices on steroids, getting from just agile to business agile [part 2 of 2]

Summary

This is how you get Microservices on steroids; apply them in a Policy Oriented Architecture. Meaning that you need to put a policy handler in between a service's interface and its implementations. The reason why you want to do this, is because you want to provide agility to your business and not just have an IT department that's agile. Or in fact have an IT department at all for that matter.

It’s time for part 2 of a two part series on Microservices on steroids. I already had part of the post ready but today I was in a presentation on the topic of Continuous Delivery and Architecture and one of the aspects that came up was about ‘Product definition’ and I figured that I needed to put that into the mix as well.

The conclusion of the previous post (Microservices on steroids [1/2]) was that adopting a Microservices based architecture, which pretty much is an SOA the way it was meant to be, will only get you technical agility. As you can read in my post on Continuous Delivery not being something for the IT department only (The Continuous Delivery IT Team Fallacy), technical agility is only part of the story and in fact has limited value if you’re not targeting business agility.

But let’s first consider what Business Agility exactly means. It is in fact exactly what it literally means, agility of the business. The main difference with the ‘normal’ agility is that it allows the business to change its course rapidly without a direct need of involving IT to make this change. So without needing to worry about whether or not IT can handle these changes.

And it allows the business to respond to changes in its marketplace when and where necessary with the help of IT. So business agility is an extension of technical or IT agility.

An example of business agility is for example bank XYZ that issues debit cards to its customers age 16 and up. One of its competitors, bank ABC, is in a marketing campaign targeting teenagers and issues debit cards to 12 to 16 year olds when a letter of consent is signed by their parents or legal guardians. Our bank XYZ will lose out on a large customer segment when not also addressing these teenagers. So it will need to issue cards to them as well. Business agility is when bank XYZ can do so, without major activities that need to take place in order to change course. Think about being able to make this business change in 1 two-week sprint.

From experience I can say that an agile business requires an agile IT that has its architecture on track. An SOA, preferably based on Microservices, is the best basis for this. If you ask me, I would stay away from centralized components like single ESB installations for an organization, shared database clusters or API management solutions that are centrally governed. It’s not so much a matter of me disliking these technologies, which I really don’t. It’s about the centralized governance issue, which results in shared resources, which seriously hampers independence. Centralization also means, artificial, restrictions in autonomy. In effect, it means that you reduce your ability for agility.

Now, don’t get me wrong, centralization doesn’t have to be a bad thing. In general it allows you to limit your costs by benefitting economies of scale. It’s a way of improving profitability by reducing costs. There are many situations in which this is the best way to address profitability. Especially in areas where business functionality is established and there is not really a need any more to figure out your product-market-fit, focusing on cost reduction is good. Within the same organization you also want to be able to improve profitability by increasing revenue. This is especially true for those situations or products for example where product-market-fit is a challenge or when the business scope of a product is changing or increasing still.

 

Think back a couple of posts where I explained my stance on Gartner’s Bi-model IT (The BI-Modal Misconception...). The two situations I mention are the closest thing that comes to Bi-model IT that actually makes sense. It’s where the (business) need for change and agility has diminished and functionality is stable vs. where the need for change and agility is absolutely there and business survival depends on it. There’s not a single thing that relates to risk or quality or ability. It’s about the need to be agile or not that defines the need to become agile or not. And typically, Mode 2 is revenue increase focused where Mode 1 is costs focused. Gartner’s emphasis on legacy transforming to the digital world is not relevant at all either.

Back to business agility. This is where the business is agile as established before. And like I stated earlier, this requires an architecture in which the different components are independent and can be independently deployed. Where ‘components’ are ‘products’. Independently deployable components are almost the same as Microservices. And although Microservices is a buzz word, it does have significant merit to base your architecture on Microservices in order to deliver agility to your business. It’s the perfect foundation for business agility.

Which gets me to the point of my presentation on the topic of Continuous Delivery and Architecture I mentioned earlier.

It is a common misconception that Agile and Architecture don’t play well together. Which is over course utter BS, and if you don’t believe me, I posted about this not that long ago (Product Owner and Architect, Agile Tag Team). Agile and Architecture play extremely well together as long as the Product Owner and the Architect(s) play well together. Especially when it comes to Microservices you need the domain architect or the business architect or both when you have that luxury. It is the architect’s responsibility to understand the business domain and together with the PO define what products make up that domain. Every product is a very likely candidate of either become a Microservice in its own right or becomes a composition (or constellation) of Microservices. And the Product Architect, which you might know as being the Application Architect, Project Architect or Solution Architect, is the single one person that together with the PO defines where the Product ends and the Platform on which it runs starts. The Product Architect therefore is the person that defines the (technical) boundaries of the Product and the domain architect defines the product’s place in the IT landscape. And like I said, a product is a Microservice or constellation of Microservices. Ideally, of course. Domain architect and Product architect should also work closely together in defining what API’s to consume and API’s to provide. Again together with the PO since it’s all about the PO’s product.

In this situation, we have a nice decomposition of a product or several products in interfaces and implementations. And that is exactly what we want from an SOA, especially one that is comprised of independently deployable components, services if you will. Or even Microservices.

Once all interactions between products and within products are based on interfaces, we can talk about a true SOA.

Now here’s an interesting detail. Interfaces are nothing but documentation. Be it fairly intelligent documentation or rather very usable documentation, but documentation nonetheless. And everybody that thinks an interface is something else is wrong. We don’t put an interface in an architectural layer because that doesn’t make sense. The same goes for API’s. An API is also nothing more than documentation that conforms to specific standards. Of course it’s a bit more than just a document, but really, just a little bit more. An API without an implementation is nothing and in fact you can’t actually deploy an API, nor can you an interface for that matter.

Having established that, it’s time to start the confusion and get on with it big time.

So, since an interface is nothing but documentation and it’s something you can’t really deploy. In order to provide business agility through an SOA, we need to put something in between interfaces and their implementations… But before I venture into that area, let me explain how you get business agility through an SOA.

We do this by not implementing an SOA but a POA. Where an SOA is a Service Oriented Architecture, a POA is a Policy Oriented Architecture. Basically this is an architecture in which you have services separated into an interface and one or more implementations of the interface. All implementations are available concurrently and based on policies, one of the implementations is selected. The policies are business policies and governed by the business. Business policies are a fancy word for business rules.

An example of such a business rule is that when transferring money between a customer’s own accounts there is no need for additional validation of the transaction once the customer is signed into her internet banking. Another rule is that when transferring money from a customer’s own account to an account of another customer, an additional signature is needed. Both are financial transactions, but based on the context, different business rules apply.

Policies are context driven, which amounts to the notion of under what circumstances a service is consumed as that determines the implementation of the interface to be used. Contexts can be everything imaginable. Within the example of the transaction as mentioned above, it can be the amount of the transaction, the age of the customer, the time of day, the geographical location of either customer in the transaction, the type of accounts, the client device used and the way a customer was authenticated. Or any combination of these contextual parameters or something completely different altogether.

In a POA, there is a clear distinction between interfaces, policies and implementations. The policy sits between the interface and the implementation. And it is business definable, meaning that the business rules can be changed without changing the interface and existing implementations. Either resulting in implementing policy compliance, i.e. some software is developed that implements the functionality to comply with the policy or by implementing another implementation of the service.

Some 10 years ago, Aspect Oriented Programming (AOP) was hot in software development country and although this way of software development is more or less forgotten, it is exactly what POA (see the similarity in acronyms?) wants to establish. By injecting new code, or rather new aspects of an implementation policies can be changed.

The problem with AOP was that was not trivial to do and it was hard to debug once there was an error in the functionality. AOP is also at the class level instead of the service level, and it is far from accessible to non-developers. Still the paradigms of AOP are what POA intends to deliver. A more flexible way of changing nuances in an interface’s implementation.

As you can understand is that the role of the architect and the PO is very significant in POA. The Product Owner needs to understand where policies are to be in place from a business perspective. Not every interface needs policies, and sometimes you don’t even want to have the ability to change a policy without strict governance. The architect, and typically the Product Architect, needs to define how policies are implemented. Since there are as many ways of implementing policies as there are use cases for policies, it is important to at least within the context of a product the same mechanism is used. This is where the architect comes in.

The power of an API, or a Microservice or even a normal service, is so much more when POA aspects are applied as the Product is enabling or rather facilitating business agility. Especially when the used framework for policy definition and implementation is one that allows for doing so by a non-developer. Unfortunately there are not that many frameworks available, most solutions are based on Case Management solutions or consist of glorified Business Process or Business Rules engines. Centralized solutions that require specific expertise to operate and maintain them. Consequently, the autonomity and therefore the agility of the teams and business users are limited. This is where this post more or less started off from. Centralized solutions are limiting agility, and when you want to extend agility to the business, you should limit it in the technology.

So this is how you get Microservices on steroids; apply them in a Policy Oriented Architecture. I hope you liked this post and feel free to comment and discuss. Although the POA is not new, you don’t see it applied that often if at all. I don’t think it’s a matter of complexity or a technology issue. The challenge is typically in that business and IT need to be considered as one and not two separate departments.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contactlist. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

Microservices on steroids, getting from just agile to business agile [part 1 of 2]

Summary

Agility is important, and business agility even more important. With this, the architect has to play a crucial role by defining the right principles, commandments if you will, and seeing to it that the product development teams live by them. Principles like KISS and Independent Deployability are two principles not to mess with. MIcroservices are an important and extremely helpful aspect of this, so is continuous delivery. But you need to get those microservices on steroids in order to be able to move from just agile to business agile. In this first of a two piece post, I am laying the ground works for explaining how you can facilitate business agility in your architecture.

Since you’re reading this post, it’s likely you’re an IT architect or have to deal with one every once in a while. Meaning that you’ve been confronted one way or the other with Service Oriented Architectures (SOA). Maybe you’re even a proponent of this architectural style.
I’m not going to explain what an SOA is, there’s quite a few very good sources available online explaining what it is and why it’s a good way of building your IT landscape, or not. As always, there are just as many proponents as there are people against and SOA. Over the years I’ve grown to like the premise of an IT landscape where fairly small pieces of functionality can be strung together to realize business functions. And with the advent of RESTful web-services, the premise of a true SOA has become more prevalent as ever.

Nowadays all you see and hear around you are Microservices, even though there doesn't seem to be a clear definition that everybody feels comfortable with, everybody is doing Microservices. One of the most interesting definitions I heard is that it is a really small service. But not as small as a nano service. But just to make sure that you know what I refer to when discussing Microservices, here're two definitions I like to use. The first is from Wikipedia, which in and by itself is no guarantee for it to be correct, but at least it's a common definition:

"Microservices is a variant of the service-oriented architecture (SOA) architectural style that structures an application as a collection of loosely coupled services. In a microservices architecture, services should be fine-grained and the protocols should be lightweight. The benefit of decomposing an application into different smaller services is that it improves modularity and makes the application easier to understand, develop and test. It also parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently. It also allows the architecture of an individual service to emerge through continuous refactoring. Microservices-based architectures enable continuous delivery and deployment."

The second definition is form Martin Fowler, which is in essence the same:

"In short, the microservice architectural style [1] is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies."

Microservices are not easy to develop, much like API's, they require a rather typical level of maturity of a development team to develop them. The hard part of Microservices is typically the "independently deployable" aspect of the services and then in particular maintaining that independence of time. Let's call it sustainable independence.

What I usually see is that over time the architecture becomes convoluted, resources entangled (the whole HATEOAS thingy is not trivial at all) and adversity against rearchitecting the IT landscape to get back to independent microservices.

To develop them,  the Microservices, you need to invest. In general, a Microservice is way more expensive to develop than a regular service. Much like an API being a lot more expensive than a simple service interface. So I would argue that there needs to be a business case to capture some functionality or resource in a Microservice.

Still, Microservices are a good thing to look into, if not only to be able to dismiss them. But more importantly, they are very helpful in becoming agile. I would almost argue that in order to be agile, you need an SOA based on Microservices and API's. Having said this, I need to emphasise that we're still talking about agility of the product development team, the IT people.

Mind that having an architecture with highly independent units of deployment is a blessing for all of us that want to do Continuous Delivery or Deployment for that matter. It makes life so much easier and as an architect you should really make sure that independence of components, services or units of deployments, however you want to call it, is a prime-directive to all your developers, great and small. It should be up there on the wall with the rest of your architecture principles. First of course you mention KISS and second the declaration of independence for your components.

But as I said, agility so far is technical agility. You've designed your software, your product, for change. Which allows you to be agile. Your PO (Product Owner) wants something different, you can change your code easy as can be. But when you think about it, it won't make your business more agile, it will make your IT more agile.

There's quite a bit more to be done before your business gets its agility... more on that in the next part.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contactlist. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

Special thanks to Sytse for pointing out one or two textual errors and helping me correcting them.

"We develop our frontends in Angular" is not an Architecture Principle

Did you ever hear that story about that architect who was telling the developers how to develop their solution? Badabing, badaboom!!!

Yeah, I know it’s a bad joke, there’s no laughter there. I better not aspire a job as a standup comedian. Thing is though that there’re still plenty architects out there that are not only defining architectures, but also tell the developers how to develop that architecture. It amazes me why this is the case.

I know that not that long ago I told you to ditch your architects when they are busy creating roadmaps. But actually you shouldn’t because there’s a real legitimate reason for defining roadmaps. The reason being that they’re an excellent tool. So ditch the architect when roadmaps are what they deliver, because those are the ones you want to get rid of.

Now there’s another breed of architects out there that I have a hard time to understand. I really think that architects should be architecting. That’s what their roles are, just like developers should develop. People should be doing what they’re supposed to do, and preferably spend some time outside their boxes while thinking about what their job really is.

For an architect, the sole raison d’etre is to have answers when asked questions and at the same time be enigmatic. Well, maybe that last part should translate into being helpful to whoever is asking the questions to get the answers themselves.

The architect, as I see it, is the one that defines the boundaries within which the developers can act freely while being confident that there’s no harm to the overall integrity of the enterprise or the organization for that matter. The strength of the architect, the real power, is that this can be done by defining Architecture Principles. These are extremely powerful because you can link them directly to your organisation’s mission, vision and strategy. Well if you can’t than you should reconsider your principles. But if you can, you’ve got your business justification for your principles and you won’t need to define the business value of those principles either. But just like any power-tool, they’re not trivial and to formulate the right principles you really need to be ready to spend some serious time. Furthermore, you need to limit the amount or principles to a maximum of 10, just like the 10 commandments. More will be hard to remember by heart and live by, which is what you should expect from your developers. Less is better, but beware of the fact that you should cover all the important bases in your environment.

There’s another tool that’s really powerful, which is the reference architecture. Which in fact is a model, one of the good kind as you can read here. A reference architecture is a model of an application that meets all the architecture principles. But beware; It’s a model, not an architecture! It’s a visualization of what the architect had in mind when he defined the set of architecture principles. And more importantly, it’s one of many possible models.

So as the architect defining the reference architecture you should be aware that everything in it, can, should and will be taken with a grain of salt. And as the developer you should understand that the reference architecture is just a tool for your architect to illustrate what was meant by the architecture principles, and definitely should not be taken literally.

Ah, so what about the joke, the bad one?

Well, there’s this group of architects that don’t understand the difference between architectures and models and instead of defining architecture principles and reference architectures, they tell the developers what and how to develop such that they comply with the principles.

There’re some serious problems here. For one, the architect omits the definition of architecture principles for whatever reason. Likely because it’s too hard, or they don’t see the benefit. Shame on them! Then there’s the case where the architect ignores the fact that the developers are very likely more proficient in developing than the architect. Oh, and let’s not forget about the fact that architecture principles are almost timeless, or at least stick around quite a bit longer than the typical technology, tool or framework. Yet, this architect dismisses this simple fact of IT and at the same time doesn’t keep the construction manual, because that’s what it is, in line with the current state of IT affairs.

For example, I discussed a couple of years ago, where ‘discussed’ is a euphemism for ‘verbal fighting’ with some fellow architects in some kind of architecture board that it made no sense to dictate that a multi-tier architecture should always be deployed such that each tier should be on separate infrastructure. Instead the application should be complying to the fact that data access logic not to be mixed with business logic nor with presentation software. Sometime ago I had a discussion, which was a real discussion, about whether or not we should define in reference architecture that either Eclipse or Visual Studio should be used for coding. Uhm, I think not, thank you very much. And yes, this was a real discussion.

There’s the discussion about what part of an application is developed in what programming language? We had the discussion at a startup I was Chief Architect at. Guess what, the developers made those decisions, I just stated the principles that I wanted them to comply with.

So what it actually boils down to is that architects need to govern, and that should take up most of their time. They should safeguard the integrity of an organisation’s IT landscape and make sure that in all cases the applications or rather the products in that landscape can comply with the architecture principles. The architect should promote and facilitate an unambiguous way of working among teams. The architect should also be concerned about an ubiquitous vocabulary within the organization when it comes to business and that it is translated by IT consistently.

The architect should not dictate that a particular programming language to be used, that a particular infrastructure is to be deployed to or that specific tooling is to be used. And in those cases where there’s a good reason for the architect to do this anyway, than it should be implied instead of explicitly dictated. In case all development is to be done in Java, the architect should ensure that only Java developers to be hired. In case everything should be hosted on Amazon AWS, the architect will have to pave the way for the teams to use Amazon AWS, making it the favorable option for the teams of the many hosting alternatives they can choose from.

As you might have guessed here, is that the architect I’m referring at is the enterprise architect or domain architect. Definitely not the application architect or the solution architect. The latter are both very much there to device a solution for a problem. They’re more like designers than architects in that sense. Which brings me to another little piece of personal amazement. Why are there so many large organisations that have domain architects on enterprise level whose domain has nothing to do with the business, but are strictly technical? Probably this is some remnant from the time that we considered IT to be a cost center and centralization of IT was the holy grail for many a CIO. Nowadays where we see IT as a product, something that is not just setting us apart from the competition but an actual product, there’s hardly any room for the these architects, at least not on an enterprise level. They should be working closely with products teams from the same product portfolio. Become Product Portfolio Architects, safeguarding the integrity of the IT landscape of the portfolio. Is it a demotion? Definitely not, it just means that they’re relevant again.

To paint a little context here, I’ve been that architect I’m complaining about. Emphasis on ‘been’. I enjoyed the technology too much to let it go, but as the domain architect I wasn’t allowed to work on the software, so all I had was telling the developers what to type. Initially I got away with it, because I was at least one of those architects that actually could still code. But really soon thereafter it became apparent that I hardly had enough knowledge or experience to really do the work. To me that was a real eye opener and I had to good fortune I was working with people that didn't hide the truth from me. They expertly conveyed to me that yes I was more than a decent architect, but as a programmer I really wasn't up to it anymore. And as much as that hurt at the time, it allowed me to focus on architecting. I still code, because I still think I need to be able to understand and to know that things might work. But as an architect, especially in an agile world I find myself more and more facilitating, communicating and coaching. Working 3 sprints ahead of the developers, not because I'm that a fast thinker, but because being an architect I need to be able to set directions. Let the team worry about how to get to our destination, be their own satnav system, and me the architect am more and more the person controlling the system's settings. Controlling whether or not to allow ferries, toll roads, to prefer back roads, go for the scenic route or the fastest or most economic route.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contactlist.But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

Principles of the API-First paradigm

API's are weird puppies. They've been around since forever and whole battles have been fought in courts to gain access to API's. This was in the old days when having a platform with an API for 3rd parties to develop for your platform meant that you had a lot of leverage over those parties. Not providing early access to them meant that they would be late to the party and all the sweets where already shared between the early guests. Having an API meant having power. Not providing access to your API resulted in lawsuits because you were a monopolist, using your position as the platform owner to bully your competition into obliteration.

Nowadays the situation is completely different. Having an API means having a future. Nowadays to survive as a business you need to have a platform, which in turn means you need to have an API. Unless you have an API, one that suits the needs of 3rd parties and that is easy to use, you're doomed. You'll be just an Application Vendor, and toast before you can say "I wish I had built a platform". An excellent book on the topic is "The Age of the Platform" by Phil Simon.

As you can read in my previous post, API's are the hardest thing to develop. It always helps to have some guidance when you're up for a difficult task, and often principles help you getting this guidance. There're quite a few other guidelines out there on the interweb that are more than helpful. Below are 5 principles of the API-First paradigm. Followed by 5 derived principles and at the end of the post, I'm listing a couple of more than interesting resources on API design.

One final note before I go into the principles. An API is not much more than an interface into the depths of your platform. It discloses your platform and makes it possible for others to leverage it for their needs. It is therefore critical to have comprehensive documentation for your API. Now comprehensive doesn't mean that you need a big fat document describing everything you can think of that might be remotely related to your API. Typically you will need to document the technical interface of the API, i.e. the call signature of the API, including the input and output parameters of the API.

In addition you will want to document it's behavior exhaustively, i.e. only develop the behavior your documentation mentions. It makes perfect sense to use BDD (Behavior Driven Development, see my post on the topic) for this purpose. By doing so, your users know exactly what to expect from the API and it allows them to create stubs that behave the same as your API without the need to have access to your API's implementation. Finally it helps to have a simple example on how to use the API without making any assumptions as to why one would use the API. This could be through showing the relevant cURL statements.

Enjoy your reading…

Principles of the API-First paradigm

The API is the product.

It is owned by a product owner, and the responsibility of a product team. A team is responsible for it, from cradle to grave. This team is responsible for managing and ensuring its proper operation. Don't consider an API as part of another product, something that's a by-product, because it means that you won't treat it with the proper dignity and soon you'll find yourself alienating from your users.

The API makes no assumptions about its Consumers.

It is robust and resilient enough to thwart of any consumer with bad intentions as well as ignorant ones that have no clue about what they're doing. All this without compromising the service level to all well intending consumers. Mind that in most cases you're working on an API while making assumption as to who and why they'll use your API, only to find yourself in a situation where your users are all but the ones you envisioned, and all using your API for completely different reasons than why the API was created in the first place.

The API is suiting the needs of the Consumer

The API is developed only when there is at least one consumer and it is tailored to suit this consumer, by this the API is creating business value through its consumer. Don't be so arrogant to develop something that's supposed to be an API without having at least one consumer lined up to actually use it and tell you how great it is. Apart from the fact that working on something that is not being used is a complete waste of time, it also means that you're not really understanding the fact that you should not make any assumptions about the API's consumers. No assumptions at all.

The API is built to last

An API is never breaking compatibility with its consumers, although it may rely on the paradigm of 'Ignore what is unknown' to allow for extending the result. API's, other than the humans that build them, never break a contract. Period! This is to say that they will respect contracts, will have addendums to the existing contract, or in those cases when the API, which is after all a product, is no longer adding value, they will void the contract, but obviously respecting the contractual notice period. On a side note, it is fair to assume that its consumers abide by the convention that they'll ignore what they don't understand or know, hence the fact that you can add addendums to existing contracts without actually breaking the contract.

The API is idempotent.

At all times, in any situation, always. It doesn't matter what you do, how you do it or when you do it, when you call an API it always behaves exactly the same. Meaning that its result is always the same and therefore perfectly predictable. The order in which API's are called doesn't matter, as long as you put in the same values, you will get the same results.

Architecture principles for the API

The API is clear in its error reporting

Errors are either caused by the API implementation, the API consumer or something unexpected. It makes perfect sense to stick with the conventions of HTTP errors, why invent something new, when the existing is fitting your purposes perfectly. But whatever you do, make a clear distinction between these three causes for errors.

The API is always managed.

An API can be managed independently from;
Technical level, i.e. it can be accessed securely and consistently throughout its lifespan -> Scalability, Availability, etc

Application level, i.e. consumers can use the API based on a published interface that is behaving per this interface -> Interfaces, authentication/authorization, etc.

Functional level, i.e. users of the consumer can be provided access to the API based on the agreements made -> Throttling, metering, monitoring and reporting, etc.

The API can be implemented independently of its interface;

They are truly decoupled. Multiple implementations can co-exist and determining which implementation to choose for each individual call can be context driven. Mind that the interface should be technology agnostic, or at least as much as possible, and in no way should the interface disclose anything about the implementation other than that it should explain how it behaves.

The API is technology agnostic.

The interface can be called by a consumer of any technology without any assumptions of the technology used in the API implementation, and vice versa. This is pretty much an extension of the previous principle, but important enough to mention explicitly.

API's are like dogfood.

The API is used by the team responsible for it in the same use cases as the consumers for which the API was intended; "Eat your own dogfood" paradigm. Always make sure that in case you need functionality that one of your API's is providing, you're calling that API yourself and making yourself dependent on the API. Only this way you understand your API, and more importantly you'll be very cautious about how you treat your API and therefore how you treat its users. After all, you'll be one of them.

Interesting Reading

• Google's API design guidelines
• 18F API guidelines
• W3C Web API Design Cookbook
• CISCO API design guidelines
• Haufe API style guide
• Heroku API guidelines
• Paypal API guidelines

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contactlist.But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

API Management in Azure using Microsoft's solution - Resources and Templates [2/2]

This is a series of posts regarding the topic of API's and API Management on Microsoft Azure. Not all posts in this series are directly related to Microsoft Azure and their implementation of API Management, that is intentional. The series also explains about API's, about creating API's and about what it construes to in order to manage them, conceptually and in reality. The reason for this series is that over the past 12 months I've come across many articles on the web, have been in many discussions and advised various clients of mine on this topic. Sometimes discussing with fellow architects, other times with vendors, still other discussions where with developers and managers. But the overall situation has always been that none of the people at the other side of the table had a full grasp of what developing API's in an organisation means, what it entails to manage them or what should be worried about when deciding to go API. I hope you enjoy reading these articles, and when you feel like it, comment on the articles. I always take serious comments serious and will reply to my best effort.

This post is the last post in of a series on API Management using Microsoft's solution.

API Management is not the same as an API Manager. In short, API Management is what you need to do with your API's, and an API Manager is something with which you can do this. There are a lot of different solutions that allow you to implement API Management, one of these is Microsoft's solution on Azure. The tricky part is that when you start reading their documentation on API Management you'll read little about how to actually implement API Management in your organization using their solution. It's all very conceptual. This shouldn't be a problem since the concept behind API Management are more important than the actual implementation… until you want to actually implement it.

Read the previous posts on the topic to learn more about the concept, continue reading this post to learn more about how to use Microsoft's solution in Azure, their cloud platform.

Resources and Templates

Finally, resources and templates, the bricks and mortar of the cloud. In the cloud you're typically dealing with resources. An infinite amount of resources, or at least that's how it feels. Everything in the cloud is, or should be a resource. Some are very basic like computing power, storage and networking. Others are more comprehensive like database, firewalls and message queues. And then there are a quite a few that are truly complex and very useful on a high level, for example directory services.

The cloud, being what it is, is like a normal infrastructure on which you host something that generates value for your business. Hence everything you need to run business applications in a traditional hosting environment, you also need in a cloud environment. Obviously there are significant differences between traditional hosting platforms and the cloud, but when you don't take too close a look, you're likely not to see these differences.

So in the cloud you also need to define systems, attach storage, put a firewall in front, put connectivity in place etc. You can do this by hand every time you need an application's infrastructure. Typically through a portal by clicking your way around and assemble the infrastructure. But more sophisticated and a way better practice, is to define the infrastructure in a text file, typically JSON for most cloud platforms, and use the cloud vendor's tooling to create the infrastructure based on this file. As such, the file becomes a template for a specific infrastructure setup you need. By providing a parameter-file you can externalize specifics of the infrastructure. For example the URL's to locate a web-service can be defined in this parameter-file to distinguish between an infrastructure intended for testing and the same infrastructure intended for production runs.

The particular template is called a resource template, it defines which resources are needed and how are they specified in order to run a business application.

One of these resources that you can use is an API manager, just like you can specify databases and virtual machines as resources. And here's your challenge.

The challenge is in that an API Manager consists of three parts;

1. Developer portal, used by developers to find your API's and their documentation.
2. Publisher portal, used by API developers and the likes to manage the API's.
3. Gateway, used by applications developed by those mentioned above in 1 to access API's managed by those as mentioned above in 2.

Each of these have their own context and is used by a different group of 'users'. The real interesting part of the API Manager is the API Gateway as it is the component that exposes the API's you've been developing. This is your product. It is the resource that is part of your software. And the thing is; it can be shared or limited in scope to just the product you're developing.

Ideally you would have one gateway per product, because the gateway and particularly the API's it exposes, are part of your product and as your product evolves, the API's that come with it will evolve as well. And of course you would want a consistent life cycle across all components that relate to your product. Since the API gateway is just like any other resource in Azure, the above is perfectly doable. In fact, it is possible to include the API gateway as part of your product's resource template and when you provision the relevant infrastructure and deploy your product on it, the API gateway is provisioned as well.
Pretty awesome, when you're willing to forget that the costs of an API gateway are pretty steep. We're talking about close to €2,5k / month. There's not really a price based on usage. Microsoft is really weird in that when it comes to pricing in the cloud. That whole pay-per-use is not really everywhere in their pricing schemes. I like Amazon better in that regard.

So an API gateway per product is not really an option in most cases I would argue. Instead, I would advise to have a gateway per product suite. In case you have teams that handle multiple products, scope the gateway to such teams, or otherwise scope the gateways to department. Use it as a rule of thumb though and not as the law.

The point here is that you want to be able to have your API's evolve with your products and that you want teams to be as independent of each other as possible. But in addition you want your API's to be operated independent of each other. And this is important. In Azure, API's don't scale, it's the gateway that scales. And you want to be able to be smart about that. Especially when it comes to policies and usage tracking or rather generating value from API's being used. When a team is responsible for the success of its products and therefore the value that is being generated, it becomes obvious that that team would want to be able in control of what is affecting their success.

The alternative would be to work with an SRE approach, where you have a team that's responsible for your platform, including your cloud platform(s). This team would then realize the API gateway for the other teams as a service. The catch here is that this platform team decides where your API's are 'hosted', or rather whether or not you share the API gateway between teams or not. Unless your platform team is really committed and more importantly has a thorough understanding of what API's really are and I mean really understand this, I would advice against this approach. I would oppose it for the sole reason that your API's are the window into the soul of your organization. When the API is not performing well, your organization is not performing well. And especially when you're going API first, and thus build a platform, you're screwed without proper API management.

In case you do decide to go the platform team route, make sure that your processes are completely automated. Including the deployment of new API's as well as new versions of existing API's. My preposition here is that you'll be working agile as can be, deploy to production as soon as you're confident that your software is up to it. Meaning that new software needs most likely new (versions of) API's. Don't make the platform team a bottleneck, so make sure that you're working with them to deploy the changes API's consistently, repeatable and consistently. Better to abide by their rules then put your own in place. Drop the whole platform team approach when they're not providing a 100% automated deployment process for your API's.

Then there's the portals. The developer portal is a tricky one because it provides access to your API's from a developer perspective. You should be really nervous when you're nervous about potential unwanted developers nosing into your API registry. Because it means your security is way, way, way below par. Remember, API's are different from regular services in that they are build such that they make no assumptions as to who accesses them. And unless you've build them that way, you'll be in for some really serious security challenges. That said, there's no reason why not to have different portals for developers within your organisation and developers from outside your organisation. And have API's exposed only to internal teams and publicly exposed API's. Just make sure that this is an exposure aspect and not, I repeat, not an API implementation aspect.

So develop API's as if they're possible accessed by just anybody and their mother. Expose API's to a subset of this group.

Then there's the operational aspect of your API captured in the publisher portal. Here you should take an approach that only the team responsible for an API should have access to the API from a management perspective in an operational environment. In other words, access to an API's policies is for the team that 'owns' the API only. You'll need to take care of that. period.

Mind that Microsoft is rapidly changing their API Management service on Azure. Most likely. as I type this, they're making life easier for you on using the service in your organization. The concepts as I've described still hold though. And as Microsoft will hopefully come to realize that also for API Management a pay-per-use model is the way to go, you'll be able to treat API Management as part of your product instead of as part of your platform.

This concludes my series on API Management using Microsoft's solution. I hope you did enjoy it.

The complete series:

• API-first design
• API's and Architecture, match made in heaven 
• Why API's should be managed and by whom
• Resources and Templates [1/2] 
• Resources and Templates [2/2]

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link to my blog to all your Whatsapp friends and everybody in your contactlist.But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect