How Charming is Laziness?

Long time ago, while I was still a student at the PolyTechnic in Enschede, The Netherlands, I would justify my choice to study computer science, by proclaiming that laziness leads to efficiency. Which of course makes no sense I now know, because it leads to effectiveness. Penny-pinching leads to efficiency. You can read all about it in this post Perish or Survive, or being Efficient vs being Effective. But there's this little concept that results in more efficiency and laziness as well. It's therefore a charm. It is called automation and closely related to that third time.

Third Time Is Automated Principle

One of the key principles at Amazon AWS is that everything must be automated. It's not just that everything should be automatable, but it should be automated.

Whether or not it's an urban legend, but word has it that when you create something that requires manual action, you're out looking for a new job. For a company like Amazon AWS it is clear why automation is such a huge thing. And it is clear as well why they have such focus on API's. API's are how automation across the board is facilitated. But most companies are not Amazon or any of the other cloud providers. Most of you my dear readers are more likely to run your systems on Amazon AWS, Google Cloud or Microsoft Azure, then run those applications for your customers. Probably, your IT landscape is in size not even close to Amazon. Probably comparing to Amazon AWS as the Netherlands compares to the rest of the EU. In most if not all of the dimensions you can think of.

Arguably, the principle of "Automate Everything" doesn't apply to you. I'll leave it up to you to think of one or more arguments why automation is not something you should hold dearly.

Challenge to you: put in the comments a good reason why you think automation is not necessarily needed. I'll make an effort to counter your argumentation as a reply to your comment.
But read on first.

The benefits of automating processes are many. Irrespective of the kind of processes. An automated process is infinitely more likely to be repeatable than any manual process. This results in higher quality since errors will either be made consistently, and can be fixed, or will consistently not occur at all. How compelling is that?
Although the automated and manual version of a process might take the same amount of time to be executed, the automated process allows a person to work concurrently on something else that cannot be automated. And automated processes do not rely on the availability of a specialist to execute the process. So automation makes you and your organisation more scalable.

Not automating processes, even IT processes doesn't make sense. Still, when it comes to IT, we hardly do this. Why?

The situation at Amazon

I've come across many situations where things weren't automated. Worse yet, they could be automated. They were not automatable.

For me this was always an interesting fact to find out. For one, we're in IT and IT is all about automation. In fact, in Dutch we refer to IT as Automation (Automatisering - Dutch). The paradox is that we apply IT all over the enterprise to automate business processes, but when it comes to the IT processes themselves, automation is very likely the last thing on our minds. And when you think of it, that doesn't make sense at all. Walk into a room full of IT people, and just pose the statement that it is hard to understand why we're so good at automating business processes, yet we don't have automation in our own processes. And you'll see at least 90% of in-agreement-nodding heads, the remaining 10% are too flabbergasted with the realisation that this is a true statement. Same statement in a room full of non IT people and the first thing you find yourself doing is explaining why this is.

The fact that IT people are not automating their own processes is tough to explain, and I for one do not have such an explanation. There is an explanation though, for why Amazon AWS's processes are all automated. It's because one of the core principles by which they do IT is "Automate Everything". At a dinner party with Werner Vogels (Amazon's CTO) I was invited to, being the Chief Architect of a FinTech startup, I asked Werner (all attendees were told that we were on first-name basis), how it was possible for a huge company like Amazon AWS to live by these rigorous principles. Everything is an API, Automate Everything, and a few more. His reply was that there were two main reasons why it works. 

1. Senior management all the way up to Jeff Bezos, were openly behind these principles. In fact many of them were mandated by Jeff Bezos himself. 
2. Everybody in the organisation experienced for themselves the validity of these principles.

I asked him to clarify that second reason. According to Werner Vogels, Amazon is dedicating a lot if not all of its time to make its processes impacting customer experience as efficient as possible without impacting customer satisfaction. 'A satisfied customer is a returning customer'. The effects of changing a process is made visible to the whole company, at all times. So compliance to the principles will result in changes to the processes and the effects would be visible. Therefore, the validation of the principles would be continuous. And according to Werner, nothing is as motivating to change your processes than to see the effects of your efforts.

Rest of the dinner I was milling this over, enjoying the food and talk to the other guests.

The situation in the 'real world'

Unfortunately, most of us work at real enterprises. And those two reasons Werner Vogels had given me why IT process automation worked for Amazon aren't obvious in the situations I have found myself in.

For example, the amount of automation in a process within IT is not a metric that anybody is held accountable for anywhere I've worked or consulted. Neither are the benefits of automation part of somebody's accountability.

IT process cost reduction (IPCR) is as far as I know not a KPI within enterprises. Nor is the time-to-market (TTM). The latter often does find itself in another incarnation on reports and dashboards, namely as MTTR, the Mean Time To Resolution. When it comes to MTTR, we do see them on reports, but the MTTR is in hardly any case part of somebody's accountability. Same goes for time-to-market. Although it is always mentioned for any improvement project, it is hardly measured or reported on. It's a project issue in most cases, meaning that an organisation is doing projects instead of delivering products.

The lack of real metrics and if you will KPI's means that from an accountability perspective, there is not a clear person that has an incentive to push automation. And if you read my blogs on a regular basis, you know that I'm big on accountability.
Visibility of the effects of changes to the IT processes is another challenge for organisations. We often find ourselves in organisations that don't have a culture to measure our IT processes. This is especially true for organisations that have been around for decades. Since IT process automation is not pushed, there is no incentive to find bottlenecks in processes or pinpoint areas that are up for improvement. Resulting in the situation where the effect of improvements are not visible in most cases.

The lack of accountability and the rather big hurdle to be taken by IT departments in order to automate result in a situation in which we are just not automating our processes, because it gets no priority on our backlogs or funding in our PRINCE2 budgets. Process automation is collateral. And it's not a matter of not being as large a company as Amazon. It's a matter of not being aware of how automation affects the bottomline. And of not being held accountable for impacting the bottomline.

Third Time Principle

Three times is a charm, or at least should be automated.

In organisations I worked previously, the principle of automation as in play at Amazon was a little bit more pragmatic. One that has worked well for me, is the principle of "Everything done a third time, will be automated", reasoning behind this is that if you do the same thing a third time, it is extremely likely you will do it a 4th, 5th and even more times. The time required to create the automation will be saved by executing the task over and over again.

Time invested is never gained, but can only lead to savings later on.

The Third Time Principle is a compromise, although one might argue that it is a Troyan horse. By adopting this principle, the argument that it creates too much overhead for mundane actions is off the table. Only those actions that are performed repeatedly are automated. Built in justification for the investment needed.

The Third Time Principle is easy to adopt. Since it is a compromise, it can be applied when the push to automation is bottom-up. We often see that engineers see the need for automation since that is where automation will solve a problem and effects are noticed. To develop the automation is often a matter of getting the time to do so. Automation is now competing with other requirements for development time, for priority on the backlog. We all know where the priorities will be. Applying The Third Time Principle will remove this obstacle. Engineers can justify the need for automation and the Product Owner can justify the priority of the automation stories on the backlog.

Continuous Delivery

When striving for Continuous Delivery (CD) and more so for Continuous Deployment (also CD), there is no other way than to automate everything. CD requires a rigorous regime to move every manual task to the left in a process defined western style (i.e. left to right notation). Product development follows the DTAP model. Development is followed by testing is followed by accepting is followed by production → DTAP. In CD we hold true to the paradigm that everything manual is done in D, and TAP are fully automated. Any manual action in T, A and P needs to be automated and the scripts for this are developed in D, because otherwise it can't be done.
So when striving for Continuous Delivery, everything in the product development cycle is to be automated.

Accountability

When process automation metrics are part of somebody's accountability, that person will also be mandated to automate as much as possible. This is for the simple reason that you can't hold somebody accountable for something they cannot influence. Therefore, when process automation is measured through some metric for which somebody is accountable, it is that person's prerogative to drop The Third Time Principle and instead adopt the Automate Everything Principle.

Accountability is a matter of top-down enforcement. It is also a key aspect of culture change. When we want to adopt a culture in which we allow ourselves to reap all benefits of automation, especially our IT processes, we can't get around the fact that we need to revisit the metrics by which we hold ourselves accountable.

Scalability

One aspect that I haven't really touched upon is scalability, although I did mention it briefly. Automation is a key aspect of scalability, not so much scalability on a technical level but organisational scalability. Manual actions always need a person to execute them. The more complex the activities become, the more experienced or knowledgeable the person needs to be. And before you know it, it requires a specific person within the organisation. Because you rely on this person, that person will become the person with all required knowledge, it's a vicious circle. Think about it for a second, and I'm sure that you can think of a process or an activity in a process where you rely on a specific person and you know that person by name. In fact, if you want the activity to be done, you will even call that person because she's among the busiest persons in the company.

When you want to keep activities simple, you need to automate them. The more you automate, the easier it is to keep the activity simple. Hardly anything is more simpler than to have a command like 'do_complex_activity.py' provided that the automation is done using Python. Anybody can run this command provided they have the rights to do so, and when done really properly, anybody can do it at any time, because all the complexity of who can do what when is taken care of by the automation code as well.

Concluding

In the end, you'll agree that automation is what we need to apply to all our processes. At least when it's the third time you're doing the same job. It will improve quality and consistency. It also means that we as a company can scale our organisation without the need to increase headcount. It will be important to monitor the benefits of automation, without metrics it will remain a matter of good faith on the short term, and it will not justify the investments on a longer term. By understanding the benefits of automation and getting insights on where automation has most impact.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect


The text very explicitly communicates my own personal views, experiences and practices. Any similarities with the views, experiences and practices of any of my previous or current clients, customers or employers are strictly coincidental. This post is therefore my own, and I am the sole author of it and am the sole copyright holder of it.

The Power of Accountability

Some months ago, I had a discussion with a friend and colleague of mine and ever since I’m planning to write this post. This post is about responsibility and accountability.

Summarising

Unless people feel accountability and responsibility instead of just writing it down, RACI tables are a playbook for the finger-pointing game and not a tool for continuous improvement. Accountability and responsibility are the keys to successful products. For architectures this means that they need to be owned, maintained and challenged. Constantly. Continuously.

The reason for the discussion I mentioned was that he was asked to write a short report on the state of the IT department of one of his clients. The issue at hand was that there were a number of serious problems with the IT systems and although outages were fixed fairly quickly, the root-causes were hardly ever addressed. Instead people would blame each other. Fervently. After a couple of interviews, my friend concluded that there was a serious lack in governance in all areas of the organisation. And over dinner we were musing over this whole governance thing. And while enjoying a good meal and afterwards a really nice cognac, we more or less stumbled upon two words that I have been using more and more ever since: Responsibility and Accountability.

So that is what this post is about. This time I’m writing about responsibility and accountability and their role in architecture.

RACI explained

I think that you’re familiar with the concept of RACI tables. Tables in which on a variety of interrelated topics it is noted who is R(esponsible), A(ccountable), C(onsulted) and I(nformed). In my experience, the RACI table is typically very much discussed and eventually agreed upon. But hardly ever relevant in practice. Reason for this is that the RACI table is listed in some document and only looked at when there is trouble.

Let’s dissect the RACI for a moment. I’ll do this back-to-front as that is serving my purpose best. Starting with Informed. This depicts all people and parties that are to be informed about something. Could be an design decision, a contract, a change in a process, anything for that matter. For simplicity I’ll stick with ‘decision’. Then there’s Consulted. Where the Informed are merely told about a decision that has been made, the Consulted are asked their opinion about that decision. Their input is weighing into the decision. The main difference between the Informed and the Consulted is the fact that the Consulted have influence. Informed listen, Consulted talk.

Thirdly, there’s Responsible. I know, it should be Accountable, but I’ll address them last. The Responsible are about making the decision based on their own knowledge and understanding, taking into account what the Consulted said about the decision as well. The Responsible are the people that are all about that the decision is a sound decision. They (should) care about quality. In addition, the Responsible are in many cases also responsible for consulting the Consulted and informing the Informed. I agree with this. Then lastly there is the Accountable. Note the plural in Responsible, Consulted and Informed and the fact that the Accountable is not plural. Accountability is not a team-sport! The Accountable is held accountable for the results, the effects, the impact of the decision.

When we look at this in a Product context, then the RACI is as follows: Responsible is about the production of the Product, the quality of the product and the quantity of the product. The Accountable is about the money made (or lost) with the (sales of) product. Now when you factor in the Product Owner, it should be clear that the Product Owner is all about creating value with the Product she owns. Because of the accountability aspect, the Product Owner needs to find the perfect balance between quality, quantity and cost in order to generate the most value... but the Product Team is responsible for achieving this quality, produce the quantity and keep the cost level stable.
It goes without saying that the relationship between Product Owner and Product Team is to be very tight and based completely on trust.

RACI for the Architecture

You're probably asking yourself where architecture comes into play. I hope you've already read my post on the Agile Architect. In that post I am explaining the role of the architect in an agile environment. Just read it if you haven't already done so.

With architecture it becomes tricky, because architecture can be implicit as well as explicit. Implicit architecture is architecture that emerges. It is the Product Team that is formulating the product's architecture while creating and changing the product. As the product is developed, the team will find, define and adopt practices. Some will become best practices within the organisation as teams share their experiences. But as the team is identifying practices that they find supportive to their product development, they will become an integral part of the way the team works operates and there you go; Implicit Architecture. Mind that architecture is not some soft of diagram with blocks and arrows, instead it is a set of rules by which the product is developed. There's a team understanding that once you live by these rules, the world will be a nice place to live. The architecture is a framework of norms and values. Everybody in the team is held responsible for working within this framework. At best, the Scrum Master can be held accountable for the fact that everybody is following the rules of the team. But with respect to the architecture, this is a stretch.
Once these principles are formalised they become explicit. You'll be able to read them somewhere. The Ten Commandments of the team according to which they develop their products. At this point the team is still responsible for complying with the architecture principles, but since they're written down they need to be maintained. Mind that an architecture is something that needs to be nurtured. It needs to be constantly challenged, validated and maintained. And that should be somebody's responsibility. And that somebody should be held accountable for the relevance of the architecture.

Who da man?

Think about it, once you define an architecture, once you write it down, you create the architecture for somebody to be used. You have created a product. And there should be somebody owning that product and somebody developing that product. Somebody being accountable for the value architecture and somebody responsible for the quality of the architecture. Give it some thought and write down who is within your organisation accountable and who is responsible for any of the explicit architectures. And here's another challenge; ask them whether they feel the same way. For argument's sake, let's call the person accountable 'Chief Architect' and the person or persons responsible 'Product Architect'. And now ask them who they think should be consulted and who should be informed, just to make the RACI complete.

Yeah, that last one is tricky as can be. Traditionally and be aware that tradition is still prevalent, users are informed about products. And traditionally business is consulted about the product. Which of course is all backwards. Which product has the best product/market fit? Exactly, it's the product that meets the needs of the users best. And who better to define what product is meeting those needs best than the user. So users need to be consulted, not informed. I repeat, users should be consulted and not informed. That also holds true for the architecture.
Since the Chief Architect is accountable for the architecture, she will have to make sure that the architecture is relevant. That it is continuously validated for its applicability. Thus, the Chief Architect will need to consult with the developers whether their needs for an architecture, for the architecture are met, and that they can apply the architecture while developing the products. Consequently, the Product Architects are responsible for continuously consult the developers about what they need, how they need it, what works and what doesn't, etc.
In simple terms, the Chief Architect is accountable for the product/market fit of the architecture, and the Product Architect is responsible for making sure that the architecture addresses the needs of the developers. Obviously, when there are domain architects or other more over-arching architects, they befall the same faith.

RACI to solve problems

Now back to the discussion with my friend. The whole accountability and responsibility thing. I hope that you understand that unless the RACI is not only written down, but actually implemented, it doesn't mean a thing and can only be used to 'properly point fingers' when things go wrong. But it will not help you prevent problems. This is because unless you feel ownership, you feel accountable, you are not motivated to pro-actively work on improving your product.
So unless people feel accountable and responsible, they will not consult the right people, inform others in a timely matter and will work on preventing incidents. Finger-pointing is the result. And this is also the case for less obvious products like architectures, contracts, privacy policies and for example employee contracts.

Accept the Challenge

Again I challenge you to name the person(s) accountable for the explicit architectures in your organisation as well as those responsible for those architectures. And I dare you to hold them accountable and responsible. And I double dare you to make them consult their users and not just inform their users. Well, actually, start by asking who their users are, and you are likely to be in for a surprise.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

Where's the Agile Architect hiding?

Nowadays everybody and their mother is turning, transforming into agile beings. But where is the Agile Architect? In October of 2017 I did a presentation at ASAS2017 about the Agile Architect. Ever since I have been contacted about the presentation and specifically about what the architect's role is in agile organisations. Because of this, I think it is time to shed some light on this. Thus... Find out in this post.

Summarising

The Agile Architect is this Product Architect that is taking care of the organisation's software product landscape and maintaining its integrity. The Product Architect ensures that based on the input of the Business Architect and the Product Owner the product landscape is sound, adaptable and delivers overall architectural agility.

The difference between doing agile and being agile

The Agile Architect is a rare beast in today's agile organisations. But before I go down this rabbit hole I need to emphasise the fact that I am talking here about organisations that are agile and not those that are doing agile.
The main difference is that when you do agile, you're living by the book, like a religious zealot. You follow the rules, have your rituals and so on. It typically means that you're organised according to the school of SCRUM. Small teams of max 9 individuals, one Scrum Master, a Product Owner. working in Sprints that take typically two weeks, have your daily stand-ups, your retro's and demo's and a backlog of user stories and epics. In contrast, when you are agile, you're taking the lead from the agile manifesto, do the hard things as often as possible, prioritise your activities according to value, have a build-measure-learn loop in place and determine your next actions based on feedback from your users. This is the short version... Obviously.

Independence and autonomy are prerequisite for Agile Architecture

So having established this, I can continue on the topic of the agile architect. Well, just before I venture into that direction, let me explain that in an agile organisation, you also find a strong longing for independence and autonomy. Mind that it is not anarchy that is sought after. The challenge here is that different teams want to move with different speeds. Often this is called bi-modal or multi-modal (you can read about this for example in my post "Let me spell out why Bimodal IT doesn't work"). It doesn't matter how you want to call it, what it boils down to is that different teams work on different products and the organisation's market dictates that each product needs to adopt to changed circumstances in a timely manner, where 'timely' has different meanings for different products. It's extremely logical and also extremely unsexy. These different requirements regarding development speeds requires independence between products and therefore teams. Obviously, whenever there is a dependence, the speeds are linked and in fact agility is lost. Something I'll cover in another post some time soon.
So independence is required here. Autonomy is also important, because if a team cannot decide by itself how to solve problems, how to implement features, how to enable proper operational management, and instead have this dictated by a third party. There is a dependency with a bottleneck and hence agility is at risk. Look for the upcoming post "The Incompetence of the CC..." that will feel enlightening in case you don't understand this right now. But again, it is in fact quite logical how this all works.

So teams should be able to act independently of each other and should have a high level of autonomy, in fact the organisation should strive for the highest level of independence and autonomy that is possible within the context of that organisation. Note the subtlety of the last part of that sentence; that is possible within the context of that organisation. I am explicitly not saying that all control should be dropped and no governance should be implemented.

With the teams being autonomous and independent, the question about the role of the architect arises. Typically the first question I get is whether the architect should be part of the Product Team or not. And my answer is at all times that the architect is not part of the Product Team, where the Product Team is the team that develops and supports a (business) product. Just like the Product Owner is not part of the team, the Product Architect (that's what I call the architect) is not part of the team. This doesn't mean that the architecture of the software is to be neglected, what I am saying is that the team itself is responsible for the application architecture, referring to the internal structure of the software. There are typically a lot of important impromptu decisions to be made during the development of the software where the right decision is determined by the specific context in which the decision is key. The team knows best. That is not to say that anything goes. I'll get to what I mean by this in a second.

The Product Architect is not so much responsible for the software architecture, but for the positioning of the product in the organisation's landscape.The Product Architect is accountable for the integrity of this landscape. The meaning of this is best described through a use-case.
[Read this post on the importance of accountability and responsibility.]

Agile Architecting explained

Consider an organisation's product landscape to consist of several business domains, each consisting of several business products. A business product is a product that is delivered to a user, often a customer of some sorts. The customer typically accesses the business product through an interface. This could be a specialised App, a webpage or maybe a customer-success-agent. This is sometimes referred to as the business front-end, or the customer interface. What is important here is that the customer is not interacting with a piece of software perse, but instead could be interfacing with another human being. Demonstrating that a business product can consist of man and machine. In addition to this, business products can be related to each other. The final step of using one can lead to using another product. There can be a sequence, just like when you go to the movies. You buy a ticket, you get some drinks and snacks at the concession stand and then see the movie. These are three different business products and clearly the buying a ticket and watching a movie are closely related. But independent since the person buying the ticket doesn't have to be the person watching the movie. Think about gift-tickets.
So you see there are several (related) business products. How they relate to each other is part of the responsibilities of the Business Architect(s), sometimes called Business Domain Architect or just Domain Architect. It is the Business Architects that will typically decide to which domain a business product belongs. Buying things could be the Sales domain and watching a movie in the Media domain, or buying the ticket and watching the movie can belong the the Movie domain and the drinks and snacks in the Concession Sales domain. It's arbitrary but needs to be decided upon, this is what the Business Architect does.
Once it is decided to which domain a (new) business product belongs, it is the (new) Product Owner that determines how the business product is taking shape. Think about the MVP. Often the MVP is a first incarnation of a product, but it can be in fact be a completely different product that is used to verify that there is in fact a need, a requirement, for the new product. An MVP could be a simple teaser webpage announcing the new product or poster with a box attached that potential customers can put a card in saying how excited they are to get hold of the new product.
When the Product Owner ties the knot that software is to be developed, something typically decided after consulting the Business Architect and the Product Architect(s) for the products in a business domain, it is the Product Architect that comes into play. This is where the Product Architect is taking the task to maintain the landscape's integrity. First of all, the Product Architect will have to determine whether or not a new product is to be developed or an existing product is to be enhanced or adapted. In case of a new product development, it has to be determined what the best technology is to do this. Here it is important for the Product Architect to understand the product. The Product Owner is a significant source of knowledge and information here. Who'll be the user of the product and how will the user access the product? What are the security requirements and what about compliance aspects? Transactional integrity is important as well. Also the intended life span of the product is important. And obviously the product's relations and integrations with other products is key and for this the Business Architect needs to be consulted. Obviously all architectural principles are taken into account and the Product Architect will define what principles are such that specific practices are to be maintained. Consider a product that does financial transactions, that could mean that specific frameworks and products need to be used for the implementation. Or the fact that the product is going to be used by students after marketing campaigns, resulting in a deployment in Amazon AWS instead of the organisation's own data centres.
The Product Architect, in response to the Product Owner's requirements for the product, will also be responsible for example to define an API that is published as part of the product offering.

Product Architect vs Product Team

By deciding all this, the Product Architect is to some extent dictating the products architecture, and still it is the Product Team that is responsible for the software architecture and not the Product Architect. So how does this work? Remember I promised earlier in this post that I would elaborate on this.
It is actually very simple and all based on the role of Architecture Principles. The organisation will have its organisation wide Architecture Principles, principles that are to be followed at all times. Possibly depending on context, though. I always like to mention that KISS (Keep It Simple, Stupid) is the prime-directive of everybody. By stating so, everybody introducing complexity is violating this principle. As a consequence, the principle enforces the policy to deliver always the minimum that is required, hence helps implementing a LEAN mindset.
Apart from principles that are organisation wide and principles that are applied in a specific context, emphasising that the context needs to be specific, a Product Architect can also define so called 'local architecture principles' that are local to the teams within the circle of influence of the Product Architect. Think about these as local extensions on the existing principles. Maybe because the domain is subject to some specific regulatory context or maybe because the domain is specific for a specific market.
The Product Teams are always to abide by these principles. So although they have autonomy on how to do things, they need to stay within the framework of these principles. They can't just decide to make an overly complex solution, because that would violate KISS.
In addition, the Product Architect can also limit the acceptable answers to implementation questions. For example the Product Architect can state as a principle that all API's are based on Open Web Standards, but in addition decide that all API's that are exposed on the internet are to be based on REST/JSON and API's that are with business partners to be based on WS-*.

Consequential impact

All the above decisions also lead to other decisions. A decision for a specific deployment platform may mean that only certain teams are capable of doing the development. Or deciding that the product needs an API as well as a web-UI may result in the situation that two teams will work on the product, one on the product with the API and the other working on the web-UI based on the API.
Also, envision the situation where timelines are not really up to the needs of the Product Owner, it is then for the Product Owner and the Product Architect to determine the architectural shortcuts to be taken. Using different technologies to leverage the availability of another team for example.

Concluding

As you can see, the Product Architect's role is very challenging, just like the role of Product Owner. But although it is very challenging, it is also very much scoped towards the external aspects of the products. The scope is about the product in its surroundings. Think about the layout of a street with houses instead of the layout of the individual houses. Of course, the type of house, or rather building is a Product Architect's call. Villa's, brown stones or apartments is up to the Product Architect, not the amount of bathrooms, the type of tiles or the kind of sofa's. That's up to the Product Team in response to what the Product Owner can turn into value.

The Agile Architect is this Product Architect that is taking care of the organisation's software product landscape and maintaining its integrity. The Product Architect ensures that based on the input of the Business Architect and the Product Owner the product landscape is sound, adaptable and delivers overall architectural agility.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

What's the story about those other user stories?

User stories expressing what the Product Owner or her Team want or need are bad practice. The PO delivers, never requests!

Summarising

Backlogs containing user stories that are catering for the needs and wishes of a Product Owner or Product Team are faulty. These kind of stories are unable to compete with stories that clearly add value to the business. By putting these stories on the PO's backlog compromise the ability of the PO to be successful. Instead these stories should be listed on a second backlog owned by the Team. This will improve the team's sense of autonomy, the Team's relationship with the PO.

In the past 6 months or so I've been actively working with agile teams on their backlogs. Specifically the Product Owners. As you might already know, I consider the Product Owner role as one of the most demanding and challenging roles within an agile environment. You might want to read my post on it, which you can find here. One of the major challenges I'm facing in working on backlogs is getting the Product Owner, and also the Product Team to understand why internal user stories are a bad-practice.

Internal user stories are those stories the PO wants, or the team itself. Most of the time, these are stories where an upgrade of some technology is needed, where technical debt is taken care of, or for example where the Product Owner wants some reporting from the tooling. There're any number of stories on backlogs where it is the Team or the PO in the 'I as a ...' clause of the story. Look at your own backlog and see for yourself.

You might have read my post "Perish or Survive, or being Efficient vs being Effective" on what agile encompasses in terms of motivation, or drive. When you commit to agile practices, you commit to effectiveness and in pretty much all cases you commit to a value-driven approach towards what has the highest priority.

Considering the above, than you realize that in most cases, internal user stories are not really creating value. Mind that you will have to consider the story and not the motivation for the story. Be pure in thought.
This is because of the simpel fact that the user, or customer requires and the PO (and her Team) delivers. That's all there is to it.

All of a sudden it is clear why a story like "We, the Product Team, want to upgrade to the next version of technology such and so, in order to reduce our technical debt" is a no-no on the backlog. Why? Because the user, nor the customer typically doesn't care and shouldn't care about the technology used.

Don't believe me? Well, have the user dictate the version of Ruby on Rails to be used in the development of an iPhone app and listen to what the Team has to say about this. I rest my case.
Having said that, it should also be clear that there is typically a more than valid reason why a Team wants to get rid of technical debt, even suggesting how to get rid of it as well. And in most cases it is related to the request of the user, demanding better stability, performance or security. The Team's story is an implementation aspect and nothing more.

Think about this a bit and then you're bound to see that when the PO or the Team will create their own stories and put them on the backlog, they will need to compete with the stories of the other stakeholders. Stories that add value, for the simpel reason that the stakeholders will ensure a clear value-add because that is what gets the story at the top of the backlog. (Provided the PO will actually prioritize the backlog based on business value and not things like vanity or volume of a voice.)
The last thing anyone should want is have the team's stories be relegated to the black-hole called "Tail-of-the-Backlog".

One of the most common ways of getting the things done, the Team wants done is by allowing the Team to spend some time on other things than stories. Sort of a budget expressed in time, the Team can spend on their own stories. I like to maintain the 80-20 rule here, as it can be applied in pretty much everything related to IT. So 80% of the time is spend on creating value, and 20% on something that is not necessarily adding value. How that 20% is allotted is up to the PO and her Team. Could be a day a week, could be two days during a two week sprint, or possible a full sprint once every five sprints.

I call it budget, because the PO's budget is expressed in time. She has a team available to her, a stable team paid for, for the lifespan of the Product she's owning. Consequently all she has is time to spend. Allowing the Team to work on their own stories, makes perfect sense. This is because the Team itself knows best on what stories are the most valuable. Whether it's the technology upgrade, the refactoring of old code or investigating some new concepts and their applicability.
Once the Team maintains its own backlog of its own stories, there is no need to compete with the stories of other stakeholders, the stories bound more clearly add stories. Moreover, the Team will more so be empowered to autonomously decide how stories are realized. Deciding when it is best to cut corners and when not. All of a sudden, the Team is not only responsible for the product's quality, but also accountable for the technical quality of the product.

The PO's backlog is one expressed in value creation, with the highest priority set to the story (potentially) creating the most value. Allowing the PO to be value-driven. The PO therefore can maintain a backlog consisting of user stories with clear stakeholders. People, not roles, that can be called upon to verify the product's ability to deliver value. And with the opportunity to ask these people beforehand how value is created, what metrics are needed to validate the product's benefits, etc. And of course, the story will contain the name(s) of the people to explicitly invite to the Team's demo of the product.
The Team's backlog at the same time can be prioritized on the account of what will reduce the product's costs the most. Accommodate for future developments as well as operational aspects, etc.

Maintaining both backlogs also strengthens the relationship between the PO and the Team, putting the emphasis on trust, accountability, facilitation and empowerment. The corner stones of agile teams.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

You know you're the Product Owner...

...when your product's users are complaining and you have to worry about your bonus.

Summarising

The Product Owner's bonus is on the line when users complain! You're not worried about the upcoming appraisals although the users are complaining? And you're not looking for that boat you fancied for so long because the users are giving you great reviews all the time? You're not a Product Owner.

In this day and age of DevOps and Agile, the most coveted job in the world seems to be the one of Product Owner. Never have I seen so many co-workers turn into a PO as recently is the case. Operators turning into Operations PO, testers turning into the Testing PO, security experts fulfilling the role of Security PO. It's amazing to see all these Product Owners mushrooming in organisations.

Understandably, because their original jobs are nearing their expiration date, or so they're let to believe.

The other day I had an interesting discussion with one of the architects of a client of mine. We're discussing a lot these days about architecture, API's, services, data warehouses and other interesting stuff. But this time around he challenged me. Seriously.

This particular client is a typical Project oriented organisation. Projects develop something and once it goes into production and becomes at times business critical, a very efficient department takes over. (Just in case you're wondering why I put efficient in italics, read this post). This architect is part of a department that is making the transition from a Project oriented towards a Product oriented way of working. It's a significant move and absolutely not trivial.

What's interesting is that the general understanding of the necessity of a mandated Product Owner has caught on with this client of mine. What hasn't caught on is that the PO is supposed to be somebody from the business. Take this with a tiny grain of salt thought, as by stating that the PO needs to be a business person, I mean that the PO needs to understands the business in which his products make a difference and generate value. Do you need to be an MBA? Nope, but you do need to understand the relevance of the product for your user.
And this is exactly the issue at hand. All the different so called PO's my friend the architect is dealing with do talk with the user, but do not necessarily understand the relevance of the products they're using. The Operations PO discusses the stability of the product, which makes perfect sense because the focus of an operations person is ensuring that the product is not crashing. One could argue that the relevance of an operations person is the fact that products will crash, which is a bit ironic. The Testing PO is of course focusing on whether or not the product is conforming to the requirements and specifications. This is what testing is all about: Is what has been build, delivering what was intended in the first place. And with all the security incidents, global incidents at that. And with all the new laws and regulations around privacy and what not, the role of the Security PO is cut out, it's focusing on limiting the risks for the organisation by having the products being used by, well the users.

Since all these PO's are doing their job extremely well, the products are up to par and are in fact creating value for the organisation. They surely do. But that is not to the credit of these PO's. The reason for this, is that none of the PO's are concerned with the best product, i.e. the product that is helping the user to conduct business. They are all focused on the product delivering what was intended, namely stability, requirements and security.
I hear your brains churning on this, so let's make an assumption here to illustrate: What if the product crashes all the time, but when it doesn't it removes the hassle of manual steps in a complex process? And although it crashes, data integrity is guaranteed? The operations person won't like it and will very likely take the product out of commission. Why? Because operations is affected when stability is an issue.
So now it turns out that the product is fully according to spec, tests are 100% green but the user will not stop complaining because the product is still not helping to drive business? The tester will not look into this as a testing issue, but as a specification issue. The fact that the tests didn't reveal this major flaw, namely usability. And even when it did, usability being a testable requirement is a novelty. It is with my client's organisation and it is in many others.
Guess you can fill in the problem with the Security officer acting as PO. Consider it a small exercise to flood your brain with some endorphin.

The issue here is that none of these so called PO's is accountable for the success (or failure) of the product. None of them is. And this is what sets the PO apart from everybody else in the organisation:

The PO's bonus is on the line when users complain!

This means that when a accountability of a product's success, i.e. the level if complaints from users about the product, is not with you, you're not the Product Owner. It also means that unless you get a full mandate to make a success out of a product, you're not the Product Owner either.
Don't accept the role of PO unless you get full mandate, which includes discretionary say about the product team, its road-map, funding, etc.

Back to our wannabe PO's, because that's the correct word for them. Their bonus is not on the line, they're not responsible for the product's success. They're definitely not accountable. But that doesn't mean that they're not responsible for making the product a success. Their knowledge, insights, experience and general professional view on the product is invaluable input to the PO to create a success out of a product. The PO shouldn't ask for their input, but when the input is not provided, the questions should be raised. They're not impacted by bad reviews, the PO is. They do have to worry about their jobs, because if the PO can't use them...

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

Microservices on steroids, getting from just agile to business agile [part 2 of 2]

Summary

This is how you get Microservices on steroids; apply them in a Policy Oriented Architecture. Meaning that you need to put a policy handler in between a service's interface and its implementations. The reason why you want to do this, is because you want to provide agility to your business and not just have an IT department that's agile. Or in fact have an IT department at all for that matter.

It’s time for part 2 of a two part series on Microservices on steroids. I already had part of the post ready but today I was in a presentation on the topic of Continuous Delivery and Architecture and one of the aspects that came up was about ‘Product definition’ and I figured that I needed to put that into the mix as well.

The conclusion of the previous post (Microservices on steroids [1/2]) was that adopting a Microservices based architecture, which pretty much is an SOA the way it was meant to be, will only get you technical agility. As you can read in my post on Continuous Delivery not being something for the IT department only (The Continuous Delivery IT Team Fallacy), technical agility is only part of the story and in fact has limited value if you’re not targeting business agility.

But let’s first consider what Business Agility exactly means. It is in fact exactly what it literally means, agility of the business. The main difference with the ‘normal’ agility is that it allows the business to change its course rapidly without a direct need of involving IT to make this change. So without needing to worry about whether or not IT can handle these changes.

And it allows the business to respond to changes in its marketplace when and where necessary with the help of IT. So business agility is an extension of technical or IT agility.

An example of business agility is for example bank XYZ that issues debit cards to its customers age 16 and up. One of its competitors, bank ABC, is in a marketing campaign targeting teenagers and issues debit cards to 12 to 16 year olds when a letter of consent is signed by their parents or legal guardians. Our bank XYZ will lose out on a large customer segment when not also addressing these teenagers. So it will need to issue cards to them as well. Business agility is when bank XYZ can do so, without major activities that need to take place in order to change course. Think about being able to make this business change in 1 two-week sprint.

From experience I can say that an agile business requires an agile IT that has its architecture on track. An SOA, preferably based on Microservices, is the best basis for this. If you ask me, I would stay away from centralized components like single ESB installations for an organization, shared database clusters or API management solutions that are centrally governed. It’s not so much a matter of me disliking these technologies, which I really don’t. It’s about the centralized governance issue, which results in shared resources, which seriously hampers independence. Centralization also means, artificial, restrictions in autonomy. In effect, it means that you reduce your ability for agility.

Now, don’t get me wrong, centralization doesn’t have to be a bad thing. In general it allows you to limit your costs by benefitting economies of scale. It’s a way of improving profitability by reducing costs. There are many situations in which this is the best way to address profitability. Especially in areas where business functionality is established and there is not really a need any more to figure out your product-market-fit, focusing on cost reduction is good. Within the same organization you also want to be able to improve profitability by increasing revenue. This is especially true for those situations or products for example where product-market-fit is a challenge or when the business scope of a product is changing or increasing still.

 

Think back a couple of posts where I explained my stance on Gartner’s Bi-model IT (The BI-Modal Misconception...). The two situations I mention are the closest thing that comes to Bi-model IT that actually makes sense. It’s where the (business) need for change and agility has diminished and functionality is stable vs. where the need for change and agility is absolutely there and business survival depends on it. There’s not a single thing that relates to risk or quality or ability. It’s about the need to be agile or not that defines the need to become agile or not. And typically, Mode 2 is revenue increase focused where Mode 1 is costs focused. Gartner’s emphasis on legacy transforming to the digital world is not relevant at all either.

Back to business agility. This is where the business is agile as established before. And like I stated earlier, this requires an architecture in which the different components are independent and can be independently deployed. Where ‘components’ are ‘products’. Independently deployable components are almost the same as Microservices. And although Microservices is a buzz word, it does have significant merit to base your architecture on Microservices in order to deliver agility to your business. It’s the perfect foundation for business agility.

Which gets me to the point of my presentation on the topic of Continuous Delivery and Architecture I mentioned earlier.

It is a common misconception that Agile and Architecture don’t play well together. Which is over course utter BS, and if you don’t believe me, I posted about this not that long ago (Product Owner and Architect, Agile Tag Team). Agile and Architecture play extremely well together as long as the Product Owner and the Architect(s) play well together. Especially when it comes to Microservices you need the domain architect or the business architect or both when you have that luxury. It is the architect’s responsibility to understand the business domain and together with the PO define what products make up that domain. Every product is a very likely candidate of either become a Microservice in its own right or becomes a composition (or constellation) of Microservices. And the Product Architect, which you might know as being the Application Architect, Project Architect or Solution Architect, is the single one person that together with the PO defines where the Product ends and the Platform on which it runs starts. The Product Architect therefore is the person that defines the (technical) boundaries of the Product and the domain architect defines the product’s place in the IT landscape. And like I said, a product is a Microservice or constellation of Microservices. Ideally, of course. Domain architect and Product architect should also work closely together in defining what API’s to consume and API’s to provide. Again together with the PO since it’s all about the PO’s product.

In this situation, we have a nice decomposition of a product or several products in interfaces and implementations. And that is exactly what we want from an SOA, especially one that is comprised of independently deployable components, services if you will. Or even Microservices.

Once all interactions between products and within products are based on interfaces, we can talk about a true SOA.

Now here’s an interesting detail. Interfaces are nothing but documentation. Be it fairly intelligent documentation or rather very usable documentation, but documentation nonetheless. And everybody that thinks an interface is something else is wrong. We don’t put an interface in an architectural layer because that doesn’t make sense. The same goes for API’s. An API is also nothing more than documentation that conforms to specific standards. Of course it’s a bit more than just a document, but really, just a little bit more. An API without an implementation is nothing and in fact you can’t actually deploy an API, nor can you an interface for that matter.

Having established that, it’s time to start the confusion and get on with it big time.

So, since an interface is nothing but documentation and it’s something you can’t really deploy. In order to provide business agility through an SOA, we need to put something in between interfaces and their implementations… But before I venture into that area, let me explain how you get business agility through an SOA.

We do this by not implementing an SOA but a POA. Where an SOA is a Service Oriented Architecture, a POA is a Policy Oriented Architecture. Basically this is an architecture in which you have services separated into an interface and one or more implementations of the interface. All implementations are available concurrently and based on policies, one of the implementations is selected. The policies are business policies and governed by the business. Business policies are a fancy word for business rules.

An example of such a business rule is that when transferring money between a customer’s own accounts there is no need for additional validation of the transaction once the customer is signed into her internet banking. Another rule is that when transferring money from a customer’s own account to an account of another customer, an additional signature is needed. Both are financial transactions, but based on the context, different business rules apply.

Policies are context driven, which amounts to the notion of under what circumstances a service is consumed as that determines the implementation of the interface to be used. Contexts can be everything imaginable. Within the example of the transaction as mentioned above, it can be the amount of the transaction, the age of the customer, the time of day, the geographical location of either customer in the transaction, the type of accounts, the client device used and the way a customer was authenticated. Or any combination of these contextual parameters or something completely different altogether.

In a POA, there is a clear distinction between interfaces, policies and implementations. The policy sits between the interface and the implementation. And it is business definable, meaning that the business rules can be changed without changing the interface and existing implementations. Either resulting in implementing policy compliance, i.e. some software is developed that implements the functionality to comply with the policy or by implementing another implementation of the service.

Some 10 years ago, Aspect Oriented Programming (AOP) was hot in software development country and although this way of software development is more or less forgotten, it is exactly what POA (see the similarity in acronyms?) wants to establish. By injecting new code, or rather new aspects of an implementation policies can be changed.

The problem with AOP was that was not trivial to do and it was hard to debug once there was an error in the functionality. AOP is also at the class level instead of the service level, and it is far from accessible to non-developers. Still the paradigms of AOP are what POA intends to deliver. A more flexible way of changing nuances in an interface’s implementation.

As you can understand is that the role of the architect and the PO is very significant in POA. The Product Owner needs to understand where policies are to be in place from a business perspective. Not every interface needs policies, and sometimes you don’t even want to have the ability to change a policy without strict governance. The architect, and typically the Product Architect, needs to define how policies are implemented. Since there are as many ways of implementing policies as there are use cases for policies, it is important to at least within the context of a product the same mechanism is used. This is where the architect comes in.

The power of an API, or a Microservice or even a normal service, is so much more when POA aspects are applied as the Product is enabling or rather facilitating business agility. Especially when the used framework for policy definition and implementation is one that allows for doing so by a non-developer. Unfortunately there are not that many frameworks available, most solutions are based on Case Management solutions or consist of glorified Business Process or Business Rules engines. Centralized solutions that require specific expertise to operate and maintain them. Consequently, the autonomity and therefore the agility of the teams and business users are limited. This is where this post more or less started off from. Centralized solutions are limiting agility, and when you want to extend agility to the business, you should limit it in the technology.

So this is how you get Microservices on steroids; apply them in a Policy Oriented Architecture. I hope you liked this post and feel free to comment and discuss. Although the POA is not new, you don’t see it applied that often if at all. I don’t think it’s a matter of complexity or a technology issue. The challenge is typically in that business and IT need to be considered as one and not two separate departments.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contactlist. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

Microservices on steroids, getting from just agile to business agile [part 1 of 2]

Summary

Agility is important, and business agility even more important. With this, the architect has to play a crucial role by defining the right principles, commandments if you will, and seeing to it that the product development teams live by them. Principles like KISS and Independent Deployability are two principles not to mess with. MIcroservices are an important and extremely helpful aspect of this, so is continuous delivery. But you need to get those microservices on steroids in order to be able to move from just agile to business agile. In this first of a two piece post, I am laying the ground works for explaining how you can facilitate business agility in your architecture.

Since you’re reading this post, it’s likely you’re an IT architect or have to deal with one every once in a while. Meaning that you’ve been confronted one way or the other with Service Oriented Architectures (SOA). Maybe you’re even a proponent of this architectural style.
I’m not going to explain what an SOA is, there’s quite a few very good sources available online explaining what it is and why it’s a good way of building your IT landscape, or not. As always, there are just as many proponents as there are people against and SOA. Over the years I’ve grown to like the premise of an IT landscape where fairly small pieces of functionality can be strung together to realize business functions. And with the advent of RESTful web-services, the premise of a true SOA has become more prevalent as ever.

Nowadays all you see and hear around you are Microservices, even though there doesn't seem to be a clear definition that everybody feels comfortable with, everybody is doing Microservices. One of the most interesting definitions I heard is that it is a really small service. But not as small as a nano service. But just to make sure that you know what I refer to when discussing Microservices, here're two definitions I like to use. The first is from Wikipedia, which in and by itself is no guarantee for it to be correct, but at least it's a common definition:

"Microservices is a variant of the service-oriented architecture (SOA) architectural style that structures an application as a collection of loosely coupled services. In a microservices architecture, services should be fine-grained and the protocols should be lightweight. The benefit of decomposing an application into different smaller services is that it improves modularity and makes the application easier to understand, develop and test. It also parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently. It also allows the architecture of an individual service to emerge through continuous refactoring. Microservices-based architectures enable continuous delivery and deployment."

The second definition is form Martin Fowler, which is in essence the same:

"In short, the microservice architectural style [1] is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies."

Microservices are not easy to develop, much like API's, they require a rather typical level of maturity of a development team to develop them. The hard part of Microservices is typically the "independently deployable" aspect of the services and then in particular maintaining that independence of time. Let's call it sustainable independence.

What I usually see is that over time the architecture becomes convoluted, resources entangled (the whole HATEOAS thingy is not trivial at all) and adversity against rearchitecting the IT landscape to get back to independent microservices.

To develop them,  the Microservices, you need to invest. In general, a Microservice is way more expensive to develop than a regular service. Much like an API being a lot more expensive than a simple service interface. So I would argue that there needs to be a business case to capture some functionality or resource in a Microservice.

Still, Microservices are a good thing to look into, if not only to be able to dismiss them. But more importantly, they are very helpful in becoming agile. I would almost argue that in order to be agile, you need an SOA based on Microservices and API's. Having said this, I need to emphasise that we're still talking about agility of the product development team, the IT people.

Mind that having an architecture with highly independent units of deployment is a blessing for all of us that want to do Continuous Delivery or Deployment for that matter. It makes life so much easier and as an architect you should really make sure that independence of components, services or units of deployments, however you want to call it, is a prime-directive to all your developers, great and small. It should be up there on the wall with the rest of your architecture principles. First of course you mention KISS and second the declaration of independence for your components.

But as I said, agility so far is technical agility. You've designed your software, your product, for change. Which allows you to be agile. Your PO (Product Owner) wants something different, you can change your code easy as can be. But when you think about it, it won't make your business more agile, it will make your IT more agile.

There's quite a bit more to be done before your business gets its agility... more on that in the next part.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contactlist. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

Special thanks to Sytse for pointing out one or two textual errors and helping me correcting them.