How Charming is Laziness?

Long time ago, while I was still a student at the PolyTechnic in Enschede, The Netherlands, I would justify my choice to study computer science, by proclaiming that laziness leads to efficiency. Which of course makes no sense I now know, because it leads to effectiveness. Penny-pinching leads to efficiency. You can read all about it in this post Perish or Survive, or being Efficient vs being Effective. But there's this little concept that results in more efficiency and laziness as well. It's therefore a charm. It is called automation and closely related to that third time.

Third Time Is Automated Principle

One of the key principles at Amazon AWS is that everything must be automated. It's not just that everything should be automatable, but it should be automated.

Whether or not it's an urban legend, but word has it that when you create something that requires manual action, you're out looking for a new job. For a company like Amazon AWS it is clear why automation is such a huge thing. And it is clear as well why they have such focus on API's. API's are how automation across the board is facilitated. But most companies are not Amazon or any of the other cloud providers. Most of you my dear readers are more likely to run your systems on Amazon AWS, Google Cloud or Microsoft Azure, then run those applications for your customers. Probably, your IT landscape is in size not even close to Amazon. Probably comparing to Amazon AWS as the Netherlands compares to the rest of the EU. In most if not all of the dimensions you can think of.

Arguably, the principle of "Automate Everything" doesn't apply to you. I'll leave it up to you to think of one or more arguments why automation is not something you should hold dearly.

Challenge to you: put in the comments a good reason why you think automation is not necessarily needed. I'll make an effort to counter your argumentation as a reply to your comment.
But read on first.

The benefits of automating processes are many. Irrespective of the kind of processes. An automated process is infinitely more likely to be repeatable than any manual process. This results in higher quality since errors will either be made consistently, and can be fixed, or will consistently not occur at all. How compelling is that?
Although the automated and manual version of a process might take the same amount of time to be executed, the automated process allows a person to work concurrently on something else that cannot be automated. And automated processes do not rely on the availability of a specialist to execute the process. So automation makes you and your organisation more scalable.

Not automating processes, even IT processes doesn't make sense. Still, when it comes to IT, we hardly do this. Why?

The situation at Amazon

I've come across many situations where things weren't automated. Worse yet, they could be automated. They were not automatable.

For me this was always an interesting fact to find out. For one, we're in IT and IT is all about automation. In fact, in Dutch we refer to IT as Automation (Automatisering - Dutch). The paradox is that we apply IT all over the enterprise to automate business processes, but when it comes to the IT processes themselves, automation is very likely the last thing on our minds. And when you think of it, that doesn't make sense at all. Walk into a room full of IT people, and just pose the statement that it is hard to understand why we're so good at automating business processes, yet we don't have automation in our own processes. And you'll see at least 90% of in-agreement-nodding heads, the remaining 10% are too flabbergasted with the realisation that this is a true statement. Same statement in a room full of non IT people and the first thing you find yourself doing is explaining why this is.

The fact that IT people are not automating their own processes is tough to explain, and I for one do not have such an explanation. There is an explanation though, for why Amazon AWS's processes are all automated. It's because one of the core principles by which they do IT is "Automate Everything". At a dinner party with Werner Vogels (Amazon's CTO) I was invited to, being the Chief Architect of a FinTech startup, I asked Werner (all attendees were told that we were on first-name basis), how it was possible for a huge company like Amazon AWS to live by these rigorous principles. Everything is an API, Automate Everything, and a few more. His reply was that there were two main reasons why it works. 

1. Senior management all the way up to Jeff Bezos, were openly behind these principles. In fact many of them were mandated by Jeff Bezos himself. 
2. Everybody in the organisation experienced for themselves the validity of these principles.

I asked him to clarify that second reason. According to Werner Vogels, Amazon is dedicating a lot if not all of its time to make its processes impacting customer experience as efficient as possible without impacting customer satisfaction. 'A satisfied customer is a returning customer'. The effects of changing a process is made visible to the whole company, at all times. So compliance to the principles will result in changes to the processes and the effects would be visible. Therefore, the validation of the principles would be continuous. And according to Werner, nothing is as motivating to change your processes than to see the effects of your efforts.

Rest of the dinner I was milling this over, enjoying the food and talk to the other guests.

The situation in the 'real world'

Unfortunately, most of us work at real enterprises. And those two reasons Werner Vogels had given me why IT process automation worked for Amazon aren't obvious in the situations I have found myself in.

For example, the amount of automation in a process within IT is not a metric that anybody is held accountable for anywhere I've worked or consulted. Neither are the benefits of automation part of somebody's accountability.

IT process cost reduction (IPCR) is as far as I know not a KPI within enterprises. Nor is the time-to-market (TTM). The latter often does find itself in another incarnation on reports and dashboards, namely as MTTR, the Mean Time To Resolution. When it comes to MTTR, we do see them on reports, but the MTTR is in hardly any case part of somebody's accountability. Same goes for time-to-market. Although it is always mentioned for any improvement project, it is hardly measured or reported on. It's a project issue in most cases, meaning that an organisation is doing projects instead of delivering products.

The lack of real metrics and if you will KPI's means that from an accountability perspective, there is not a clear person that has an incentive to push automation. And if you read my blogs on a regular basis, you know that I'm big on accountability.
Visibility of the effects of changes to the IT processes is another challenge for organisations. We often find ourselves in organisations that don't have a culture to measure our IT processes. This is especially true for organisations that have been around for decades. Since IT process automation is not pushed, there is no incentive to find bottlenecks in processes or pinpoint areas that are up for improvement. Resulting in the situation where the effect of improvements are not visible in most cases.

The lack of accountability and the rather big hurdle to be taken by IT departments in order to automate result in a situation in which we are just not automating our processes, because it gets no priority on our backlogs or funding in our PRINCE2 budgets. Process automation is collateral. And it's not a matter of not being as large a company as Amazon. It's a matter of not being aware of how automation affects the bottomline. And of not being held accountable for impacting the bottomline.

Third Time Principle

Three times is a charm, or at least should be automated.

In organisations I worked previously, the principle of automation as in play at Amazon was a little bit more pragmatic. One that has worked well for me, is the principle of "Everything done a third time, will be automated", reasoning behind this is that if you do the same thing a third time, it is extremely likely you will do it a 4th, 5th and even more times. The time required to create the automation will be saved by executing the task over and over again.

Time invested is never gained, but can only lead to savings later on.

The Third Time Principle is a compromise, although one might argue that it is a Troyan horse. By adopting this principle, the argument that it creates too much overhead for mundane actions is off the table. Only those actions that are performed repeatedly are automated. Built in justification for the investment needed.

The Third Time Principle is easy to adopt. Since it is a compromise, it can be applied when the push to automation is bottom-up. We often see that engineers see the need for automation since that is where automation will solve a problem and effects are noticed. To develop the automation is often a matter of getting the time to do so. Automation is now competing with other requirements for development time, for priority on the backlog. We all know where the priorities will be. Applying The Third Time Principle will remove this obstacle. Engineers can justify the need for automation and the Product Owner can justify the priority of the automation stories on the backlog.

Continuous Delivery

When striving for Continuous Delivery (CD) and more so for Continuous Deployment (also CD), there is no other way than to automate everything. CD requires a rigorous regime to move every manual task to the left in a process defined western style (i.e. left to right notation). Product development follows the DTAP model. Development is followed by testing is followed by accepting is followed by production → DTAP. In CD we hold true to the paradigm that everything manual is done in D, and TAP are fully automated. Any manual action in T, A and P needs to be automated and the scripts for this are developed in D, because otherwise it can't be done.
So when striving for Continuous Delivery, everything in the product development cycle is to be automated.

Accountability

When process automation metrics are part of somebody's accountability, that person will also be mandated to automate as much as possible. This is for the simple reason that you can't hold somebody accountable for something they cannot influence. Therefore, when process automation is measured through some metric for which somebody is accountable, it is that person's prerogative to drop The Third Time Principle and instead adopt the Automate Everything Principle.

Accountability is a matter of top-down enforcement. It is also a key aspect of culture change. When we want to adopt a culture in which we allow ourselves to reap all benefits of automation, especially our IT processes, we can't get around the fact that we need to revisit the metrics by which we hold ourselves accountable.

Scalability

One aspect that I haven't really touched upon is scalability, although I did mention it briefly. Automation is a key aspect of scalability, not so much scalability on a technical level but organisational scalability. Manual actions always need a person to execute them. The more complex the activities become, the more experienced or knowledgeable the person needs to be. And before you know it, it requires a specific person within the organisation. Because you rely on this person, that person will become the person with all required knowledge, it's a vicious circle. Think about it for a second, and I'm sure that you can think of a process or an activity in a process where you rely on a specific person and you know that person by name. In fact, if you want the activity to be done, you will even call that person because she's among the busiest persons in the company.

When you want to keep activities simple, you need to automate them. The more you automate, the easier it is to keep the activity simple. Hardly anything is more simpler than to have a command like 'do_complex_activity.py' provided that the automation is done using Python. Anybody can run this command provided they have the rights to do so, and when done really properly, anybody can do it at any time, because all the complexity of who can do what when is taken care of by the automation code as well.

Concluding

In the end, you'll agree that automation is what we need to apply to all our processes. At least when it's the third time you're doing the same job. It will improve quality and consistency. It also means that we as a company can scale our organisation without the need to increase headcount. It will be important to monitor the benefits of automation, without metrics it will remain a matter of good faith on the short term, and it will not justify the investments on a longer term. By understanding the benefits of automation and getting insights on where automation has most impact.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect


The text very explicitly communicates my own personal views, experiences and practices. Any similarities with the views, experiences and practices of any of my previous or current clients, customers or employers are strictly coincidental. This post is therefore my own, and I am the sole author of it and am the sole copyright holder of it.

Why CDM resonates with Waterfallians and not so much with Agilisto's

Architecture often is a matter of perception.

Architects that consider that architecture to be a noun, often consider, for example, that a CDM (Canonical Data Model) is a solution to a problem. Architects that consider architecture to be a verb, are very likely not considering a CDM at all. And although I have very strong personal feelings against architectural artefacts like a CDM, I'll try to explain in this post, why they can be perceived as an addition to an IT landscape. I think that is also very much an issue with a historic touch.

The School of Waterfallians

When you're a Wterfallian, when you come from the Waterfall school, then architecture is part of the analysis and design phase. This phase ends with 'The Architecture'. And that "picture with the 1000 words compendium" will have to last for the rest of the project. So you have to design something, come up with an architecture that will remain unchanged for months if not years. Your data model is obviously part of your architecture and will be included. Soon you'll draw the conclusion that a durable architecture requires for CDM, because how else can you ensure that your design fits within the landscape of existing and future designs?
If you have been raised with the standards and values ​​of a waterfall organisation, then you also know that deviating from previous decisions is out of the question. That only causes delays and budget overruns. Waterfallic architects will often focus on edge-cases, because they have the experience that these are the reasons to double back on previously made decisions. Logical behaviour is therefore also that you want to have it, your list of edge-cases, as complete as possible before you get started. Waterfall is a self-reinforcing approach when it comes to architecture and culture.

The Agilista School of Architecture

If you come from the agile school of architecture, then architecture is part of the development phase. Architecture emerges. The architect is thus a developer, or better said, the developers create the architecture. The agile architect therefore only design the rules-of-engagement, they merely create the play-book. A comply-or-explain concept ensures that the architecture can emerge and the best architecture at that point of time can be defined by the developers. The best architecture within the current context emerges. So you do architecture (verb), and you do that by adhering to the rules. This is a continuous process, you have to play by the rules continuously. When we look at a CDM, we find that the CDM itself is not that exciting in an agile world, instead the rules that a (C) DM has to comply to are. They are the grammar of the language.
When you were raised with the norms and values ​​of an agile organisation, then you also know that sticking to earlier made decisions is out of the question. At least, if this is against better judgment. That only causes a huge waste of resources instead. The agile architects will focus on the most common cases, because they have the experience that these have to be realised first in order to create value, validate hypotheses and provide insight into the next steps to be taken. Logical behaviour is therefore also that you want to have the first use-case defined as soon as possible in order to get feedback. Agile is also a self-reinforcing approach when it comes to architecture and culture.

Culture and Values

The crux is in culture and therefore in standards and values ​​and therefore in accountability. When the performance of an architect is measured based on the number of times that decisions have to be reconsidered, then an ever decreasing number of this metric is, for a waterfallian, an indication that things are improving. For the agile architect the opposite applies and an ever increasing number of reconsiderations will be better... or not.
Because the agile architect does not double back to earlier decisions, as the agile architect tries to find the best decision for a situation every single time again.

Hmmmm, then how would you evaluate the performance of the agile architect? That question is in fact not that hard at all to answer. The agile architect is not about the architecture but about the rules. The more stable they are, the better. At the same time, the rules have to be sufficiently complete to be able to develop products that offer the organisation a bright future.

The Architecturally Challenged

Who has the more challenging task? The Agilista or the Waterfallian?

That can not be answered, because they can not coexist. But it does make it clear that both have a big challenge. Each in their own system.

The waterfallic architect is mainly concerned with analysing and specifying everything in advance. It is someone who is good at overseeing many concrete aspects. She is someone who knows a lot, someone who thinks in as-is and to-be. That is to be expected, because in a waterfall organisation the architect is often the person with the most experience with a product or product group. The focus is mainly on 'what' and 'how' in the 'why, what, how' system.

The agile architect is mainly concerned with understanding the dynamics of the organisation. She is someone who is good at working at the abstract level and thinks in concepts. It is someone who understands a lot, someone who thinks in from as-is towards to-be. That is to be expected, because in an agile organisation is often the architect who has the most experience in how the organisation has developed. The focus is mainly on 'why' and 'what' in the 'why, what, how' system.



Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect


The text very explicitly communicates my own personal views, experiences and practices. Any similarities with the views, experiences and practices of any of my previous or current clients, customers or employers are strictly coincidental. This post is therefore my own, and I am the sole author of it and am the sole copyright holder of it.

The Power of Accountability

Some months ago, I had a discussion with a friend and colleague of mine and ever since I’m planning to write this post. This post is about responsibility and accountability.

Summarising

Unless people feel accountability and responsibility instead of just writing it down, RACI tables are a playbook for the finger-pointing game and not a tool for continuous improvement. Accountability and responsibility are the keys to successful products. For architectures this means that they need to be owned, maintained and challenged. Constantly. Continuously.

The reason for the discussion I mentioned was that he was asked to write a short report on the state of the IT department of one of his clients. The issue at hand was that there were a number of serious problems with the IT systems and although outages were fixed fairly quickly, the root-causes were hardly ever addressed. Instead people would blame each other. Fervently. After a couple of interviews, my friend concluded that there was a serious lack in governance in all areas of the organisation. And over dinner we were musing over this whole governance thing. And while enjoying a good meal and afterwards a really nice cognac, we more or less stumbled upon two words that I have been using more and more ever since: Responsibility and Accountability.

So that is what this post is about. This time I’m writing about responsibility and accountability and their role in architecture.

RACI explained

I think that you’re familiar with the concept of RACI tables. Tables in which on a variety of interrelated topics it is noted who is R(esponsible), A(ccountable), C(onsulted) and I(nformed). In my experience, the RACI table is typically very much discussed and eventually agreed upon. But hardly ever relevant in practice. Reason for this is that the RACI table is listed in some document and only looked at when there is trouble.

Let’s dissect the RACI for a moment. I’ll do this back-to-front as that is serving my purpose best. Starting with Informed. This depicts all people and parties that are to be informed about something. Could be an design decision, a contract, a change in a process, anything for that matter. For simplicity I’ll stick with ‘decision’. Then there’s Consulted. Where the Informed are merely told about a decision that has been made, the Consulted are asked their opinion about that decision. Their input is weighing into the decision. The main difference between the Informed and the Consulted is the fact that the Consulted have influence. Informed listen, Consulted talk.

Thirdly, there’s Responsible. I know, it should be Accountable, but I’ll address them last. The Responsible are about making the decision based on their own knowledge and understanding, taking into account what the Consulted said about the decision as well. The Responsible are the people that are all about that the decision is a sound decision. They (should) care about quality. In addition, the Responsible are in many cases also responsible for consulting the Consulted and informing the Informed. I agree with this. Then lastly there is the Accountable. Note the plural in Responsible, Consulted and Informed and the fact that the Accountable is not plural. Accountability is not a team-sport! The Accountable is held accountable for the results, the effects, the impact of the decision.

When we look at this in a Product context, then the RACI is as follows: Responsible is about the production of the Product, the quality of the product and the quantity of the product. The Accountable is about the money made (or lost) with the (sales of) product. Now when you factor in the Product Owner, it should be clear that the Product Owner is all about creating value with the Product she owns. Because of the accountability aspect, the Product Owner needs to find the perfect balance between quality, quantity and cost in order to generate the most value... but the Product Team is responsible for achieving this quality, produce the quantity and keep the cost level stable.
It goes without saying that the relationship between Product Owner and Product Team is to be very tight and based completely on trust.

RACI for the Architecture

You're probably asking yourself where architecture comes into play. I hope you've already read my post on the Agile Architect. In that post I am explaining the role of the architect in an agile environment. Just read it if you haven't already done so.

With architecture it becomes tricky, because architecture can be implicit as well as explicit. Implicit architecture is architecture that emerges. It is the Product Team that is formulating the product's architecture while creating and changing the product. As the product is developed, the team will find, define and adopt practices. Some will become best practices within the organisation as teams share their experiences. But as the team is identifying practices that they find supportive to their product development, they will become an integral part of the way the team works operates and there you go; Implicit Architecture. Mind that architecture is not some soft of diagram with blocks and arrows, instead it is a set of rules by which the product is developed. There's a team understanding that once you live by these rules, the world will be a nice place to live. The architecture is a framework of norms and values. Everybody in the team is held responsible for working within this framework. At best, the Scrum Master can be held accountable for the fact that everybody is following the rules of the team. But with respect to the architecture, this is a stretch.
Once these principles are formalised they become explicit. You'll be able to read them somewhere. The Ten Commandments of the team according to which they develop their products. At this point the team is still responsible for complying with the architecture principles, but since they're written down they need to be maintained. Mind that an architecture is something that needs to be nurtured. It needs to be constantly challenged, validated and maintained. And that should be somebody's responsibility. And that somebody should be held accountable for the relevance of the architecture.

Who da man?

Think about it, once you define an architecture, once you write it down, you create the architecture for somebody to be used. You have created a product. And there should be somebody owning that product and somebody developing that product. Somebody being accountable for the value architecture and somebody responsible for the quality of the architecture. Give it some thought and write down who is within your organisation accountable and who is responsible for any of the explicit architectures. And here's another challenge; ask them whether they feel the same way. For argument's sake, let's call the person accountable 'Chief Architect' and the person or persons responsible 'Product Architect'. And now ask them who they think should be consulted and who should be informed, just to make the RACI complete.

Yeah, that last one is tricky as can be. Traditionally and be aware that tradition is still prevalent, users are informed about products. And traditionally business is consulted about the product. Which of course is all backwards. Which product has the best product/market fit? Exactly, it's the product that meets the needs of the users best. And who better to define what product is meeting those needs best than the user. So users need to be consulted, not informed. I repeat, users should be consulted and not informed. That also holds true for the architecture.
Since the Chief Architect is accountable for the architecture, she will have to make sure that the architecture is relevant. That it is continuously validated for its applicability. Thus, the Chief Architect will need to consult with the developers whether their needs for an architecture, for the architecture are met, and that they can apply the architecture while developing the products. Consequently, the Product Architects are responsible for continuously consult the developers about what they need, how they need it, what works and what doesn't, etc.
In simple terms, the Chief Architect is accountable for the product/market fit of the architecture, and the Product Architect is responsible for making sure that the architecture addresses the needs of the developers. Obviously, when there are domain architects or other more over-arching architects, they befall the same faith.

RACI to solve problems

Now back to the discussion with my friend. The whole accountability and responsibility thing. I hope that you understand that unless the RACI is not only written down, but actually implemented, it doesn't mean a thing and can only be used to 'properly point fingers' when things go wrong. But it will not help you prevent problems. This is because unless you feel ownership, you feel accountable, you are not motivated to pro-actively work on improving your product.
So unless people feel accountable and responsible, they will not consult the right people, inform others in a timely matter and will work on preventing incidents. Finger-pointing is the result. And this is also the case for less obvious products like architectures, contracts, privacy policies and for example employee contracts.

Accept the Challenge

Again I challenge you to name the person(s) accountable for the explicit architectures in your organisation as well as those responsible for those architectures. And I dare you to hold them accountable and responsible. And I double dare you to make them consult their users and not just inform their users. Well, actually, start by asking who their users are, and you are likely to be in for a surprise.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect