Arc-E-Tect

June 8, 2017

‘The Continuous Delivery IT Team’ fallacy

Summarizing

Considering Continuous Delivery something for your IT department is throwing your money out the window when doing an Agile Transformation program. If you want to do so, make sure you throw it into my direction. IT is a business concern, Continuous Delivery is a business concern.

Over the past couple of months I did a series of awareness sessions on Agile, Continuous Delivery and DevOps at a large client of mine. As is rather common, also at this organization the initiative to move towards Agile ways of working, Continuous Delivery and a longing for DevOps is with the software developers. This makes perfect sense when you think of it, because it's the developers that want or rather need to make changes and the pace by which they have to deliver changes is only increasing. But I guess I don't need to tell you this.

But Continuous Delivery (or Deployment for that matter) is only possible when you don't consider it a software development thing. There really is no point in spending any time to move to Continuous Delivery when you are not planning to do it broadly. I'll get to that in a second.

During these awareness sessions, which I do with colleagues from the same team, we discuss with non-developers like risk officers, sys admins, project managers, architects, test consultants, etc. we outline what Continuous Delivery and DevOps mean within the context of my client's organization. It's a common story and I won't bore you with the details, but obviously we touch upon the benefits of small increments, feedback loops and so on. And the fact of the matter is that our story actually does sound like it's the perfect thing. And we consistently get the question: "So, really, it can't be all awesomeness, so what's the catch?". And in the rare occasions we don't get this question, we still answer it.

The biggest problem with Continuous Delivery and everything following from it, is that it is not a software development thing, or even an IT thing. When you think so and still go down that road, spare yourself the frustration and disappointment, drop me an email asking for my IBAN number and transfer the budget for your transformation project into my account. At least one person will be happy with you spending that money.

And yes, even when you think it's an IT thing, you're better of giving me that money. This is what I call:

'The Continuous Delivery IT Team' fallacy

Let me explain. First of all, let's make sure you understand what I consider Continuous Delivery. It's the process that produces a product that results in business value in order to be able to sustain the organization up to the point that it can be delivered to a user and it will be delivered to a user as soon as possible. It's not a perfect definition or even a formal definition, much because there's so much more to it. But what's important is that it defines work on a product as being done, when it is ready to be delivered to a user. The actual delivery is an explicit manual step which is decided by the Product Owner to be taken as where the rest of the process is preferably fully automated. This as opposed to Continuous Deployment, where the delivery to the user is also automated and hence triggered by the developer when committing his code to the source code repository. Again, this is a definition close enough to what it is and fitting the purpose of this post.

With Continuous Delivery and agile working in general you want to receive feedback for what you've been doing as soon as possible. And preferably you want this feedback to be such that for one you know that you've done well and secondly you're actually contributed to the bottom line of the organization you work for. This is why we like to work at the granularity of user stories and epics and delivery on a per user story or at least on a per epic the changes to a user.

As you now understand, for every single user story or at least epic, you need to do everything that needs to be done for a release, because you're delivering to a user. Somebody is going to actually use that little piece of software you've worked on with such a passion. Fact may be that the complexity is limited because increments are small, still you need to release new or changed software. And there's the catch.

With software delivery, or product delivery in general, it's not just the product development team that's involved, or more specifically the software development team. It's other teams and people involved as well. Think about marketing, legal and compliance, worker's associations, security and risk management. These are all teams that are not part of the IT department. And no, security officers are not part of an IT department, and in case they are, they most definitely shouldn't be. And the biggest catch of all, the 'business' needs to be involved from day -1. Unless all of these different roles, teams, people, stakeholders, however you want to call them are on board and work in those same small increments, not becoming a bottleneck and automate as much as possible, your Continuous Delivery efforts are a waste of everybody's time and your organization's money.

Back to your 'business', it's them that request for features and not the user. It's them that pay for the development of the product not using it per se. When that 'business' is not capable or willing to define the product's features such that they can be delivered in tiny chunks, than you're out of luck and not much will come from Continuous Delivery in your organization.

The Product Owner is key in all of this. Being the hinge between the Product Team and the rest of the world, the PO is the single one person that can and must ensure that the product is delivered incrementally, with business value visibly added with each increment. PO can't do this, you've got yourself some trouble. The PO, at all times, must be able to relate every single feature, one way or another, to an improved life for the user of your product and therefore a positive effect on the organization's bottom line.

So, Continuous Delivery is not an IT thing, it's a business thing. And don't let anybody convince you otherwise. Being as it is, moving towards an Agile way of working and Continuous Delivery or Deployment, in fact would mean that you no longer consider your IT as being delivered by a separate IT department, but as an integral part of your business.

This does make perfect sense, considering your IT as part of your business I mean. It makes perfect sense because more and more business are all about information and are run based on information. IT is no longer a tool, like a glorified typewriter, it is in fact what is producing business value. No IT no business.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contactlist. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect

June 7, 2017

Microservices on steroids, getting from just agile to business agile [part 1 of 2]

Summary

Agility is important, and business agility even more important. With this, the architect has to play a crucial role by defining the right principles, commandments if you will, and seeing to it that the product development teams live by them. Principles like KISS and Independent Deployability are two principles not to mess with. MIcroservices are an important and extremely helpful aspect of this, so is continuous delivery. But you need to get those microservices on steroids in order to be able to move from just agile to business agile. In this first of a two piece post, I am laying the ground works for explaining how you can facilitate business agility in your architecture.

Since you’re reading this post, it’s likely you’re an IT architect or have to deal with one every once in a while. Meaning that you’ve been confronted one way or the other with Service Oriented Architectures (SOA). Maybe you’re even a proponent of this architectural style.
I’m not going to explain what an SOA is, there’s quite a few very good sources available online explaining what it is and why it’s a good way of building your IT landscape, or not. As always, there are just as many proponents as there are people against and SOA. Over the years I’ve grown to like the premise of an IT landscape where fairly small pieces of functionality can be strung together to realize business functions. And with the advent of RESTful web-services, the premise of a true SOA has become more prevalent as ever.

Nowadays all you see and hear around you are Microservices, even though there doesn't seem to be a clear definition that everybody feels comfortable with, everybody is doing Microservices. One of the most interesting definitions I heard is that it is a really small service. But not as small as a nano service. But just to make sure that you know what I refer to when discussing Microservices, here're two definitions I like to use. The first is from Wikipedia, which in and by itself is no guarantee for it to be correct, but at least it's a common definition:

"Microservices is a variant of the service-oriented architecture (SOA) architectural style that structures an application as a collection of loosely coupled services. In a microservices architecture, services should be fine-grained and the protocols should be lightweight. The benefit of decomposing an application into different smaller services is that it improves modularity and makes the application easier to understand, develop and test. It also parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently. It also allows the architecture of an individual service to emerge through continuous refactoring. Microservices-based architectures enable continuous delivery and deployment."

The second definition is form Martin Fowler, which is in essence the same:

"In short, the microservice architectural style [1] is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery. There is a bare minimum of centralized management of these services, which may be written in different programming languages and use different data storage technologies."

Microservices are not easy to develop, much like API's, they require a rather typical level of maturity of a development team to develop them. The hard part of Microservices is typically the "independently deployable" aspect of the services and then in particular maintaining that independence of time. Let's call it sustainable independence.

What I usually see is that over time the architecture becomes convoluted, resources entangled (the whole HATEOAS thingy is not trivial at all) and adversity against rearchitecting the IT landscape to get back to independent microservices.

To develop them, the Microservices, you need to invest. In general, a Microservice is way more expensive to develop than a regular service. Much like an API being a lot more expensive than a simple service interface. So I would argue that there needs to be a business case to capture some functionality or resource in a Microservice.

Still, Microservices are a good thing to look into, if not only to be able to dismiss them. But more importantly, they are very helpful in becoming agile. I would almost argue that in order to be agile, you need an SOA based on Microservices and API's. Having said this, I need to emphasise that we're still talking about agility of the product development team, the IT people.

Mind that having an architecture with highly independent units of deployment is a blessing for all of us that want to do Continuous Delivery or Deployment for that matter. It makes life so much easier and as an architect you should really make sure that independence of components, services or units of deployments, however you want to call it, is a prime-directive to all your developers, great and small. It should be up there on the wall with the rest of your architecture principles. First of course you mention KISS and second the declaration of independence for your components.

But as I said, agility so far is technical agility. You've designed your software, your product, for change. Which allows you to be agile. Your PO (Product Owner) wants something different, you can change your code easy as can be. But when you think about it, it won't make your business more agile, it will make your IT more agile.

There's quite a bit more to be done before your business gets its agility... more on that in the next part.

April 13, 2017

Product Owner and Architect, agile tag-team

Picture yourself in an environment where agile practices are adopted as the standard way of developing solutions. Now if you're really ambitious, you're picturing yourself in an environment where there are no projects, only products. Consider yourself to be part of a team working on a product, a product team. The team is comprised of all the necessary expertises to create the product it's developing. The team you're part of is responsible for the product, cradle to grave.
But wait a minute, are you really part of the team? Or are you external to the team, but do consider yourself heavily involved with the team? You might be the Product Owner or an architect.

Is the PO not part of the team? Maybe, maybe not. It's not that important actually. And what about the architect? Is the architect not part of the team? Well, the architect, I would think and this is just my very humble opinion, is not part of the team. The architect is concerned with the team, but not this team, but all the teams that are in the architect's domain. The architect is there to ensure that the products form a cohesive landscape that, by synergy, creates the most value for the organisation as is thinkable. Back to the PO. The PO might own many products in a portfolio...

Your picture. Did you picture yourself in this organisation as part of the team, of a team, or as somebody outside the team? It doesn't really matter too much. Really.

The interesting part of such an organisation, be it a for-profit or a not-for-profit or even something governmental, such an organisation is catering for its business, for the users of its products. And the products are there to generate value, pure and simple, life's better with every new version of the products the organisation releases. The product teams are there to work on these products, to make sure that the users are actually able to use these products and once they have those products at their disposal, they can continue to use them.

Other than in more traditional organisations where products are developed in projects, where there is a project organisation and a operations department to keep the solutions running. In these organisations teams, the project teams, are only concerned with getting the products out. And in many organisations it doesn't even goes as far as that and the project teams are primarily concerned with ensuring the products are accepted by the operations department. This is how these organisations work, and this is how teams operate. That's not to say that the individual members of the teams are not concerned about the quality of what they create or whether or not users actually like their products and use them. And that's not to say that the people working operations department are not concerned with the development of the software and that the quality is great etc. Organisationally, there's a divide and more importantly there's no one single entity that is responsible cradle to grave for the products.

What about this tag-team, this agile tag-team?

The effectiveness of the PO-Architect team, this tag-team, is the greatest in this product oriented organisation. The more 'agile' is considered a business trait instead of just an IT trait or even solely a software development trait, the bigger the impact the tag-team can and will have on the bottomline of the organisation, in a positive way.

Here's why. The PO, when peeled like an onion is at the core only caring about creating business value. And the PO has two currencies that he can use to invest in order to create even more value. For one there's the crude oil of product development: €'s. Here we have the simple fact that the the business needs to be able to earn more money with the PO's products than the PO had to invest in creating the products. It's a simple matter of cost/benefit and ROI. The less the PO needs to spend, the more the business makes and the shorter the time for the business to earn back the investment the better.
But there's also the second currency, which is time. Time taking to create the product, or a new value creating feature of an existing product. Time has no ROI. You have to invest time, knowing you'll never get it back. You might save time with your product, but the time you invest, you can only invest once. This is important because you want to invest as little of that time in something that generates the most value. Think about this for a second... what it means is that you want to create a new product (or feature) that will generate the maximum value in a minimum timeframe. The crude oil, your €'s are secondary to this, as a PO, because you have a product team available that you need to pay for anyway. Remember, you're working in a product oriented organisation not a project oriented organisation. Your teams are pre-funded! This is also important to note.

So the priority of the PO should be generating as much business value as possible. Therefore the PO needs to prioritise the various products and their features such that the most value creating features are worked on by the product team first. And this is extremely tough for the PO, especially when the PO is not a business person because the PO needs to know what's good for the gander as that is good for the goose.
And in many cases the PO will need to experiment in order to find out whether or not a feature is really needed. Whether it is really going to be used. Whether the form it will be delivered in is the most optimal form. Experiments cost time as well, so the balance between experimenting and applying is critical.

This is where the architect comes in. Without the architect the PO is lost, or at least the PO is susceptible to overspending. First of all, the architect's view is more than just a product and is therefore prone know about solutions to partial problems available elsewhere in the organisation. And leverage them. The architect is the right person to lay the foundation for future features. In part by ensuring that there is a generic (application) infrastructure that can be leveraged throughout the lifecycle of the product. When the PO has the insight to define a product roadmap or a business direction for the product(s) to evolve in, and shares this with the architect. The architect will be even better equipped to pro-actively define the foundations of the products, where a little investment upfront pays itself back at a later stage. See the parallels with the PO, the architect is there to understand where and how best to invest product development time to save more product development time at a later stage.
But there's more, obviously. The architect, and this is the 'secondly' belonging to the 'first of all' from above, is best equipped to define where what corners can be cut for most impact and what experiments to run in order to get the most out of new or specific capabilities. The architect will be the single one person that understands how to scale the organisation's technologies. How to move from a simple, crude yet effective archaic technology to a more complex, elegant and sophisticated version and when to do so. The architect therefore can and will save the most valuable currency of the PO, feature development time, at every step of the way.

Remember that PO and business value creation? Value should be created as quickly as possible, so short development times are important. As much as possible value should be created, so the most desired product features with the biggest impact on the bottomline should be worked on by the team. But which are these features? Well the architect doesn't know these answers, but is perfectly suited to limit the amount of time it takes to either develop a feature or to find out what feature should be developed by devising an experiment. The PO still needs to run the experiment though.

Ah, here comes the hardest part of the post. How to set the right priorities? Which is a question the PO will need to answer. The common way to do this, the wrong way indeed, is to work on the simplest feature to develop. Or the one that can be delivered the fastest. Often these are the same features by the way. This will only result in the PO and the team to feel really good about themselves because they added many features to the product. Something that will in most cases be far from adding the most business value to the product.

Again the architect is a great team member for the PO in this as architecture is not about simple to develop features or quickly realisable solutions. Architecture is about ensuring durability, sustainability. It's about creating a playing field in which complex solutions can be developed quickly, because the groundwork has been done already and therefore a solid foundation is present.

The challenge is with the PO still as business value should be the key to the priority of the feature backlog. So with every feature the PO will need to define how much value will be created, preferably in terms of what will be changed, and what will be the benefit of the change. From a user's perspective, always from a user's perspective. The PO will also need to establish a baseline for each feature he's planning to invest development time in. This is for the simple reason that there's no way to determine whether or not you've improved life as you know it, without knowing what life is before you started your improvements. And consequently, the PO will need to start gathering the right metrics in order to know those improvements. Defining the reason why you want to change something, i.e. defining what the benefits will be, will more or less define what metrics will show progress and therefore what needs to be done to establish a baseline.
Leaves the PO with defining the top priority of all the features that need working on. And this is encoded in the metric used to measured progress and the reason for the change to be developed. From these a projected generated business value can be established.
And this is where the architect can shine again. Not with this little piece of math, but with the part about metrics and measurements. The architect will be the instigator of the existence of a set of architecture principles that allow for metrics and measurements. Principles that lead to the existence of comprehensive measurements, to profound standardisation of those aspects that need no specialisations, to extreme automation of setting up environments. This can be left with the product team, but when there are more than one team, there will be a need and at the least a benefit, of coherence across teams, which is where the architect is acting.

Concluding, the PO and the Architect make a wonderful team that jointly work on the best prioritisation of the feature backlog for the products of the PO. By doing so, they'll minimise the required investments in feature development time for the product team to develop a feature, all the while maximising the created business value. And as icing on the cake, they are in the process ensuring that the hypothesised improvements are validated. You want a cherry on top? With this awesome tag-team you have a perfect opportunity to define a ubiquitous language which is business domain specific and spoken across the whole of the organisation.

Translate