I started on these posts some time ago after numerous discussions with several of my clients on the topic. My initial post started to become really long and after I discussed some of the concepts with peers at the Lean Startup Summit Amsterdam 2018 I decided to have several posts on the topic of metrics, starting with measuring agile teams.
In this series of posts I will cover the following high-level topics:
- An introduction to this series.
- Measuring Agile Team performance. This post.
- Agile Maturity.
- Agility Metrics.
- Business value and Metric Harmonisation.
- Visualisation and Dashboarding.
So that's at least 5 more posts on the topic for you to read. Depending on the time I have to spare in the coming period, you'll be able to find the posts sooner or later. Please keep in mind that I will most likely post some articles on other topics as well. I'm dying to shed some light on the challenge of BizDevOps in large organisations. The real challenge I mean.
When we talk about agile teams, we typically talk about Scrum teams. But just to be clear, I want to make sure that you understand that when I talk about agile teams, I talk about Product Teams. Product Teams are teams that are responsible for the delivery of a product to a user of the product. And often that is a Scrum team since most of the organisations I know that are in the process of an agile transformation, or already have performed this transformation. But this is besides the point, for now.
And in most cases I've come across in the past couple of years, we don't want to know anything about the team, but about the team's members. The reason for this, is that we have performance agreements defined on the level of the individual employee of an organisation. Usually in the form of SMART objectives. Most, if not all organisations will in fact have these objectives defined in terms such that the employee's efficiency within the organisation is measured. Which makes perfect sense when you understand that the performance of anything is in most cases associated with its utilisation.
Key Performance Indicators, KPIs, therefore are almost always associated with utilisation and thus with efficiency. Too bad that doesn't make sense in an agile environment since agile is not about efficiency but about effectiveness.
Measuring the performance of an individual team member or even a team is not helping in determining how the individual or the team is performing. And in my experience it is actually counterproductive and not supporting the agile transformation when your organisation is in the middle of one. We will see that we need to define agile performance either in terms of value creation (preferably business value) or in terms of an agile team's behaviour. This post is about the latter, where the first is all about Key Value Indicators, KVIs.
Going back to the basics of metrics leads us to reconsider why we do take measurements. And the main reasons I can think of are that you want to be able:
One of the main reasons I advise clients to implement metrics and take measurements is to ensure that it is possible to direct the course of action of an organisation. When it comes to measuring a team you want to make sure that the team is meeting the desired objectives, or at least is on the right track or that it is more capable to meet these objectives. And more importantly, the objectives are business objectives.
The more challenging part is that you want to be able to combine this with autonomy. In fact, from a leadership point of view, you want to facilitate the team to have as much autonomy as humanly possible. Which boils down to a trust issue. The amount of autonomy is based on the amount of trust. The more trustworthy a team is, the more autonomy you're willing to give the team. But I am getting ahead of myself.
One of the perks of a team in an agile environment is autonomy. The reason being that the fewer times a team needs to ask and wait for permission, the faster it can react on feedback. (It isn't for nothing that in many organisations that have embraced an agile way of working, there is bound to be found a poster that says something in the lines of "Ask for forgiveness, not for permission".) Agility is all about being able to react on a changing world in a timely manner. This is all there is to it.
Logically, the more time is needed to react, for example because certain signatures are required, the less agile a team can be. Consequently, when a team can make its own decisions, it will be able to react faster on feedback and hence the team is more agile. So the team needs to be able to process the feedback loop, whatever the feedback loop is.
There is a catch in all this, especially when you do come across that poster I mentioned before, the one about forgiveness and permission. The catch being that there needs to be a trust-relation with the team. And not only within the team, but with the rest of the organisation and the team as well. Trust in this perspective is what makes or breaks the agile ecosystem.
Measuring Agile Team performance
In this and the other posts in this series, I will use a metric-analogy that has proven very helpful in the past. The analogy is one of driving a car. As you will see, driving a car is very much like managing a team, especially an agile team. And don't be alarmed, even when you don't have a driver's license or when you've never driven a car yourself, the analogy helps.
While reading this post, you might think that the post is about agile teams related to software development. This is not intended and I refer to the teams in this post as product development teams. These are teams that create a product that is to be used by somebody, the user. This could very well be a computer program, but just as easily a security policy, a reference architecture or even a toilet-bowl.
While reading this post, you might think that the post is about agile teams related to software development. This is not intended and I refer to the teams in this post as product development teams. These are teams that create a product that is to be used by somebody, the user. This could very well be a computer program, but just as easily a security policy, a reference architecture or even a toilet-bowl.
Key Performance Indicators
When we talk about agile teams, we typically talk about Scrum teams. But just to be clear, I want to make sure that you understand that when I talk about agile teams, I talk about Product Teams. Product Teams are teams that are responsible for the delivery of a product to a user of the product. And often that is a Scrum team since most of the organisations I know that are in the process of an agile transformation, or already have performed this transformation. But this is besides the point, for now.
Measuring an agile team's performance is basically the same as measuring anything else. Although, as we do in IT, we don't treat it the same.
When we measuring a team, we typically want to be able one of two things.
- Know how the team is performing.
- Know how the team has been performing.
Key Performance Indicators, KPIs, therefore are almost always associated with utilisation and thus with efficiency. Too bad that doesn't make sense in an agile environment since agile is not about efficiency but about effectiveness.
Measuring the performance of an individual team member or even a team is not helping in determining how the individual or the team is performing. And in my experience it is actually counterproductive and not supporting the agile transformation when your organisation is in the middle of one. We will see that we need to define agile performance either in terms of value creation (preferably business value) or in terms of an agile team's behaviour. This post is about the latter, where the first is all about Key Value Indicators, KVIs.
Back to metric basics
Going back to the basics of metrics leads us to reconsider why we do take measurements. And the main reasons I can think of are that you want to be able:
- To verify that a change has led to meeting the desired objectives.
- To ensure that you are on the right track to meet the desired objectives.
- To validate that you're more capable of meeting desired objectives.
As you can see, all these are around objectives, achievements and capabilities. There is nothing there about performance.
Metrics in an organisation
One of the main reasons I advise clients to implement metrics and take measurements is to ensure that it is possible to direct the course of action of an organisation. When it comes to measuring a team you want to make sure that the team is meeting the desired objectives, or at least is on the right track or that it is more capable to meet these objectives. And more importantly, the objectives are business objectives.
The more challenging part is that you want to be able to combine this with autonomy. In fact, from a leadership point of view, you want to facilitate the team to have as much autonomy as humanly possible. Which boils down to a trust issue. The amount of autonomy is based on the amount of trust. The more trustworthy a team is, the more autonomy you're willing to give the team. But I am getting ahead of myself.
One of the perks of a team in an agile environment is autonomy. The reason being that the fewer times a team needs to ask and wait for permission, the faster it can react on feedback. (It isn't for nothing that in many organisations that have embraced an agile way of working, there is bound to be found a poster that says something in the lines of "Ask for forgiveness, not for permission".) Agility is all about being able to react on a changing world in a timely manner. This is all there is to it.
Logically, the more time is needed to react, for example because certain signatures are required, the less agile a team can be. Consequently, when a team can make its own decisions, it will be able to react faster on feedback and hence the team is more agile. So the team needs to be able to process the feedback loop, whatever the feedback loop is.
There is a catch in all this, especially when you do come across that poster I mentioned before, the one about forgiveness and permission. The catch being that there needs to be a trust-relation with the team. And not only within the team, but with the rest of the organisation and the team as well. Trust in this perspective is what makes or breaks the agile ecosystem.
Trust
I think the first part of this trust relationship is between the product team and the Product Owner. There is always a discussion whether the Product Owner is part of the team or not. In my opinion the Product Owner should not be part of the team, but instead is the link between all stakeholders and the team. The PO is accountable for the products the team creates and is accountable for the business value of the products. (Mind that I am talking about accountability, because I am genuinely convinced that metrics and measurements are directly linked with accountability.) Where the PO is accountable, it is the team that is responsible for the product being delivered. It should be clear that there needs to be a relation of trust between PO and PT, obviously. Because when you’re held accountable for something that somebody else is responsible for, you need to be able to trust that other person. And when that trust is not there, you want to apply as much control as possible. The normal reaction of anybody that is given a task to do, but is also told how to do it, would be a clear “Don’t you trust me?”. For many managers, especially in the more established organisations, it will be a challenge to tell their direct-reports to do something without telling them how to do it. They will have this attitude only towards those that are identified as the manager’s protégé or right hand.
Now take it a step further, which I think is required in an agile environment, if not only because otherwise things wouldn’t scale. Taking this step will mean that the team is not told what to do, let alone how to do it, but instead the team is told to meet an objective. The PO is giving the team an objective instead of a list of tasks. Basically this means that the focus is on meeting objectives instead of executing tasks. The team is responsible for meeting the objective and is allowed to use any means necessary. Albeit within limits. Trust is now about “trusting the team not to overstep set boundaries”, instead of “trusting the team to do as told”. What is interesting with this paradigm shift is that the team is now responsible for getting the job done instead of for executing the tasks at hand. Consequently, trust is gained when jobs are done… and this means that trust is gained when value is delivered. And that is exactly what the PO is held accountable for. The loop is closed.
Read my posts [You know you're the Product Owner...] and [The Power of Accountability] on the topic of PO's and their accountability as well.
Now when it comes to trust in an agile world, we're bound to consider how things work in product oriented organisations. Ideally these organisations work with Product Owners that are held accountable for the product(s) they own. And the Product Owners work closely with product teams to develop those products. The PO has the mandate to change the priority of the features, the functionality, that is developed by the team, which is a (reasonably) stable team throughout the lifetime of the product they develop.
The Accounting Dream
Interestingly enough this should be the financial controller's wet-dream since it takes away one variable from the P&L equation. In a traditional project oriented organisation, year over year there are two variables that determine the P&L of a product development department (e.g. an IT department). For one there is the project portfolio which is different every year and may very well change during the year as well. And secondly there's the variable of the resources needed to do these projects, the people working in the projects. Year-over-year the staffing requirements will be different, resulting in budget headaches as well as staffing headaches.
So on the cost side there is a lot of uncertainty, but also on the revenue site. Because revenue is always expected revenue when it comes to budgets.
Do consider the case where the team working on products is stable throughout the year, and all of a sudden a key variable has become a constant. Mind that typically staff is a major cost aspect of any organisation. Of course the other variable, delivered product or rather delivered value, will remain variable. But that is fine.
I am always very curious why organisations have such a hard time with using stable teams. Especially since in a lot of large(r) organisations the CIO reports to the CFO instead of the CEO or even the COO. Wouldn't the CFO love to have to deal with one less variable in his books? I guess that's why I almost flunked economics classes in high school, I don't get this. My guess is that most of the time, organisations are focusing on efficiency instead of effectiveness, and that means that people need to be kept busy as much as possible, which means that they can't work on a new business proposition when they're already working on another product. So new people are hired instead.
Trust and Agile - I
Having stable teams does not only help the CFO, it also means that a team can be assigned to work on a product and gain business knowledge about the product instead of only technical knowledge needed to develop the product. The value of the team itself in terms of the organisation is increasing as well.
With a PO on the product defining what the team is working on in what order with the proper mandate, should ideally result in on the one hand stable so-called 'pre-financed' teams. On the other hand it should result in objectives to be met by the PO from a business perspective.
Remember that the PO is held accountable for the (lack of) success of the product that is owned. From that perspective, the PO will be extremely motivated to understand the product(s) that are owned and especially the value created with or by the product. Unless the PO does understand this, the product will lack success and that will be on the PO's plate. That motivation will very likely be intrinsic as well. Since a lot of the value will be in pretty much all cases be depending on timing of the release of the (new) product (version) to the users, the PO will need to be able to control this as well. Which is also a major benefit of being in charge of feature prioritisation.
Referring back to some earlier paragraphs in this post; the PO is accountable, but the team is responsible for the development of the product and its features. And with quality already covered as being a matter of trust. It is now time to consider timing a key aspect of the trust-relation between the PO and the Product Team. In project oriented organisations the quality aspect is also an important trust relation between PM and the Project Team. But in an agile organisation, the PO is depending on the timeliness of his product as agile means being able to respond in a timely manner to a changing market. Hence the PO will need to be able to also trust the time aspect along with the quality aspect of the Product Team as both directly relate to the success for which the PO is held accountable.
Just as a side note. In almost none of the organisations I have worked with, a project manager was held accountable for either the budget or the timelines. In fact the PM is typically receiving both as constraints or requirements to be dealt with. The PM is responsible for meeting timelines and staying within budget, but since there is no mandate for either, the PM cannot be held accountable. You simply can't be held accountable for something you have no control over. Even legally this is the case.
Gaining trust
I think that the general theme of this post is about trust. I am convinced that the relationship between PO and team is one primarily determined by trust and respect, but typically respect is coming with trust. Automatically.
Even between the PO and the customer there's a trust-relation. But since this post is about measuring agile teams and not the PO, I'll refrain myself from that aspect in this post.
The general question is: How does a team gain the trust of the PO? The answer is quite simple: The PO must be able to depend on the team. Which does make perfect sense, since the PO depends on the team when it comes to the PO's accountability.
As a consequence, a product team's performance is closely related to its dependability. Which you'll likely consider counter intuitive. Reason being that performance is typically associated with speed. Adding to this that traditionally we add the cost factor to it, and we associate performance with efficiency as well. So good performance is the highest speed at the lowest cost. Which in an agile environment doesn't make sense.
First of all, agility is about effectiveness instead of efficiency, so forget about lowest cost start thinking about highest revenue. Agile is about being able to seize the opportunity. Respond in a timely manner, remember. The speed factor is about being able to change direction while still delivering value.
With this, gaining trust means showing consistency in delivering value and ability to respond.
Measuring performance in terms of Agile Traits
With the above in mind, we now know that we need to measure performance of a team in terms of agile traits. Because when we talk about agile, we talk value driven. And that is business value. So when we talk about performance, we talk about business performance and not about performance in a technical sense. And business performance is all we should care about in the first place.
How do we measure business performance in terms of agile traits? That's easy, we need to make sure that we have metrics in place that focus on what we want to know, which is: How dependable is the team? And the one (and only?) person that should worry about is, is the one that depends on the team: The Product Owner.
In my experience there are only two aspects here that make sense to have metrics on. First of all we need to accept that trust to a high degree is emotional, subjective, so we need to make it objective. And that can be done by measuring a team's commitment. Secondly, we have to take quality into consideration. This is because the PO depends on the quality of the product, because only something that is actually used can deliver value.
Commitment, in my experience, is best measured in keeping track of what a team commits to working on during a sprint. Since the team will tell the PO what they can do in the allotted time and a good Scrum Master will see to this, the team will commit itself to deliver those features, that functionality. And we also need to keep track of what the team actually delivers at the end of the sprint, because that gives us a clear understanding of not only what is being delivered, but by looking at the difference between what the team was committed to deliver and what was actually delivered we know whether a team over- or under-commits. Over time you want this difference to be 0.
And since there always is a subjective part on the trust side, the PO will have to take the burn-down chart in consideration as well. As long as the PO is available to review features that are reported to be finished as soon as they are. And as long as size of the work being done is small enough, the burn-down chart will be more or less a straight diagonal downwards, starting at the top on day one of the sprint and all the way down at the bottom by the end of the sprint. Any deviation will require investigation.
As a side-note: Mind that this also requires small work items. My experience is that the best way to do this is when the team takes as a rule of thumb that a work item needs to be 'finish-able' within a day, max 2 days, but preferably one day. This has a number of benefits. One being that at the team's stand-up ritual the PO can listen in and have an overview of what will be made available to be reviewed by the end of the day, so time can be reserved to do that review and feedback from the PO can be input to next day's stand-up (via the Scrum Master or the developer). In addition, anything that wasn't finished most likely will need help from another team member, which can already be signalled after a day. Another benefit is that, especially when working with Behaviour Driven Development (or Test Driven Development), every story addresses a single behaviour.
What you want to see from a team is that the difference between committed and realised work items is close to zero with a steady burn-down chart. When this is what you get as a PO, you know that your team is dependable when it comes to promises.
Then there's the quality of the work being done. This is a little more involving, but shouldn't be too hard. In general it means that it is being tracked how often the product doesn't work/perform as advertised. The challenge here is more often than not that there needs to be a feedback-loop in place in order to be able to collect this metric. A truly mandated PO will have the loop in place, but I see more often than not that the loop is not there, typically because the PO is not truly mandated. The fall-back scenario is counting incidents. But ideally you want to keep track of the user's perception as more often than not the main quality problems are related to user-experience.
Nevertheless, regressions need to be tracked as well. Maybe this is even more important than first-time issues. Because regressions mean that the team's product development lack quality, which is far more serious than a product lacking quality.
Conclusion
Concluding there are only three metrics required to get a good grasp of the (business) performance of a team:
- Difference between committed and delivered by the team.
- Number of 'incidents' where the product doesn't behave as advertised.
- Number of times regressions occur.
Based on these metrics, the PO will be able to judge whether or not the Product Team is dependable and thus can be trusted with the responsibility to develop a product with which the PO can create business value for which the Product Owner is accountable.
To-be-kept-in-minds
There are some things to be kept in mind on this topic. For one, there's the metric velocity. The velocity of a team is a team internal metric and I really want to advice to not ask for it at all. Velocity is based on extremely subjective metrics, typically story points. These can be gamed in order to maintain a constant velocity. Which defeats the purpose of story points, as they are a team internal indicator regarding what the team will find on its path during the sprint. It can refer to complexity, dependencies, etc. Anytime story points are related to time to be spend you know you're using them wrongly.
Secondly, although it is very tempting to compare teams, you should try to refrain yourself from it. Instead you should look at how teams compare considering in progression based on the metrics. It is way more important to know whether a team is evolving, improving, or not. Teams that do not improve or not as fast as other teams are most likely facing an impediment for their growth, one that the PO cannot control. This is where intervention is required from management.
Thirdly, in general, and especially in agile environments, metrics should be based on objectives and achievements and not on tasks and activities. There is only one reason why one would want to know about what a team is doing, or rather how a team is doing things, and that is because you want to distil practices and identify best-practices.
But what about Team Happiness?
That is an interesting question. Team happiness is an important metric to measure a team's performance as well. This is not so much a metric from an agile perspective. I would say it is a leadership metric as leadership should be held accountable for the happiness of the teams. I will cover this in another post. As other metrics that are important.
Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.
Arc-E-Tect
The text very explicitly communicates my own personal views, experiences and practices. Any similarities with the views, experiences and practices of any of my previous or current clients, customers or employers are strictly coincidental. This post is therefore my own, and I am the sole author of it and am the sole copyright holder of it.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.