March 18, 2018

The natural beauty of organic convergence

... in an agile world.


I am a big proponent of the concept of 'Organic Convergence'. Something I have come up with and what goes on from the assumption that within a social group norms and values ​​become a standard, a common ground. The degree of interaction between the members within the group is decisive for the speed with which this happens and the benefits that can be reaped from it. More interaction means faster and more valuable convergence. And in doing so, I foresee that when the different teams work together, forming a social group, uniformity will result. With the added benefit that the result is broadly supported but not imposed.

The world is organic and evolutionary. At times it is revolutionary, then we are dealing with disruption. Disruption is beautiful because that turns you into a market leader and increases your chances of survival. This, of course, only when you are the disrupting force.

So where is this coming from? Well, it's coming from a discussion I had several times in the past weeks about Canonical Data Models (CMD), Enterprise Resource Models (ERM) and centralisation of governance in an organisation that just decided to execute a full-blown agile transformation over the coming years.

This organisation's legacy is, like so many other organisations with a challenging bureaucracy, is one of centralisation of capabilities (see my view on this in Perish or Survive, or being Efficient vs being Effective and an duo of posts on Competence Centres in huge organisations here and here) and a fairly directive management style. Although, as in many organisations, this one has come a long way, but still the overall cultural view is that teams are not autonomous nor self-proficient and Enterprise or Domain Architects are to define standards across the enterprise. Mind that these standards are not on enterprise level topics like principles, frameworks, etc. No in many cases the standards are about what technology to use, what architectural framework to use, or what model to implement. Down to the level of programming language and style.

Some context on the CDM

In this section I am elaborating on the concept of a CDM and why it is a remnant of centralisation in IT organisations. You can skip to the next section if you would like so.
The discussions I referred to at the beginning of this post were about whether or not an enterprise wide model had to be enforced top-down. Or possibly should emerge bottom-up. The discussion was about CMD (Canonical Data Model) and ERM (Enterprise Resource Model). Now the thing here is that the 'M' in the naming is not helping. We're not talking about just a model, but also its implementation. The CMD is a remnant from the days in which Enterprise Application Integration (EAI) was something new. In the late 1990's and early 2000's integration of software systems and thus creating synergy was key in many IT organisations. Different systems used different information models. What one system considered a Customer would be different from another system's definition. Typically across business domains these differences were huge. Then there is the problem where two departments in an organisation have different names for the same concept. Understandably, the CDM is going to be extremely helpful. In many ways, a CMD resembles the synthetic language Esperanto.
But unlike Esperanto, the CDM is a model. It's a definition and not an implementation. It becomes all the worrisome once the CDM gets implemented using one single implementation. The problem here is that definition and implementation all of a sudden become one. This is a problem because implementations require technical specifications like data-types and formats, which diverts the discussion from semantics towards syntax. Or in other words, the CDM becomes an IT concern instead of a business concern.
Typically the CMD gets implemented in a centralised part of an architecture. The main database, the ERP system or as part of the ESB. And historically this sort of made sense. When the CDM was invented, it had to also be implemented and preferably in a single location. Considering the focus on efficiency in IT, centralisation was paramount. Definition and implementation would go hand in hand, developed in parallel. Because of the challenges at hand, mainly the integration challenges, the CDM evolved into the 'language' on the ESB. Everything on the bus would speak the same language, when needed a connector would translate from and to the common language.

In those early days of EAI, the challenges were merely integrating the organisation's own systems. All within the organisation itself. Considering the context, a CDM made perfect sense and I would argue that based on the peculiarities of these environments, having a CDM and use it as the common language on the bus is the right choice. Think about the CDM not being Esperanto, but Hindi in India. Within a very neatly scoped context it was a common language.
In many occasions I have struggled with inconsistencies in information models for different enterprise systems and a CDM would have been or was the best solution. In my experience, the best way to deal with a CDM is on the ESB. Selecting the ESB as the integration model of choice allows for the best leverage of a CDM implementing the integration.
When you add a centralised department that will govern the CDM to the mix, preferably with the organisation's recognised authorities, like domain architects and enterprise architects, and you're in for a treat. Mind that a CDM needs tight governance and is high maintenance, so centralised governance would allow for relatively low costs and huge gains. Even in environments where the number of to be integrated systems is small or even tiny, like in the 1990's and the first 5 years of the 2000's, it is important to have just a single definition of each item in the model.

The Directive Culture

If you assume that you know what is necessary from a business concept perspective, then it is obvious to centralise and decide for yourself what to do next and how. This is how traditionally CDM's came to be around and currently that translates into Enterprise Resource Models. You'll find a group of (self-proclaimed) experts that define an enterprise wide business concept model (ERM) and by decree require everybody to use this model for all inter-operability between business applications. Within the context of a small business scope, like a business unit, this is sometimes doable...

However, in today's complex organisational environments this is no longer feasible. For all sorts of reasons, but within the context of today's enterprises, the complexity, also in vocabulary, of cross domain businesses and the movement to work more and more on short-term goals with ever shorter cycle times, make this approach practically unfeasible. For the simple reason that centralisation automatically also means that you introduce a bottleneck. After all, everything has to go through the same funnel in the organisation. More teams that invoke this centralised department more often generate a higher burden, which can only be solved through organisationally scaling-up, i.e. more people. And that is costly if at all possible. Automation is not a solution either, because this central department has to invent and deliver new things. Think of it as a team of linguists inventing a new language, enforcing everybody to use that language, but not teaching them to speak the language. And at the same time, requiring this team to come up with translations of new words from a multitude of languages into that new language. It's just not doable.

It get worse when it is expected of this centralised team to not only come up with translations, but to actually invent new concepts. And keep in mind that it is often these teams themselves that have this expectation of being able to prescribe the ERM and/or CDM.

The answer to this problem is to decentralise this task. Something that is not unheard of, because crowd-sourcing has been the solution for organisational scalability issues for years. The central role is then one of governance. This requires two major efforts.

  1. Clear definition of what the rules are. 
  2. Clear processes to maintain the rules.

Mind that both are critical for success and extremely hard to accomplish. Granted, defining a workable set of rules by which to live is hard by itself and requires not only the proper mindset and experience, but also intrinsic authority. But unless there is a workable process to maintain these laws in order to keep them relevant and actually act accordingly is one that requires quite a bit of perseverance.

The Meta Problem

The interesting thing about this is that it is almost a meta problem. Because with the introduction of another centralised function there is again a bottleneck. Scaling-up will again become a problem. However, now there can be some automation. After all, good rules are ideal for automating verification of compliance. When the work is automated, decentralisation of the execution of that work, bringing it closer to the source, nothing stands in the way. From the LEAN idea, that is also a specific form of waste to eliminate.
And this is where it becomes interesting. After all, is it not necessary to check that these automated processes are being carried out? And of course the answer is a well sounding and heartfelt 'Yes'. And if we set off from a Directive Culture, we will of course centralise that control again. But in a Learning Organisation that is of course a no-go, because centralisation does not solve the problem and ultimately results in decentralisation. Much more valuable for the organisation is that on the basis of accountability and responsibility the culture is such that there is a clear sense of ownership in which quality assurance is intrinsic. This is not possible if your starting point is scepticism regarding the ability to take ownership, or in other words to succumb to the tendency to create a controlling authority. The basic assumption must be that quality assurance becomes a common practice that can be realised with (intensive) guidance and positive incentives.

The above therefore argues for and will result in an environment in which trust predominates and is therefore a good breeding ground for a safe working environment.

The Decentralised Models

First of all I want to establish the notion that for an ERM or a CDM to work, we need to remember that these are merely models. There should never be an implementation aspect involved with these models.

For these models to work, it is required that they are broadly supported within the user-base. The model will only be adopted by a user when it is applicable in her problem domain. In other words, the model needs to be relevant. And relevance can only be established when there is practicality.
If you think about this, you'll understand that in general there, initially, not be a single resource model that can be used across the whole organisation, probably not even one that can be used across value-chains. Which of course makes perfect sense, because two separate value-chains implement two separate business products, which in turn will address two separate business needs. Hence, different business concepts will be at play.
It would be futile to assume that one can come up, out of the blue, with a resource model that can be applied in both of the two value-chains. And any attempt to do so, should be stopped, immediately. Instead, two different models should be created and a third as well when a third business product is developed.
Meanwhile the involved architects, both domain and product architects, should start identifying the commonalities and differences in all resource models and formulate a shared model across these value-chains. (Read my post on agile architecting to understand how architects provide a huge benefit to organisations.)
This interaction between architects, especially when you join domain and product architects in this interaction, will result in a core of related business concepts that form a model on which the organisation defines its business products. Generates business value if you will. In a large organisation with several business units, operating in different verticals, it is not uncommon to have several of these core models. It would be the wrong activity to combine these into a single model. Doing so would result in an artificially created model which is not applicable other than in academic discussions and of no benefit to the organisation.

Organic Convergence

The model that stems from this, be it a resource model, a data model, an information model, is one that emerged from practical use. By definition it is therefore applicable and usable. In addition, the model consists of aspects of several models that converged such that commonalities and differences are considered and leveraged to add value.

I call this 'Organic Convergence'.

It is something came up with during my discussions and what logically results from the assumption that within a social group norms and values ​​become a standard, a common ground. The degree of interaction between the members within the group is decisive for the speed with which this happens and the benefits that can be reaped from it. More interaction means faster and more valuable convergence. And in doing so, I foresee that when the different teams work together, forming a social group, uniformity will result. With the provision that the result is broadly supported but not imposed.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.


March 6, 2018

The bright future of the CC in the agile world

This is the second in a two part blog post on Competence Centers. Read the first post here.


The role of Competence Centers in large organisations is diminishing. In many organisations there actually is a notion forming that these centers are obsolete. Considered being a hindrance in an ever more agile environment, the roles of the engineers in these CC's are no longer found in job descriptions. But on close examination, we see that where Competence Centers are fading a way, platform teams are arising as the new home for these experts.


In my previous post I elaborated on the reasons why in today's day and age of business agility the Competence Centers are no longer wanted. The main driver is most likely the notion that we no longer see IT as a way of doing business more efficiently but more effectively. Take this in for a bit and try to see the significance of this difference in looking at IT.

In my post you could read why the CC is obsolete and why maintaining a CC in an organization that is transitioning towards (business) agility is bad practice.

With that said, I want to explain why I think that the CC still has a place in today’s agile organisations, but in an extremely different way than we’re accustomed to. In my post I hinted to this already in the final part of the post.

Competence Center Classification

Typically Competence Centers consist of experts in a particular functional but more often than not, a technical domain. In my experience, large organisations that benefit from a CC have CC’s addressing aspects like application integration (teams of specialist for a specific ESB product), database systems, infrastructure and networking, mobile app development and often it is the case that there is a team that is very proficient in collaboration software like Lotus Notes, Sharepoint etc. What you’ll most likely also encounter is a group of experts organized as a CC for capabilities like testing, project management and coaching on specific topics (like Agile, Scrum and other new hip buzzwords).

I think that coaching is something that is really of extreme benefit to an organization and having a group of coaches in your organization makes perfect sense. I see them more as grease to the software development machinery than a way to make products more efficiently. Good coaches are the conscience of a team, a department, an organization. They continuously and relentlessly challenge an organization on a variety of aspects.

Capabilities and roles

Read my post on ProjectManagers to read more about how I feel about their role in today’s product development environments. Competence Centers around testing and other activities are to some degree, a large degree, a waste of time and effort. Don’t get me wrong, you really need to test, but that should be part of the team’s efforts and not, definitely not part of an external, at least to the Product Team, team. Testing is part of the product development cycle and should be done by the Product Team as an integral part of that cycle. It should also be automated as much as possible. The same goes for Requirements Engineering, who are often smart enough not to call themselves a Competence Center, but still act as one.

About Testing and Requirements Engineering a departments alike, there’s only one thing that makes sense, which is that they should be dissolved, their members become part of product development teams, considering they do have an engineering mindset. And the practice of testing and the further development of a testing practice should be instigated by a Test Engineering Guild. Same obviously goes for Requirements Engineering etc. (Guild: [...] an organization of persons with related interests, goals, etc., especially one formed for mutual aid or protection [...]). I really don’t think that it makes any sense to have a testing department, and it is counter-productive to have such a department and have testing done outside of the Product Team.

Architecture Centers of Excellence

There’s also the typical architecture group in many large organisations. They are often working in isolation of the developers. There’re a multitude of reasons why there is a distance between developer and architect and often it is due to management expectations. This also results in a reputation that is far from good when it comes to architects. Read my post on the role of architecture and architects in an agile environment. I really am convinced that architecture is a key aspect of real business agility, real meaning measurable instead of gut-feeling and goose-bumps. Especially when an organization needs to be agile in order to work on their long term strategy, i.e. adopt business agility as a strategy, they need to have architecture at the core of their product development. Architects should not be in an Architecture Competence Center, often called Architecture Center of Excellence. Instead, they should be extremely close to their user-base and form a guild. Senior management should abolish any formal group of architects and facilitate an architecture guild, up to the point where active architects in such a guild should be openly credited for their practical contributions to their users.

Back to the more technical CC’s.

The Engineering profession

As I mentioned before, CC’s are often consisting of an organisation’s experts. Which makes sense because the whole of the organization should be able to benefit from their expertise. Mind that this notion and the creation of a CC means that the organization understands the problem of scaling. Experts can’t be scaled up, i.e. cloned, so their time is booked as efficient as possible.

Because the CC will become a bottleneck, we need to find something different to solve this problem. And there is a very effective way of doing so.

Technical Competence Centers should be transformed into Platform teams. Platform teams are teams that develop a product, just like Product Teams, but instead of building business products, they create a platform onto which products can be run. The clearest example would be a hosting platform, developed by a traditional infrastructure team. Business products need to be hosted and instead of every Product Team catering for its own infrastructure to host their products, the hosting platform is leveraged. This happens automatically when products are hosted in the Cloud. The likes of Amazon, Microsoft and Google to name three behemoths, understood early on that infrastructure should be a commodity. Since hardware has been a commodity since the advent of the PC, and with the advent of virtualization technologies server hardware joined these ranks, infrastructure as a service (IaaS) has become the norm. Mind that there is still a lot of proprietary hosting platforms out there, we call it on-premise, but what it entails is a sub-standard platform.

Too often I hear that SysAdmins are reluctant to create a true platform of their on-premise systems because of job security. Cynics stating that once they’ve created a user-friendly experience in their data centers as found in the cloud, the SysAdmins are out of a job, are in my opinion wrong. I think that unless infrastructure in the data center is made as accessible to developers like the Cloud is, the SysAdmins will be out of a job. The fact that all around me I hear people nowadays referring to infrastructure as a ‘hosting platform’ instead of server infrastructure as was the case several years ago, means at least to me, that SysAdmins understand this.

But infrastructure is not the only platform to be considered. Take any of the examples of CC’s I gave earlier on in this post. For example the database experts should be working on a database platform instead of creating and maintaining databases. The database team would create a platform that allows for Product Teams to host their database on. It would ensure all compliance aspects, performance, security, resilience etc. The database platform would be an extension of the hosting platform, building on top of it. Consider the situation where the supported databases are SQL Server and MySQL. Product Teams would use these when needing a relational database. But when requiring a NoSQL database, say Mongo, they would do it by themselves on the hosting platform. Who would be better equipped to create a database platform then the database experts that where in the Database CC?

A similar approach would be taken with the former Application Integration CC. They would develop an Integration Platform that other teams can leverage for their integration needs.

From required-to to required-by

The point with a platform is that it is a product as well. So there are users of the platform and it is the Platform Team’s task to ensure that their product is adding value to the business. This means that the platform they create supporting a more agile business, that the platform is user friendly and addressing the user’s needs as much as possible. The user here is the Product Team or the Product Teams building their products using these platforms.

There is an immense shift in mindset here that you should pay considerable attention to. What I mean is that the expert in the CC was required to be invoked in the product development process as part of the process. The product had to be integrated with other products, so the Integration Expert had to be involved. If not only because the means of integration are so complex that mere mortals shouldn’t be allowed to access them. The same goes for databases. Nowadays almost everything needs a database, so in almost every situation the database experts had to be invoked to create one for the Product Team and maintain it.

The CC’s, being the result of rigorous cost-cutting and an efficiency-fetish, have become integral parts of the product development process. Policies dictating their involvement. But platform teams are different. They deliver a product that can be leveraged by Product Teams, but there is always an alternative if not only to do it yourself. The sense of autonomy in agile environments, dictates that Product Teams decide themselves how to develop their products. This holds true for every product, including platforms. So our Competence Experts need to shift from a ‘you have to run it by me’ attitude to a ‘you will want to run it by me’ attitude. And that’s a huge shift.

Many of the people I know that are part of a Competence Center, a Center of Excellence or similar organisational body are really having a hard time wrapping their heads around this. When explaining to them that they’re now supposed to create a product that people will have to want to use instead of just have to use, they have a hard time to fathom what this entails. In my experience, platform teams are among the teams that are the most urgently in need of intensive coaching on concepts like agile and lean.

Not Platform but Product

These teams are often called Competence Centers, but they are merely Product Teams grouped around a certain business aspect using specific development technology. The classification of ‘Competence Center’ is not warranted and these teams should avoid becoming a platform team and instead focus on obtaining business knowledge. Since they are in fact developing business products. Interestingly enough, these teams are more and more converted into Competence Centers although they are probably the teams that are actually in close contact with the user. Being a specialist in a certain technology, or being a team that uses a specific technology to create value doesn’t make you a Competence Center, and certainly not a Platform Team.


The role of the Competence Center in large organisations is diminishing as agile development methods are adopted. The shift from efficiency towards effectiveness is emphasising this even further and CC’s will be obsolete in most if not all cases. Does this mean that all efforts exerted in the past in honing the skills of the experts corralled in these centers were in vain?
Absolutely not. Guilds, Platform Teams and new Product Teams are taking hold and further enhance the agility of the business. Provided organisations are adopting as part of their Agile Transformation strategies also concepts around Autonomy, Facilitating Leadership and User-Centric Design, the future of those that once formed Competence Centers will be bright.

Thanks once again for reading my blog. Please don't be hesitate to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Please subscribe to my blog to show your support.