When Scrum needs to mature in the enterprise, architecture comes to the rescue.

This post is the consequence of a comment on my previous post that startled me as it stated that in Scrum there's no place for the architect. Scrum has no future were it not for the architect.

First of all, I am not a fan of Scrum, mainly because it sounds too much like an acronym and I hate acronyms. The other reason why I don't like Scrum is because it entices too many religious zealots to start a jihad against all non-believers. And then there's the third reason, the reason why I don't like Scrum is because too many so-called Scrum practitioners use it as an excuse for not going for longevity and quality of their deliverables.

So, with that of my chest, let me tell you that I really love what Scrum stands for. That whole thing about delivering something useful to who ever pays you for building it as soon as possible. Allowing for the customer to change his mind constantly about what is important and what not. And the very notion that sometimes you do something and than you have to redo it but differently is part of the deal is excellent.
Scrum is awesome and it addresses a lot of very significant aspects of old-school project management. Especially when it comes to long analysis phases, and even longer design phases and excruciatingly long development phases, Scrum has introduced a lot of benefits.

One of the main reasons why Scrum is such a success from an adoption perspective is the fact that it has been introduced in the limited scope of development project teams. And then in most cases those teams that had to face a lot of changes in requirements and priorities, namely front-end, user facing systems. Systems that directly addresses the needs of an end user.

As the common enemy called "customer" united all developers, Scrum thrived.

The rebellious attitude introduced with Scrum appeals to many developers, and I don't mean any disrespect to developers. I am a developer and often I am my own customer, or in Scrum terminology, I am my own product owner and I suffer from changing priorities and requirements all the time. It's part of usable software.

In many organizations these days, development projects are done with the Scrum manifesto in one hand and a sincere lack of documentation in the other.Here lies already a big problem, namely the fact that in these organizations people, even Scrum zealots, still talk about projects. In and by itself it is impossible to do projects using Scrum. The very notion of projects is preposterous, but that'll be covered in another post sometime.
These enterprises have introduced Scrum in their software development projects and those organizations that are a bit more mature have introduced DevOps, which is the second stage in Scrum-olution if you ask me.

Havoc is wrecked when a Scrum team is developing software, functionality that depends on the deliverable of another Scrum team. How are sprints aligned? How are interfaces co-developed, how are release dates, even in continuous delivery situations, coordinated? Scrum, nor DevOps, can answer these questions. The reason for this is that the scope of Scrum was never intended to go beyond the activities of the Scrum team. Inter-scrum-communication is, well not existent.
Of course it exists, because it happens all the time. Initiatives like meta-scrums, meta-sprints and meta-other-Scrum-buzzwords are popping up all over the place.
The fact is that once you look beyond the scope of sprints and epics and what the Product Owner is looking at from a back log perspective, there's not a lot that you can't can do without the quality and talents of an architect. It is always the architect that delivers the bigger picture.

Scrum is, I believe, something that stems from the sport rugby. And if not, than still for the purpose of this post, it does. And where the Scrum teams play the game, it is the architects that define the playing field as well as the rules by which to play. Mind you, Scrum does not equal developer's anarchy. It allows for the freedom of the team, if suitable, to apply a pragmatic approach to realizing the product owner's products. But the freedom is within the limitations set forth by the architect in terms of policies, principles and standards. Because of the architect, features integrate and systems can communicate. But more importantly, it is architecture that allows for cohesion and consistency between different applications and services, thus ensuring for example compliance to laws and regulations. Especially in environments that are predominantly based around SOA (WS* and REST both), it is imperative that policies and standards are defined and adhered to in a consistent and cohesive manner in order to be and stay compliant.
This becomes very apparent when principles like:

• "One Version of the Truth"
• "Re-use of business logic"
• "Role based Access Control"

Are to be followed. Without an architecture framework, it becomes extremely expensive from a governance point of view for an organization to enforce or even ensure adherence.

Do you need an architect in a Scrum team? Probably not as long as you've got one or two members in your team that are taking care of the relevant design work, whether or not implicit to the Sprints deliverables. But in the bigger picture, there is definite need for an architect, especially when you not only want your development teams to be agile, and capable of adjusting to the whims of the product owner but also your whole business to be agile and capable of adjusting to market demands.

Thus, unless you want to keep Scrum to be Waterfall's baby brother for ever and ever, you need to mature your agility and that is only possible by applying architecture and involve architects. But not so much within Scrum teams, but across these teams.

Oh No!!! Our environment is homogeneous, we are doomed!!

Abstract: What you want, or rather what you need to be successful is a heterogeneous environment. There is no place for homogeneity in today's enterprises.

Hi,

Lately I've been involved at a number of discussions regarding application replatforming. The main reason for the replatforming is to make the environment more homogeneous. The reason for this is that there is an overall perception that this will allow for a lower TCO of the complete IT landscape. Which is, or seems to be, the goal of many architecture initiatives lately.

I, unfortunately, have to agree with this. To a point. And yes, no typo there, I do state that it is unfortunate. I'll get back to that later.

I agree to a point, because homogeneity is the result of standardization and when we talk about a homogeneous IT landscape we mostly think about far reaching standardization of the IT assets. Typically the infrastructure, operating systems, middleware, database (I don't consider databases to be middleware), and even business solutions like CRM, ERP and the likes.
The more you can standardize here, the more of the same you'll get in your environment and therefore the more benefits you'll get from the economy of scale as a result. As such, economy of scale results in lower TCO as typically you'll be able to do more with less, especially in the area of support and maintenance, i.e. the operation. Add to this some form of virtualization and you'll be utilizing your resources, computing resources that is, more efficiently, meaning less hardware investments.
There's another benefit here, that has less to do with economy of scale (and arguably the TCO) and more with standardization and therefore quality. This is the fact that when you standardize on your assets, you can also standardize on your processes to maintain them once you've reached a high level of quality and more importantly, the people you have to support your environment need less (diverse) training and can achieve a higher level of quality by experience. Ask any SysAdmin how long it took until they felt really comfortable with supporting an OS, and they'll mention years of experience.
Take it up a level and you're in the realm of middleware and databases and this is where you can really lower the TCO by closing strategic enterprise license deals with middleware and database vendors.

So where am I getting at you might ask yourself. Well, that's easy. All the above is why many if not most of the enterprises and SMB's for that matter, do standardize and do want to get to a homogeneous IT environment. I've been there myself as well. Evangelizing the mantra of homogeneous, standardized IT environments.

The issue the issue here is that we do all this work out of reducing cost. And the reason for that is to improve profitability. And this is because IT is most often seen as a cost-center within and enterprise just like staff, so IT reduction, just like staff reduction is something you do when times are challenging and profit is hard to be made. This is because the accountants dictate this. Look at the books, where are the most costs on the balance sheet, cut those and you're done. Awesome...ly narrow sighted.
What we tend to forget is that IT, just like staff, is an enabler. An enabler of revenue and therefore an enabler of profit.

Let's think about that for a second, forget about TCO for convenience because I am aware of the 80/20 rule when it comes to IT expenditure. So instead of thinking of IT being a cost-center, consider it to be a business enabler. Ignore the screamings of accountants and listen to the mantra of the business people. Strangely enough, they hardly ever want to cut costs. They'll tell you to reduce costs because that's what the accountants say, what the CFO dictates, what the shareholders say, not because they think that's the right course of action. Really, they don't. I'm 110% certain of that because otherwise they wouldn't ask for more new business (projects) to be realised and instead would order IT to execute projects to reduce IT expenditure.

So those business people in the enterprise want new business projects, pursue new business ventures, address new markets, jump on band wagons. It's all "spend, spend, spend". Why? Because they want more revenue, because that means more opportunity to improve profit numbers on the balance sheet and it is more fun and it is more long term.
IT therefore should not focus on being less costly, instead IT should focus on being more business enabling. And trust me when I say that a homogeneous IT landscape is less enabling than a heterogeneous IT landscape.

Do you get my 'unfortunate' at this point in the post?

Let me elaborate. First of all, heterogeneity is evil, unless you not only accept it (because it is inevitable!), but also embrace it. Choose heterogeneity as an architecture principle. Assume that nothing in your environment is standard. Design your solutions based on this, define your strategy based on that realisation. Unlike other assumptions that make an ass of you and me, this is a safe one to make. It's safe to assume that nothing is standard forever.
Thing is, everything is always changing, everything is always in motion so nothing can be standardized without the need to revisit your standards and as such what once was standard will no longer be. Hence, everything is or will be legacy. In the near future. An old saying in IT operations is:

Legacy is what we have in Production.

Not feasible? Not doable? Not wanted? Well, have you ever seen a fully standardised, homogeneous environment? I'll give you some slack and only ask you this question to be answered for only the OS level.
Forget about the layers underneath and forget about the layers on top of the OS. It's just not there, there are always different versions of the standard OS. And no, they're not the same because if that would be true, there wouldn't be the need for a difference.

Okay, so adopt heterogeneity and embrace it as a child that is always misbehaving but still it is your child so you love it, no matter what. But you can't wait until she's 18 and on her own. (Yup, I've blogged about off-spring and the need to be able to let go in a previous post)
This means that you need to learn how to live with it. Your processes of supporting and maintaining that IT environment need to allow for heterogeneity.

Why was that again? Uhm, I don't think I covered that, but if you still can't guess, think back of the IT being more enabling.
The trick here is that you need to understand that the IT is servicing the business, which means more revenue needs to be generated as part of the IT endeavors. So if the business wants something, IT needs to facilitate it. Without question. IT answers, it doesn't question. It also answers truthfully. And yes, this not only implies, but explies (if that were a word) that the environment is heterogeneous and will become more and more heterogeneous as time moves on. Which is awesome, because every request of business is delivered.
It's going to be a mess, unless you embrace heterogeneity. When you don't, when you require homogeneity, demand it in fact. Question the requirements of the business, actually be so arrogant as to know better what the business wants than the business itself, than you will not be able to deliver on every single request of the business. You'll be less, probably a lot less, than adequate and you'll be less of an enabler.

What you want, or rather what you need, in order to be successful is a heterogeneous environment.

There is no place for homogeneity in today's enterprises.

Really? No, not really. We're almost there... You can only do this when you do architecture within your company. I mean define conceptually how your environment looks like, model it. Remember it is only a model! Ensure that you have the right governance in place to apply this model. Govern.
Realize that in the above, I never stated that this is related to IT, it is related to the enterprise, however small it is. Also realize that nothing in there is fixed, in fact, all is subject to change. So don't consider the model as dogma (see a previous post on the topic of dogma), don't lay down a doctrine.
Make sure your conceptual model is backed by a logical model which is governed. Again, there's nothing specific IT in there. It's the enterprise again that is modeled on a logical level. Also remember that logical means that the model is logic to all stakeholders.
Easier said than done? Well, I would say, easier done than said as long as you do architecture in the right order. Meaning that business requirements dictate architecture dictates technology dictates tools dictates implementation. Most of the times we do it the other way around... that's why you end up with easier said than done, as that is also the other way around.

Ah, yes, there's the CFO and the shareholders who are kind of important in the eyes of many, and in any case they have that pile of money you want a piece of. And they want cost reduction, maintaining revenue, increase profits... This is where you want to standardize, but in an abstract way. Which means, move those parts of your heterogeneous IT landscape that are costly out, away from being your responsibility. In other words, where it makes sense, move your IT to the cloud. Again, don't make this doctrine, but move those parts that make sense to you to the cloud (read my position on the cloud with respect to enterprises in a previous post). These are the systems that you need elasticity in some form to be achieved, where you need certain facilities (security related, availability related, connectivity related) to be at a level where you just can't handle it yourself. This won't be an issue as long as you've been accepting and embracing heterogeneity. Because from that perspective, the cloud is just another something non-standard.
By moving to the cloud, you can actually benefit from the economy of scale... the scale of somebody else. And wasn't this the proposition of making things homogeneous? Of defining standards and standardize? As it is with cloud, it only works with the cloud provider because he standardizes. Not so much on getting a homogeneous environment but on the way you perceive that environment. And since you're not (completely) free about what the environment looks like (with IaaS more than with PaaS more than with SaaS more than with BaaS), you're forced to think about how things work together. About how the applications that meet your business needs provide these and enable your business to increase revenue. And with the limitations of the freedom you have on this, you're bound to standardize on a conceptual level.

So in short:
- Homogeneity -> IT is a cost-center
- Heterogeneity -> IT is an enabler

I hope this post has made sense, or at least has provided to you a different view on standardization in IT. As always, you're more than welcome to provide your insights. The more views, the more perspectives, the more complete the picture.

Thanks once again for reading my blog. Please don't be reluctant to Tweet about it, put a link on Facebook or recommend this blog to your network on LinkedIn. Heck, send the link of my blog to all your Whatsapp friends and everybody in your contact-list. But if you really want to show your appreciation, drop a comment with your opinion on the topic, your experiences or anything else that is relevant.

Arc-E-Tect


The text very explicitly communicates my own personal views, experiences and practices. Any similarities with the views, experiences and practices of any of my previous or current clients, customers or employers are strictly coincidental. This post is therefore my own, and I am the sole author of it and am the sole copyright holder of it.

Flexibility in the 21st century: Agility

Hi,

I've been a developer for pretty much ever since I got introduced to computers. First I developed for my Sinclair ZX81, then I moved on to MSX, made the transition to DOS in 1989 and moved along with Microsoft to Windows. I've developed on OS2 and various flavors of Unix and I've been using a multitude of programming languages ranging from BASIC to Modula2 and Pascal to C, C++ and Java.

I've been to university studying Computer Science in the 20th century and back then we were thought to design and develop for flexibility. The reason behind it has always been that you should be prepared for the fact that once you're done and you've delivered there will be new releases, new requirements, changed requirements, etc. Possibly even while you were working on the system you would be confronted with new insights, new ideas, new and more important requirements. Re-use was all the rage and you should prepare yourself for changes by being able to re-use 80% of the code, of the features, of the functionality already created and only add an additional 20% of design, code, testing and documentation for the new stuff.
Even in the wonderful world of OO, CBD and Modularity, the beat of the drum was about being flexible. Create a flexible solution, one that can bend and twist all over the place to accommodate for every change in the requirements. Be it functional or non functional. The whole idea was and with many of us currently still is, that when you design or develop something it should be so flexible that a new requirement would be already implemented by 80% of the existing code.

I remember having reviewed a system that was almost entirely written in database tables. An engine would simply read a set of tables and an object model was instantiated and executed. Which is not all that unfamiliar when it concerns for example workflow engines or rules engines. But this little piece of software concerned an actual programming language in database tables. I could write a simple "Hello World" application in the database. The reasoning behind this little piece of ingenuity was that there wouldn't be any deployment stress with testing of the software etc once the application was deployed. Just updating the tables would allow for new functionality. Of course until after the first update of the tables wrecked the system because there was no syntactical or semantical checking of the 'new program'.

Flexibility is something that needs to be provided by frameworks. This is why you would create frameworks. But the thing with frameworks is, is that it is extremely hard to actually write a framework that is actually usable as a frameworks. They tend to be too restricted to a specific purpose or too generic. Most of the popular frameworks started out as a specific solution that was repeatedly implemented, based on a well known enterprise design pattern. After the 3rd time of implementing the pattern the designer or developer recognizes the parts that are consistently changing and those that remain and he'll be able to start developing the framework. This even holds true for design-time frameworks.

With the advent of eXtreme Programming and Agile project management (any flavor) the world of software development also got an overhaul. No more lengthy analysis and design phases prior to development and instead short cycles to get the product out. In the old days (which in IT means 'not that long ago') analysis and design was considered extremely important because one wouldn't want to miss a requirements, be able to prepare for the next version, build in hooks and nooks to facilitate additions without corrupting the overall architecture. But the longer the time between conceiving an idea and realizing, the further apart the idea and the realization of the idea are. Hence the short cycles of today, which still have an analysis and design phase, but shorter. And no, one can't do without analysis, design in agile or even without architecture, but that in another post.

Now, what is so dramatically different in terms of flexibility in the 21st century compared to the previous millennium? Well, the difference is in mindset of the developer. The developer and even the designer creates for replacement not for changing. By this I mean that when designing or developing nowadays, one needs to keep in mind that pieces can be ripped out and replaced by something completely else. Without heart feelings. And yes, letting go of your baby while still in its infancy is a change in mindset. Today you create something that is not supposed to last forever, but something that lasts until the next big thing. No more stone carved COBOL systems that will survive humanity.

The big difference here is that every application is a business functional framework, one that targets a specific area of business functionality but caters only for that functionality that is required. New functionality is added in a consistent manner, existing functionality is removed when no longer valid or replaced by a new piece of code when required to be changed. Mind that the key here is 'replaced'. Code is not amended to implement the change. Because it is way to cumbersome to change the code, it is better, faster and more convenient to build the new functionality from scratch and add it to this business framework. This is flexibility through agility allows IT to build business applications that are consistently 'correct' in design and easy to support and maintain.
The whole premise of agile development is to scope an iteration purely to those requirements that are immediately necessary and developable within the constraints of the iteration (timelines, budget, resourcing, etc), while not taking (too much) into consideration what might be required in the next iteration. Which basically means that whatever you create may be thrown away in the next iteration because it conflicts with the requirements of that next iteration.
The idea that in agile development there is no place for analysis and design and certainly not architecture that many agile adepts have embraced is of course a consequence of architects, designers and analysts not being able to analyze, design and architect keeping in mind that every single component in the solution should be throwawayable. They're quite often still create flexible systems instead of agile systems, hence the conflicts with the developers.

Agile systems are designed by keeping in mind that the components in the systems are decoupled, on a functional level. On a requirement level that is, which boils down to the fact that the analyst needs to define the requirements and their impact on the technology such that the requirements themselves are decoupled from each other. Dependencies between requirements result in tightly coupled implementations. When two requirements depend on each other, they should be treated as one. Consequently the designer should design the solution for each requirement independent of the other requirement. Re-use of functionality already designed for one requirement to be met should not be in the design... or at least not in the first iteration of the design. Preferably, re-use should only be taken into account in the technical design of the solution if at all in the design. Why not leave this decision to the developers?
This will allow for the assumption that what is designed today will not be there tomorrow.

So where does this leave the architect. It is clear that every iteration will have its analysis and its design phase. You cannot get around it. It is the rule of problem solving: understand the problem (analysis), think of what the solution(s) might be (design), implement the solution that is most feasible if feasible at all (development). You just cannot solve a problem you don't understand.
But where is the architect in all this. Surely you can't architect a system for which the requirements are not clear at all, they're made up and prioritized as the project goes. Or rather as time moves on. The architect defines the architectural principles that the solution with every iterations needs to comply to, to safeguard the consistency of the solution, the project's deliverables, in the grand scheme of all systems and solutions in the environment. The architect will define that at all times user interface and business logic need to be separated in design and in implementation to run on different (logical) servers. The architect will define that users of the system that are external to the company need to explicitly be authenticated using a secure token, where as users that are accessing the system through the local LAN can be authenticated using userid/password only. The architect will define that the allowed technologies for implementing the solution should be platform agnostic, allowing the solution to be deployed on Windows, Unix, Linux and OSX systems. And the architect will verify compliancy with the architectural principles prior to deployment in production.
The architect is or heads the design authority.

Without architectural principles that guarantee the integrity and consistency of the IT landscape yet support or at least allow agility in this same landscape, agile methodologies will create chaos and therefore significantly increase the TCO of the landscape.

As always, I welcome any comments. Whether you agree or disagree or just find this post informative. Please let me know where you think I am wrong or right and moreover why you think this. It will be very educational for me as well as the other readers of this post.

Thank you for reading this far.

Iwan