Innovation Inertia — What, How and Why it happens?

First, some definitions:

Innovation in its modern meaning is “a new idea, creative thoughts, new imaginations in form of device or method”. Innovation is often also viewed as the application of better solutions that meet new requirements, un-articulated needs, or existing market needs.

Inertia, noun, a tendency to do nothing or to remain unchanged.

I put these two words together one night when I couldn’t sleep. You see something was frustrating me massively on a recent assignment. I was hired at the start of 2019 to be part of a successful e-commerce company on a journey towards an “Agile Transformation”. It was sold as the best and brightest minds from the business re-assigned to this high-priority project whose main aim was to split their massive monolith down into, you guessed it… Microservices!

When I started there was certainly lots of good things happening. The architecture had gone through several iterations before settling on a CQRS based approach to their API for products (which ran as a Spring Boot application on AWS ECS). The front-end served pages via a page composition layer which comprised of NGINX, NodeJS and React components. A very modern and impressive tech stack.

The teams were writing test upon test to ensure automation was at the core of what they were doing and manual testing was extremely light touch, if any. Performance testing was even part of the development workflow ensuring their services delivered the required performance for Black Friday and beyond. Design teams were crafting responsive designs based on the latest customer research. Our Infrastructure was code, Continuous Delivery was at the heart of everything, Stand Ups, Retros and Sprints were all in full swing too. Hell, we even had Squads, Tribes and Guilds.

The project had all the hallmarks of a modern software project which should pretty much guarantee success, right?

So then, why in over a year (at the time of writing) had close to nothing from this project made it in front of a member of the public? The only notable success had been a new “static” footer added to a low traffic page on the site as a test-bed. Very little else from this herculean effort to transform our architecture and the delivery culture had left the door.

We were still doing all of the aforementioned “Best Practices” — They were hiring like crazy, yet delivering very little.

This is what led me to come up with the phrase Innovation Inertia.

Culture

You’ve likely come across the phrase “Culture eats strategy for breakfast.” Accredited to Peter Drucker and later on Mark Fields, CEO of Ford. That quote is so, so, accurate. When you mention company culture you tend think of the free food, beer fridges and beanbags type culture. Culture, is more subtle and nuanced than that. It’s not localised to the t̵e̵a̵m̵s̵, sorry squads either. It’s more broad than the squads building the products. Culture extends to how the organisation operates, which silos control which pieces of the estate. For example, the production site is fronted by a Content Delivery Network (CDN). To get anything new into production required altering that CDN’s configuration; something that only a small team of people who trust no one but themselves and communicate via change tickets and CAB Meetings can change. Those semi-invisible barriers have been put there for a reason, my guess would be a knee-jerk reaction to someone screwing up a change causing an outage and the business to lose money. However, these types of roadblocks cripple a business’ agility.

I was once in a meeting where developers were berated for causing an outage and given a rough figure of how much AdWords budget had been blown while the site was out of action. Good times, however — I digress.

This CDN team wasn’t the only team with significant control over the production estate who’d amassed an abundance of swagger over the years. The Security team also held a similar position. They could often be found inhabiting the intellectual high ground, sharply sucking in air through their teeth, stroking their chins while gladly standing in the way of progress just to get their opinions heard. Everyone will listen and lap up the three weeks of remediation work to fix a vulnerability in Docker container’s layers. No one wants to be responsible for a security breach - so the process starts over.

The Path to Production

If you’re approaching a transformation project with cultural blockers like the aforementioned teams in the way, you’re in for some very painful conversations. Leaders need to figure out and be comfortable with the easiest path to production. Firstly, get good at shipping your monolith, try releasing more than once a fortnight to understand the pain points. Doing this for a course of at least six months will likely remove a lot more blocker than any radical technology change.

Acknowledging that processes such as raising tickets for changes and CAB meetings all smack of a blame culture where people were reprimanded for making mistakes, thus guardians were put in place to ensure similar errors never be repeated! Over time more and more safeguards are layered on until people are eventually put off from making any risky changes. They can’t complete the change process or they begin to work around it. Either way, things begin to stagnate and any agility is quashed.

By analysing and uncovering flaws in your current delivery process you will be able to head off any potential road blocks when it comes to your new shiny architecture. Without looking at your current delivery pathways and unblocking that first, it doesn’t matter what tech stack or architecture you pick, the risk averse culture will still be there.

Mistakes happen, it’s a fact of life. Make them easier to fix.

The Planning Fallacy Strikes

Non-technical stakeholders tend to get heavily involved during projects like a Monolith to Microservices transformation. After all they’re the ones who are feeling the pain on a daily basis when they can’t change their content on the website or get the reports they need to best serve customers. They can be excused for being blinded by the dazzling lights of the “Brave New World of Microservices” and the possibilities it brings. They start asking you to go big, because Amazon and Netflix have Microservices and we want to be like them!

So, you embark on this journey and prematurely involve people before really knowing what’s truly involved. You show them the new front-end design of a website for example, the excitement builds. But, if you’re breaking this into components or micro front-ends you’re gonna need quite a few parts before you realise your Microservice nirvana, even on a small scale. As an absolute minimum you’ll need:

• Page Header component
• Page Body Component optionally with CMS integration
• Page Footer component
• Page Composition service
• Data Layer or Service to provide page data

Not to mention all the infrastructure, monitoring and logging, automated tests, code repos and build pipelines just to get this code out of the door. To the untrained eye, it may only be one simple web page but the reality is wildly different. Estimation is hard. You’re going to have dependencies coming out of your ears (map and track these). Things need to be co-ordinated, things need to be prioritised and tested before being deployed, integration strategies need to be determined etc. It’s a heck of a lot of work to be swallowed whole. I would hazard a guess you’re looking at least six months for a single page if you’re starting afresh.

Coupled with the under-estimation of the work, stakeholders won’t be interested until you’re demoing full pages — because that’s what they wanted and were promised, right? Your fancy light weight, performant APIs just won’t get their juices flowing in the same way. Therefore the teams involved will keep that goal in their heads “we have to build a full page, we have to build a full page”. A hybrid approach just won’t do. This is Waterfall people. This mindset got us into trouble years ago and yet here we are trying to convince ourselves we are Agile by doing the same thing but different because we’re building Microservices, working in sprints and adopting the Spotify model.

One thing experience has taught me is that you can’t build all of the above in one massive hit. My Continuous Delivery Moment outlines that. You have to find the smallest possible piece and do that. The Lean Startup and the Agile Manifesto talk about “what’s the least amount of work we can do to validate this approach” and “minimising the work not done”. Yet we continually seem to bite off more than we can fit on our plates, let alone chew.

When making the case for a Monolith to Microservices project. The solution is to not promise to build a full page or a myriad of services. Start with one tiny service, a component that serves the copyright notice or some other tiny piece of your estate. Embed that into your existing monolith, watch what happens and learn from it.

Build one service at the start and integrate it into your monolith. Leave it to mature, monitor it. Get good at deploying it. See how it performs and learn from it.

That’s it. One thing using your new architecture and ways of working et al. It will likely do very little, stakeholders will ask “what’s going on, when’s the full page going to be done?”. But, your answer will be “later, we’re learning”. You may find Microservices aren’t the best fit right now and to just stick to larger services. You might discover your technology choices aren’t a good fit. A colleague of mine recently did a week-long spike into a new email marketing solution the boss had mandated should be the new norm for marketing campaigns. His boss came over at the end of the week asking why they weren’t just cracking on and getting it built? Their spike had deemed the new solution totally unworkable. Had they just blindly pressed on as the boss man initially demanded without the upfront preparation they’d have encountered a world of pain.

Fail fast and fail small.

Corporate vs Startup Mentality

When you’re in a large corporate enterprise like this one the tendency is to gravitate towards the “that’s the way we’ve always done it” style of thinking letting those corporate Ephors of Security and CDN teams etc wield their power. Yes, your two weekly release cycle and days of manual testing has got you to where you are today, but it’s not going to keep you there. Consumer trends are changing and the ability to be where your customers are is crucial to your success. So, you need the ability to experiment, to try different things that come with a higher degree of uncertainty. Experiments like rebuilding your applications to be deployed straight to production with only automated testing can still happen, but carry a higher degree of risk. That’s where the corporate mentality kicks in and people start thinking “We’ve never done that before, s**t, if this goes wrong, we’re all going to look daft and we’ll never get another opportunity to do anything like this again”. Fear and caution take over, meetings start to become more frequent and involve more and more people which only serves to delay progress. Before you know it that ninety day timeline you’d set to launch a single component has slipped because you can’t go live without getting sign off from your security teams and the umpteen other people that represent the different areas of “the business”. Six months pass and there’s still people who have no business being there placing demands on the project. The development team are now facing inwards and building a sophisticated monitoring, logging and alerting platform capable of handle millions of transactions, support rotas are being agreed because you want to appear busy and yet… you’ve not launched a single service that powers even a tiny bit of your architecture — because of your culture of fear!

You’re not thinking like a startup and your management are not treating this “incubator project” like a startup.

But that’s exactly what it is!

In a startup, you have a limited time and limited budget. You have to get creative to get stuff done. I once heard an anecdote from Amazon that a new idea was pitched to Jeff Bezos and he simply replied “Great, you’ve got three months and $250,000 or the initiative gets canned”. Brutal, but also fantastic. A similar experiment took place in the gov.uk when Martha Lane-Fox presided over a rebuild of the UK Governments’ digital services. They took one service and digitised it. That team was small, only eight people and again given three months to come up with a working prototype to prove the concept. Look up AlphaGov for more information. These types of constraints force people to think and work differently. People have to have something to show for their time and the autonomy they have been afforded. They need clear, defined goals to work towards — they are ultimately accountable for the success of the project. Teams have to be resourceful and work smarter before the money or time runs out.

With a corporate mentality the purse strings are centrally managed, everyone is still essentially on the payroll and the project seemingly has a bottomless pit for a budget. There is little or no accountability, people are answerable only via status reports and output, not outcomes. There has to be someone or something to base the success of the project on. It cannot be a long-term woolly goal like “We want a platform for all teams in the business to use” or “We want to make it easier for our customers to buy stuff” — Microservices might help you with that, you could just as well refresh the design of your site though, right?

When making small bets like AlphaGov etc., you have to be prepared for that bet to fail, admit defeat and be ruthless enough to pull the plug on the whole thing. Small bets hurt less.

Get specific, measurable and time boxed and be prepared to pull the plug or pivot if it’s not working.

Overcoming Innovation Inertia

In summary, treating any revamp of your architecture like a run-of-the-mill IT project is likely to keep people stuck in doing things the way they’ve always worked. You have to radically change the conditions of daily work. Place constraints on teams, give people autonomy and full control over their destiny. No more centralised teams controlling infrastructure or access to production environments.

Remove as much bureaucracy as possible before you embark on this journey, get comfortable shipping your monolith two or three times a week so people stop fearing deployments and releasing becomes boring. Remove things like CAB meetings and minimise manual testing, it’s 2020 people!

Keep the team size small, this assignment has shown me that in excess of forty people cannot ship one service. Forget Squads, Tribes, Guilds and all that Spotify nonsense — it’s not even real. Get your two pizza team (no more than eight people, five maybe?) and give them a realistic three month target. Something tangible, yet tiny. Work with them to come up with something that provides enough learning to unlock the next three months of your journey. Be realistic too and acknowledge potentially cancelling the whole thing if the approach doesn’t work!

Keeping everything else small too is key. Don’t try and build several parts right away. Think about how pages are made up — what service / component could operate completely independently? Pick that one. Figure out how it’s going to be launched, how will you integrate it alongside your existing products? How will you know it’s working well? Buy some of the fine DevOps monitoring products available like New Relic or AppDynamics before sinking time into building your own. Enterprises can use these too! Many do!

I recently listed to an episode of syntax.fm and they were asked how they would build the next Airbnb or Uber. Not once did they mention AWS, Azure or Google Cloud. Everything was off the shelf. Replace these off the shelf products if/when you outgrow them!

As a C-Level or Project Sponsor you have to give projects like this your full backing. Trust the team you have chosen to do the right thing. Give them freedom to chose their own destiny and cut away the red tape hampering them. They all want the best outcome for the business, no one sets out to screw up and sabotage the bottom line!

Your culture is the largest factor in making these types of project a success, more so than any technology choice. Creating a safe space for small, rapid experimentation is the key along with setting measurable constraints a close second. Get everyone on the project team treating this like a startup, like it was their own company and their own money they were spending and the outcome will be vastly different.

Innovation Inertia — What, how and Why it happens. was originally published in Dev Genius on Medium, where people are continuing the conversation by highlighting and responding to this story.