Subordinating the Scrum Team

Agilists intuitively get that it's a bad idea to build an inventory of detailed specs way ahead of when they are going to get turned into software. Why? Because we know that as our product emerges, we are going to want to make changes to those requirements. There is a real risk that a lot of that work will end up being waste. That's a big part of why we document agile requirements as user stories... we want the placeholder now, but we don't need the details until we get closer to building that part of the product. It keeps our options open and our overhead low.

Building an inventory of requirements has a more insidious impact than just opening up the possibility of waste. Sure, we risk specifying stuff that the business might not need, but we also build in assumptions and risk that can't be validated. Everything that we define now, everything that can't be built and tested immediately after specification, represents something that we'll have to validate or mitigate somewhere down the road. Those uncertainties make it really hard to forecast when we are going to get to done. They make it tough to be predictable.

Furthermore, requirements gathering an analysis shouldn't be done in a vacuum. We know that we're going to want some help from our development team to discuss what's possible from a technology perspective. When we pull our team into meetings to discuss these things, we are taking their attention away from the task at hand. Our team is working on the highest priority features, and here we are pulling them away to help forecast lower priority requirements that may or may not ever make it into the product. It's a pretty solid formula for going slower than we have to.

Even though we may have some capacity to do the requirements work... and it would be nice to get ahead of the game a little... it is not something we generally do on agile projects. Managing all those requirements changes, tracking all those assumptions and risks, and absorbing all the productivity impact to the development team is just too great a cost. We KNOW it is going to cause us problems down the road. Any benefits we'd gain from getting ahead of the requirements work would be lost due to the overhead necessary to maintain our requirements inventory. That inventory slows us down and makes us more resistant to change.

We understand... in this kind of a scenario... that the team has to subordinate the requirements gathering process to the overall production cadence of the delivery team.

Now, let's extend this metaphor up a little and talk about our financial services company... and specifically our new hyper-productive Credit Card Processing Scrum team. With regard to the rest of the organization... our Scrum team is able to produce working software much faster than everyone else around them. The question we're dealing with is... SHOULD they produce working software much faster than everyone else around them? In this context, under this set of circumstances, I'm saying no.

I'm suggesting that the Scrum team in our complex product organization becomes very much like the requirements gathering function in our earlier example.

When the Credit Card Processing team builds software features faster than the rest of the organization can consume those features, they risk the possibility of waste and rework. They introduce assumptions and risks that have to be managed and mitigated throughout the life of the project. They will disrupt the other development teams around them and force those teams to multi-task. The other teams will be forced to pay attention to features that they won't be ready to build for another several months. All that code will slow us down and make us resistant to change.

While it is technically possible for the Credit Card Processing team to get a head of the game... and it might make them feel more productive... it might be great for their team morale... ask yourself what impact they are having on the rest of the organization. It might not be as positive as you think. So... for all the same reasons you might ask a PO or a BA to slow down writing requirements to fill the product backlog, you might want to ask your Scrum team to slow down building software until the rest of the organization can catch up.

It's counter-intuitive, but if you start thinking past the single team, it might be exactly what you need to do.

Subscribe to Leading Agile

There Are No Spherical Chickens!

Just in case you are wondering... I'm not suggesting that our Credit Card Processing team shouldn't do Scrum. I'm not suggesting that they shouldn't improve. What I am suggesting, is that if the organization stops with this one team, they won't get the business benefit from Scrum that they expect. All the value created by the Credit Card Processing team will be obscured by the underperformance of everyone else. Furthermore... you shouldn't be surprised when your Scrum team gets downsized, outsourced, or reorganized... just like everyone else who didn't deliver.

If you don't have the ability to influence the rest of the development organization, it probably won't make any difference if the Credit Card Processing team adopts Scrum. Sure, the team will probably enjoy their work more. Maybe if they get really good at delivery, they can jump in and help out some of the other teams. If the culture is open enough, maybe everyone else sees the benefit of Scrum, and they become the catalyst for more widespread adoption. That just hasn't been my experience with this kind of grass roots hopefulness.

Most large companies I have worked with are full of silos, full of politics, and full of protectionism and fear. Accountability structures don't necessarily reward working well across teams. Incentives are geared toward individual performance first, and team performance second... if at all. Organizational or divisional objectives are so far off people can't directly influence them. We are allowed to build empires that only serve to grow the headcount in our particular group, often at the expense of our ability to work well with those around us.

The thing we most often forget when adopting agile, is the impact that agile needs to have on the culture of our organization... and the impact it needs to have on the fundamental structures that reinforce that culture. So... I have to tell you I am a bit cynical about the whole... give agile a try and everyone will be enamored with it's goodness routine. Most people would rather try to be individually successful in the system they know, than risk being unsuccessful trying something risky and new.

In a complex product, all the pieces and parts have to work together to deliver end business value. If you agree with that fundamental premise... and you are willing to agree that team level adoption does not typically spread (unintentionally) out to the rest of the organization... you are left with two options. You can 1) subordinate your Scrum team to the delivery cadence of the rest of the enterprise... or 2) plan to systematically introduce Scrum everywhere else, and learn how to manage the flow of value across the various Scrum delivery teams.

I am clearly in the camp of option 2.

Before we wrap for the day... I want to go back to why I think this topic is so important. A year or so ago, Ken Schwaber famously said, and I paraphrase: '75% of the organizations using Scrum won't get the benefit that they hope from it". If you care about agile, and you believe in the value that it brings to people and teams, and you have seen it work, that is a really depressing statistic. Our community's response seems to be... get more dogmatic about Scrum. If you aren't doing Scrum by-the-book, that is why you are failing.

I'm sure there is some of that going on, I am sure that there are some people bending Scrum to hide their dysfunction. My take is that this complex product idea strikes more the heart of why the underlying dysfunction exists. Vanilla Scrum does not work for these organizations because the value stream is bigger than a single team. Rather than assume that we live in a world of spherical chickens, I'd like us to explore how agile works, why it fails, and most importantly... what are our options for making things better when we can't wave a magic wand and change our cultures and structures overnight.

So... in the context of our financial services organization, and our Credit Card Processing team in particular... I want to explore a bit more why, in the absence of a broader change initiative, you have to subordinate their performance to the cadence of the rest of the enterprise. After that we'll get into how to scale Scrum from the single team, to multiple teams, to projects and portfolios, and then the enterprise. We'll explore blending approaches, how to leverage Lean and Kanban, and how to deal with multi-tiered values streams. We'll talk about how to break down requirements, deal with architectural challenges, how to do planning and context setting, etc.

I might take me a while... be patient.

Subscribe to Leading Agile

How to Build a Large Agile Organization

Okay... consider this scenario. We have a 300 person IT shop responsible for developing control systems that automate large buildings. These systems require front end developers, middleware developers, firmware developers, and hardware engineers. A feature in this system requires us to write code that touches every layer of the overall systems architecture. As it stands right now, the teams are organized around the major subcomponents within the overall solutions framework.

How does a company like this adopt agile? First inclination might be to break up the component teams and create cross-functional feature teams. Each team would have some number of front end developers, middleware developers, firmware developers, and hardware guys... right? The general idea is that any one of these cross functional feature teams would be able to write software for any part of the overall system. Everyone on the team becomes a specializing generalist.

While it's not impossible to cross train someone who specializes in UI development with someone passionate about firmware (or hardware)... but my guess is that most folks aren't up for it. My bet is that most organizations don't have the test coverage to make this kind of experiment safe. I've talked with several organizations that have given this a shot, and the change was so disruptive, they couldn't get anything done. Seriously... nothing done.

I'd like to suggest for a moment, that our goal here isn't really creating cross- functional feature teams; the goal is to build teams that have everything they need to build an increment of working software. I think that there is a difference between the two. To be agile, we need to to build teams that can stay together over time, become high performing over time, and build trust with their customers over time. They need to build working product.

We tend to assume that in order for this to happen, we need to organize teams around end-to-end features. I agree with this approach when we are talking about small teams and small products. This advice breaks down when we talk about larger teams, and larger... more complicated... more technologically diverse product lines. The agile community needs a more stable, more robust scaling metaphor for discussing these kinds of organizations.

Here is how I have started thinking about this over the past few weeks, and I am starting to think that the language works... let's start talking about building teams around those things in our organization that are least likely to change.

If we are a small product company, with a single product... the thing that doesn't change might be a product. If we are a larger product company, with a more complex product offering, I might organize teams around a somewhat static grouping of features. If we are in a really large organization, one with multiple interdependent products, where investments are not level over time... the thing we organize around might be a component or some set of services.

The thing is... it's not always the flow of features into a product that is often the least transient. Sometimes we can't build a team with enough specialists... or specializing generalists... to get the job done. When that happens... and at some scale it will happen... we want to start looking for the stuff that doesn't change. When we find the stuff that doesn't change, that is where we start building agile teams. That gives us our best shot at keeping teams together.

When you take this approach, you have to keep in mind, that there is no longer any one team that is responsible for end-to-end value delivery. Each team delivers 'done-done' work every iteration, but as an overall enterprise, you'll have to develop the capability to manage the flow of value effectively across each of your various agile capability teams. This is where agile stops giving good guidance and we need something else.

That something else is Lean/Kanban.

So think about this... build agile teams around the persistent objects in your enterprise, whatever they happen to be. Use Scrum or XP or Kanban or whatever other agile method floats your boat at the team level. The teams can become hyper-productive and have as much agile goodness as you can get out of them. To scale all this agile goodness, use a Lean/Kanban approach to manage the flow of value across teams.

So, getting back to my initial question... what's the most effective agile strategy for our large control systems company?

Organize agile teams around the front end, the middleware, the firmware, and the hardware. Create an enterprise backlog of value features that get broken down into user stories and distributed across teams. Subordinate each team's backlog to the enterprise backlog, and create a pull system, where new features only get started once the teams have capacity to work on them. Set work in process limits for each team to reduce the amount of work we can get started before we roll back up into a value feature. Invest in constrained capabilities to improve the overall flow of value across the system.

Make sense? Any questions?

Subscribe to Leading Agile