Top-Down "Agile"

March 23, 2015Giles Bowkett

The practical meaning of “doing Agile development” has changed over the years. To quote two Agile consultants:

From a programmer-centric, Extreme Programming focus in the early days, to a more inclusive approach in the mid-2000s, to a project management and Scrum focus in more recent years.

On the one hand, it’s very healthy for a software development process to acknowledge that engineers aren’t the only experts involved. But I feel that Extreme Programming was healthier than Scrum, and that the shift towards Scrum signals that project managers regained control of a seismic shift that was throwing them out of work.

I’m not alone in this. In what looks like a terrific presentation, Ernie Miller quotes Agile Manifesto co-signer Robert C. Martin:

Who do you think was taking… Certified Scrum Master courses? Was it developers? No… Project managers!

A Scrum Master was a coach. A Scrum Master was… not supposed to be a manager.

…the role was supposed to rotate through the various team members. …And frankly, the idea of the role was to slowly kind of fade away, so that, in the end, you wouldn’t need a Scrum Master.

I don’t think the project managers who took the Certified Scrum Master course liked that particular interpretation.

I know there’s more to it, but I’m tempted to think that this is the core user story for Scrum: As a project manager, I want to pollute the workplace with jargon, so I can draw a paycheck without adding value.

I Call Shenanigans

In many corners of the tech industry today, there’s a very popular paradox: an “Agile process” imposed on development teams by upper management. For instance, a CTO decides “at this company, we’re going to do Scrum.” But that isn’t purely Agile, because it violates the Agile principle that “the best architectures, requirements, and designs emerge from self-organizing teams.” Which allows Agile consultants and advocates to have their cake and eat it too.

Say you’re a Scrum trainer. You take good money from some corporation to impose an “Agile process” on a variety of teams, all reporting to the same CTO. When people discover flaws in this process, you get to say that it was never really Agile, because the teams should have had the option of using Scrum, or not using Scrum, as they choose. And, since it wasn’t really Agile, it wasn’t really Scrum, either.

It’s kind of a shell game, and I hope nobody is brazen enough to practice it outright. But if you certify Scrum trainers as such for a living, and you don’t warn them that their certification becomes invalid if they ever participate in this shell game, then you’re pretty much in the same position. You also risk weakening your own brand so much that it’s impossible to say what is or isn’t Scrum.

I’m not a huge fan of the Scrum guru Ron Jeffries, but I can agree with him that Scrum certification should die in a fire.

Normalized Feedback: Sympathy For The Devil

A lot of people think top-down Agile should die in a fire, too.

My friend Francis Hwang said this to me:

as for process consistency, I think that [stuff] is overrated. If the iOS team wants to use Scrum and the auth server team wants to use Kanban, let 'em try. Tech companies need to always be experimenting, and monocultures are death to that stuff.

I heard similar sentiments on the This Agile Life podcast after I ranted a little about Scrum on my personal blog.

But let’s be fair to the CTOs. Managing engineers is difficult, and you’ve got a ton of projects which all need to be done on time. Somebody tells you there’s a system which allows you to normalize all the feedback you receive from all your development teams, to capture it and quantify it. Does that sound useful?

Of course it does. You grab it immediately, because what you can measure, you can manage.

Scrum produces an artifact like this, called velocity. It wasn’t designed for this purpose, and it doesn’t fit this use case well at all, yet “Agile” tools such as Jira track velocity, automate tracking it, and graph it too, and CTOs use velocity to compare their teams. As with many things in Scrum and Agile, the semantic drift is so advanced that it’s sometimes hard to remember the words involved ever had any meaning at all.

I think this is where you get the weird paradox of an “Agile process” imposed from the top down. I also think that Scrum thrives on this niche, despite being maladapted for it. I think plenty of Scrum gurus would agree that this isn’t what Scrum is actually for, and that corporations which attempt to use Scrum in this way are misusing the process. Yet I also think there are plenty of Scrum trainers making money from this misuse, and looking the other way when it happens.

But let’s say you’re a CTO in this position, and you decide to throw out Scrum. What you wanted was normalized feedback across development teams, and the ability to track their progress in a consistent way. You’ve given up on Scrum providing that for you.

What do you do instead?

Open Allocation

You could do worse than going back to that Agile principle:

The best architectures, requirements, and designs emerge from self-organizing teams.

This was a revolutionary idea. In some places, it had revolutionary consequences. In others, it fizzled out.

Say it’s 2001, the Agile Manifesto and its principles have just been published, and you make your living by organizing development teams. For instance, you’re a project manager. If the industry switches to self-organizing teams, you have to find a new career. In some cases, many project managers found a new title instead - Scrum Master - and things continued as before.

Other projects embraced self-organizing teams. The open source movement provides countless examples of self-organizing teams excelling in volunteer projects. Self-organizing teams are also thriving at companies like Valve, Basecamp, GitHub, and Gore, which creates Gore-Tex synthetic fabrics. Most of these companies are software companies, but Gore is a materials engineering organization which embraced self-organizing teams all the way back in 1967. They called it a “lattice organization” at the time, but the popular term today is open allocation.

This isn’t precisely how we work at Panda Strike, and there are caveats which I’m going to skim over because they lead to a much wider discussion. But open allocation can be much better than Scrum at providing normalized feedback across development teams.

Analyzing Data vs. Mandating Rituals

GitHubbers often quip that the company’s “org chart” is a social graph. What might not be obvious is that they’re being very literal. In an excellent presentation, GitHubber John Britton showed screenshots of an internal GitHub tool which derives any individual GitHubber’s responsibilities by counting the commitments they’ve made, and the nature of those commitments:

Clicking on an area of commitment shows you who’s pledged their time to that area:

If I understand correctly, this tool also does some math based on the actual conversations you have, and who participates in them. If so, the GitHub org chart isn’t just a social graph - it’s a social graph derived from data. In most companies, an org chart is a social fiction, a theoretical construct imposed on top of the subtle, natural complexities of human communication within a group. At GitHub, it’s a data visualization, created by analyzing those complexities.

To my knowledge, GitHub’s the only open allocation company with a toolset like this, but, at over 200 employees, GitHub’s probably a lot bigger than the average open allocation company. Open allocation isn’t a new idea, dating back to 1967 as it does, but it is a new trend, and as such, it’s easier for small companies to explore.

Again, I’m making some assumptions here, and those assumptions could be false. But I built git analysis tools for a Rails rescue project, and they told me almost everything about the structure of the team I was working with. They told me far more than the CTO did. So I know for a fact that you can surface useful organizational information with code and data.

Conclusion

Top-down Agile is a mistake, but it happens for a reason, and I think that when Agile consultants hand-wave this away by saying upper management “just doesn’t get it,” they’re abdicating the responsibility to think seriously about the interests of every stakeholder.

If a CTO wants normalized feedback, that’s a completely legitimate goal. But Scrum velocity is an unreliable source for this data, and comes tightly coupled with a flawed system. Pulling data from modern development tools is better, and GitHub’s done some impressive work in this area. Today, few other open allocation companies are large enough for their CTOs to have the normalized feedback problem which may be mistakenly driving Scrum adoption. But that may change over time, and if it does, the open allocation world is ready.

Relocating to 1992, as Ernie Miller would say, is not necessary.

Even if you're CTO of a large company, you don't have to give up modern development processes like asynchronous communication and distributed teams just to get the normalized feedback you need. If you abandon fictions like Scrum velocity, your feedback can become more accurate. Modern software development generates lots of data as a byproduct, and you can use it to surface an org chart which is much more real than any org chart you could ever impose.