Last year sometime, I had the pleasure of hearing Jeff Sutherland speak at the Agile Atlanta group here in town. One of the things that Jeff always brings up in his talks, is that Scrum creates hyper-productive teams. I asked him how he defined hyper-productivity, and how he measured it. He told me, but rather than explain my recollection of what he said, I want to reference how he explains it in print.
According to his paper Shock Therapy: A Bootstrap for Hyper-Productive Scrum:
"We define Hyper-Productivity here at 400% higher velocity than average waterfall team velocity with correspondingly higher quality. The best Scrum teams in the world average 75% gains over the velocity of waterfall teams with much higher quality, customer satisfaction, and developer experience."
Pretty impressive, but I am curious what the improvement is measured against. Going deeper into the Shock Therapy paper, the results from their MySpace experience say that:
"The baseline velocity (100%) is established for a team during the first Sprint. The Product Owner presents the prioritized Product Backlog in the Sprint Planning meeting. This is estimated using Planning Poker and story points. The team selects what can be accomplished during the first Sprint and the Product Owner determines exactly what is "Done" at the end of the Sprint. The number of points completed is the baseline velocity."
When I first read that, I was like "what am I missing here". Hyper-productivity is defined as improvement over the first iteration of a brand new Scrum team. That didn't strike me as a very fair metric, you'd think *most* brand new teams would get dramatically better after a few weeks of working together, using Scrum or not.
But here is their key assertion that supports the measurement:
"At MySpace, the baseline velocity is often significantly higher than their previously chaotic implementation of waterfall, so the baseline is conservative"
I don't know about you guys, but I'm kind of bothered by the assumptions, and the numbers that support them. Can we always assume the first sprint of a Scrum team is better than the same team doing Waterfall? I'm not sure. Are all Waterfall projects chaotic? I can tell you first hand they aren't. It might also be interesting to see the team stabilize first, and then see how much better they'd get by systematically removing impediments.
These numbers have been nagging me ever since I read this paper and first heard Jeff explain them. Is it only me? What am I missing here?
Subscribe to Leading Agile
I was likewise at that event and inquired about what is the criteria for calling a team "high performing". My recollection is that Jeff told me that a high performing team had to produce more than a given number of function points per man-month, and that number varied depending on the type of code that was being developed. (I trust my recollection on this because it involved a couple emails to Jeff, a study of "Applied Software Measurement: Global Analysis of Productivity and Quality" by Capers Jones, a weighty tome to be sure, and the writing of a couple lengthy white papers detailing the productivity of 3 development teams using different processes. I'm looking for that email and those white papers on a thumb drive -- not having quick success with that.
ReplyDeleteHere's that email from Jeff (11/07/2008) that I mentioned in my previous reply. For context, below his email is my very long question to Jeff. The net is that his definition of hyper-productive teams related to a higher than average number of FPs/man-month.
ReplyDelete-------------------------------
Andrew,
Thanks for the note.
To compare to Xebia numbers, you need to remove all test code from
your numbers and fully burden the cost. It must include all testers,
all administrative staff, any managers or project leaders as FTEs. If
half you code is test code you will go from 20 to 10. If you include
testers and other staff it will drive you well below 10. Xebia's cost
is fully burdened with anyone associated in any way with the project.
Regards,
Jeff
On Thu, Nov 6, 2008 at 5:35 PM, Andrew Fuqua wrote:
> Hi Jeff.
>
> I enjoyed your presentation at Turner in Atlanta on the 27th. (I was the guy
> who gave the book away before your talk.)
>
> I've used your approach to compute the function points per developer per
> month for two of my teams. The 1st team is an XP team. The second is not as
> extreme. I've come up with 20 FP/man-month (558,633 java sloc, 537
> man-months) and 22 FP/man-month (107,647 java sloc, 90 man-months). During
> your presentation you said that 15 would be a hyper productive team and that
> 20 was the highest ever recorded and that 12.5 is the industry average. My
> team's quality is very high (102 defects = 0.183/kloc, and 11 defects =
> 0.102/kloc), but we thought we were slow to develop so I was very surprised
> by this FP/man-month being 20+.
>
> I can think of a couple possible explanations. First, these source line of
> code counts include JUnit based unit tests, functional tests, and acceptance
> tests. We have about equal amounts of test code and production code. I used
> the 53 loc/fp for java ratio that you mentioned in your presentation. Should
> I remove the test code from the sloc count?
>
> Second, my man-month figure only counts the programmers. At the peak when
> these projects were running at the same time, we had 15 programmers. We also
> had 3 professional testers. They would do some by-hand testing during the
> iteration after each story was demo'd. They would do more manual testing
> after each 2 week iteration. They also automated a regression test suite
> using Rational Functional Tester, but those lines of code was not included
> in my sloc count above. Should I have included these additional tester
> man-months in my calculation?
Andrew, not sure your point.
ReplyDeleteJeff clearly defines 'hyper-productivity' in his paper Shock Therapy. It's a pretty easy Google search if you are interested. The point is not cost/line of code or function point. The point is that Scrum teams measure their performance baseline against, what I consider possibly, an invalid baseline, and against an invalid set of assumptions.
Take a look at Sutherland's paper and let me know what you think.
Well, you can argue about the data. If you don't like the MySpace data there is the Xebia or Systematic data. I had Capers Jones teams measure every team at IDX (now GE Healthcare) for a couple of years on actual function points delivered at every release of every product (dozens of products). While most teams only doubled velocity the hyperproductive teams were all 10-20 times more function points that the teams in Capers own database.
ReplyDeleteThe point here is to measure the velocity of your teams. In a group of 30 people in training today only 3 of them knew the velocity of their team. Are you in the 10% or the 90% Scrumbutt teams. If you are in the 10% your job is to multiple their baseline velocity by 500% something that was possible on every team at MySpace when the ScrumMaster was Scott Downey.
The point is not whether you like my data. Gather your own data.
Jeff Sutherland