The problem with Scrum

You keep using that word. I do not think it means what you think it means – Inigo Montoya, The Princess Bride

I’ve been pondering a problem with Scrum and agile in general for a while now.  Why do so many people use these named processes that encapsulate agile ideas as swear words?

Others seem to have been pondering this too with Ebenezer Ikonne writing his thoughts at and Ron Jefferies at  These are worth reading.

If you’re doing Scrum or XP or being agile or doing some other process / philosophy that focuses on quickly learning what to do and executing on it and then checking to see what to change and readjusting continuously – it can have awesome results. So why do so many people use these named processes that encapsulate these ideas as a swear words?

A Problem

Scrum has been a highly successful marketing success. Everyone is doing it. That is part of the problem. It is accessible as it is “just some practices”. But it isn’t just some practices. And huge numbers of adopters are getting that wrong.

This has resulted in Scrum and agile being about control to many that have experienced it.

I have two friends whose formative experience of Scrum was an implementation that literally created waterfall documents in sprints for several months. To them – this is what Scrum was about. Their opinions have changed since, but this was their formative experience of Scrum. If they hadn’t since had good experiences – or good conversations about why this isn’t what it is about – then that would be how Scrum would be for them.

As Sarah Mei puts it:

how can we reclaim autonomy and control over our work in an industry where “agile” is most often a tool of top-down corporate control? – Sarah Mei [1]

It is hard to refute someone’s personal experience. This is what it means to them.

Words have power. And when you let them mean something to people – that is what they start to mean. As Ron Jeffries puts it:

 Agile is what people who call themselves Agile do.

It seems to me often that every good idea gets watered down and ultimately perverted. “Agile” is no exception. There are many bad things done in the name of “Agile”, and many more quite inadequate things. – Ron Jeffries [2]

It turns out the personal experience of people is a powerful factor in determining what people think. It doesn’t matter if what they experienced was completely not agile or not Scrum to those that understand and experience agile or Scrum in a positive way. Their experience was done in the name of Scrum / agile – and in many ways that’s just how it is.

You’re Doing It Wrong

There is a trite saying that I heard early in my Scrum days – “Scrum does not fail. You fail at Scrum.” It is so easy to say you’re doing it wrong. You’re not being agile.

Software development is hard. And as can be seen by the rise of Safe, DAD, Mom, Pop, LeSS – people who are unable to bring the existing lightweight – but easy to misinterpret – processes to bear are reaching for other silver bullets to help.

 “@twykowski: There are more companies trying to scale Scrum than to do Scrum right.” Yes, isn’t it curious? – Peter Hundermark [3]

An early idea that I read about was the metaphor of Scrum an abstract class. All implementers of Scrum will need to do it within their context and by definition that will be slightly different. So if you’re doing it differently to someone else, this is by design. But the abstract ideas and principles and practices should manifest in the same way.

Could you do it better?

Perhaps.   Has everyone read the Scrum Guide / Agile Manifesto?  Does everyone have an equal say – particularly in debating whether something that is being done is true to the principles and values from there?  Strong debate on how to do the work grounded in achieving the same principles and aligned with the same values will make the team stronger.  And they will then buy into how the work is done when they are empowered to do it well and improve.

Does it actually matter?

Maybe? Maybe not?

I used to be a Scrum zealot. It worked well for the implementation that I was doing. We were highly successful in the things we did and I learnt a lot.

I used to think that Agile was a thing, and that everyone should do it. I guess I kind of still do think that, for values of “Agile”. But the term is so watered down that it scarcely matters. – Ron Jeffries [4]

Then I joined a company that was doing Scrum and it was a mess.  The spell was broken.

Then I joined a company that didn’t do Scrum and we used bits and pieces and things were better than when we started. This was a Good Thing.

Mostly, what matters is operating your work and your life so as to maximize whatever you consider to be happiness. – Ron Jeffries [4]

Stop worrying

I’ve mostly stopped worrying about the name of the process that I use. I’ve moved back to values and principles.  Most of the bad things done in the name of Scrum and agile are due to lack of understanding of these values and principles.

To me, it is important to know why we’re doing practices and where they come from. So rather than Scrum, Agile, XP – I’m more interested in Feedback and Simplicity, Humanity and Reflection. I still value the structure of Scrum / XP – with iterations, POs, SMs, retrospectives – but I value more understanding from the values and principles that they are derived from.  And I’m happy to let those specific practices go, if the values and principles that underly them are still important and optimised for.

Positive Sharing

Lets positively share more of what works well for you – in your context – and WHY it works.

Lets stop spending time trying to negatively pull down what you think others are saying you should do.

As Ron Jeffries posits

More and more, I try just to talk about the specific things I do, without proper names. Maybe that’s best. – Ron Jeffries [2]

This is a good thought.

It might also be an idea to not give your thing a name 😉








The same things don’t always work

Over my years of getting to know Scrum and the agile way of working, I have experimented with a lot of things.  I have found things that didn’t work and I have found things that did.  I’ve kept the things that did work and tweaked them as needed.  They were good tools for me and informed my thinking around how I succeeded using Scrum.

Then I moved jobs.  I took my toolset with me.  And I tried to use my same logic and thinking.  And people heard my words and too often for my liking interpreted them to mean something completely different.  It was very educational and taught me very strongly that The Same Things Do Not Always Work.

Context is King.
If you’ve built up context around certain ways of working then people know how you got there as they were there with you.  They understand.  They emote.  And when you bring those ideas fully formed into another organisation that have strangely not lived in your head for the last couple of years, they don’t necessarily immediately understand or emote.  And this isn’t their fault…

My failure has been in not understanding that my tool set held the tools that I had decided upon by applying the principles that I understood and in order to use the same tools at a new organisation I had to first back away a little and bring out the principles again to see if those same tools would still uphold those principles in this new organisation.

A simple example: Ship when you’re ready
For more than a year I had been working with a team delivering software which was officially shipped on days that weren’t the sprint boundary.  The team were fine with this.  We always aimed to finish before the sprint that we shipped in.  If we could plan it on the boundary we would, but sometimes it didn’t work out that way – and it didn’t matter.  We were completely fine shipping when we were ready – instead of waiting for an arbitrary date boundary for the sprint end.  Everyone was good.  It worked well.  It felt obvious.

Obviously if there was a large amount of work to do and you asked the team to commit, they can’t commit to earlier than a sprint length.  But – if we’re done, we’ll ship – why wait?

And then it didn’t work…
Fast forward to a new organisation.  We have some work to complete.  We have a ship date.  So we discuss and using the previous pattern from my tool set I suggest – if we’re ready we’ll ship.  If we’re not, we won’t.  Somehow this was interpreted down the lines as: we are going to change the sprint length to 1 week and people will deliver by the deadline or else.

What was the simple “if we’re ready we do it, if we’re not, we don’t” turned into an angsty changing the sprint cadence rush to complete.  But that was the organisation’s interpretation of what was, for me, a clear and obvious way of working.

Which made me think
When going into a new organisation – go back to basics.  Say no to all the broken rules – until you know which ones you can safely break without someone abusing the situation.

Another example: Velocity
For several years I had been using velocity and planning stories in a reasonably reliable fashion. The teams I had worked with weren’t highly passionate about velocity but were focused on the work at hand and usually knew what the next sprint or two held and were willing to push their capacity to try and achieve more points in a sustainable way.  The combination of measurement (to aid planning – and replanning every sprint) with knowing what you’re doing for the short term helped ensure that the team was both productive and reasonably predictable.  This was great for building trust with stakeholders who had legitimate concerns about delivery in the past and it also enabled us to go a faster.

And then it felt pointless…
Fast forward to a new organisation. No sizing. No sprints. So I enthusiastically said we should try a little Scrum.  So now we do a little Scrum.  But we don’t use the velocity or plan beyond the current sprint.  And it works.  And no one actually is worried.  And the stakeholders are okay with everything.  And everything is roses.  So why measure?  And why plan?  When you can be agile and make it up each sprint – because the work is still known in a reasonable fashion.  And we’re as successful as is required of us.

The tool set that I brought with me didn’t result in the changes that I anticipated and in fact possible adds little value right now. I suspect many reasons for that.

Which made me think
Scrum isn’t just a framework.  Context remains King.  You can’t just walk in and apply your learning from another context to the new one without understanding the context and working with the people who are in it.  That doesn’t mean you can’t ask questions and make suggestions – but do just that – rather than judging too early.   Agile is about principles – these lead you to the learning and the tool set.  Always go back to the principles and the spirit and validate – particularly when approaching a new team or new organisation with your existing experience.  Unless, of course, you have the remit to cause revolutionary change.  In which case, go wild!

And there is a point
My failures over the last year have fuelled much introspection and learning.  I’ve opened myself up to question myself around my understanding of Scrum and agile.  And I’ve seen very clearly how no one size fits all.  I have found this a powerful learning experience.  Seeing what you know does work not working any more deepens ones understanding of what it is that you’re really doing.  I’m thankful for these new experiences that have allowed me to grow a deeper understanding of what has worked by understanding why it hasn’t worked as well.

I now hope I keep remembering to not apply my tool set too soon in the future in the hopes that I’ll more effectively apply it with a deeper understanding of the actual context.  Or perhaps I’ll find a more universal tool set to apply.

I want it by October

It is often the case that a client or a boss (or his boss and so on) comes to you and says “I would like have the Foo module by October”.  There is generally very little technical understanding of what Foo may be.  Often there is very little business understanding of the scope of what Foo might be as well.  So what do you say?

Step 1: We can do something

There are a myriad of options.  Yes. No. Maybe.  What is Foo? I’ll need a full spec before I can commit to anything.  I’ve said them all.  I’ve worked with them all.  None of them make anyone completely happy.  In general – if the date is in the future – I now go with “Yes, we will be able to do something, I’m just not sure yet what exactly that will be.”  So that isn’t no, but it isn’t yes either.

The problem is that no one may know what Foo is yet.  So how can you with provide an answer to this question with any amount of value?

So we’ll be agile about it – and if we prioritise well – we’ll get the right things done.  That is what our track record says and we’ll continue to do it.

Aside: What if “something” isn’t enough?

But what if there isn’t enough of a Minimal Marketable Product by the date?  Do we work harder?  Hire more people? Fire some people?  “We don’t know what will happen when you add 2 people to the team” isn’t a particularly useful answer but it is the correct one.  The business side of the equation needs to understand that as well.  Adding more people is well known not to be a short term solution – even if you can find people good enough to add.  We need to work to manage demand and optimise the flow of delivery.  Everything can’t be delivered in October – unless we only do one release a year – and then it becomes “which year?” 😉

Step 2: Define what it is

In the past I have worked starting with the one liner of “I would like Foo completed by October”.  From this often an RFC is initially written up by my PO.  I’ve worked with the PO to generate a list of stories.  Most recently those stories  arrived with no input from me.  Based on that list of stories we can now start discussing both what is missing and the realities of what it might mean.

Step 3: Size it

At a point where we believe the backlog is defined enough (which may be quite early) we then do some sizing.

I have in the past done a 4 hour estimation session – 2 to 3 minutes per story – with the whole team.  It was unpleasant for all concerned.  Since then I’ve done affinity / magic estimation and it works like a bomb.  In the most recent session I tried to compare my estimates made before the team did the sizing to that which the team came up with.  They were pretty close – and as the team progressed and resized items as they better understood them they came far closer to my  original sizing.  But I haven’t yet used my sizing for initially planning yet 😉

Step 4: Plan and forecast

So now we have a backlog and sizes have been applied by the team to the items.  This is very rough!  But it is good enough to manage within a couple of sprints – assuming that there isn’t a lot of unexpected work.

From the size we can take average, highest, lowest, last velocity achieved for the team and provide some forecasting as to how long it will take to deliver the known work.  Generally I’ve been working in 3-4 sprints worth of work and this has been relatively reliable within a sprint or so.  So you can agree on a reasonable date based on past experience of the real velocity and padding for stabilisation and backlog growth where you needed.  Everything changes, so expect a little change – but sometimes the client / boss would like you to not – in which case show the change visibly in order to show why you should have planned for it in the first place.

Step 5: Manage expectations transparently

From here on out I report on progress with complete transparency.  I report on change – up and down.  I report on actual work completed.  I report on the amount of remaining work and how that may have changed.  I work on how the last sprint has changed the average, highest, lowest and last velocity and how that influences the potential end date spectrum.  I report on this Every Single Sprint.

Practicalities: Sometimes you need a date, but try not to

My current process that I’ve been using for the last two years is possibly not very agile.  But when a date is asked for and that is what the client / boss is managing to then you sometimes need to provide one.  And then you possibly need to argue what to change in order to stop managing to the date.  I’ve (maybe) just won that argument – but we’ll need to see how that project continues.

Step 6: Replan every sprint – the numbers don’t lie

The beauty of this is that the numbers don’t lie.  It is simple as that.  We can be hopeful and optimistic and plan on the best velocity – but if we’re consistently not achieving the best velocity that will continue to self-correct every single sprint and the end date will move accordingly.  If we don’t discuss the information that is radiating out of the figures then we have our heads in the sand and we’ll fail.  Managing the expectations continuously and early will ensure more probability of success.

Step 7: Be Successful at what you know

Success is a relative term.  Managing expectations and ensuring that we’re successful with delivering what we know ensures that we’re successful.  Managing the change so that we can be as agile as the client / boss will allow us to be ensures that we’re successful.  We can’t do magic.  But we can manage the reality or what is actually happening Every Single Sprint.  And everyone can embrace change and be successful.  Or fight it – and hope to be successful despite the need for change – it is possible.

Are there other ways to plan?

Of course there are. There are vast options on the planning tree.  When it comes planning without involving the team there are many options.  At my current organisation there is a magic spread sheet that turns input into days based on numbers of developers to understand a guess at the size of the work.  The bottom line is that no matter what the original plan is on day 1, there is a good possibility that it will change at the end of the first sprint based on the feedback cycle from that sprint.  Embrace that information and deal with it rather than hide it and pretend we’ll do better next sprint.

Scrum provides an amazing opportunity it empirically measure progress and hence reset the planning at the end of each sprint.  Ignore it at the risk of failure.

But is it agile?

I’m not convinced that what I’ve done in the past has been very agile.  The PO and myself have done a lot of preparation work in order to ensure that the backlog is defined enough and able to be sized.  All the stories are small enough.  We haven’t been looking at epics.  This can take a reasonable amount of effort and be wasteful.  But it avoids including the full team until a full understanding of what is wanted is obtained.

Recently we’ve had the scenario where the RFC was understood but wasn’t “signed off” so we couldn’t start working on it – despite knowing that there were solid chunks of work that were very unlikely to changed.  That is waste and not very agile at all.  That is one of the things I’ve been frustrated about and have wanted to change.

Recently the PO has started working more closely with one or more team members to define the backlog better.  It will be interesting to see how that pans out with a more collaborative generation of the backlog.

I’ve also pushed very hard for us to be more agile about the requirements definition.  Let’s work it out as we get there – let’s rather embrace a little more ambiguity in order to avoid the in depth up front work for the PO and stop fighting the ambiguity as the RFC isn’t perfect.  However we do work in a remote scenario so this might turn out to not be such a good idea if the PO isn’t always available.

Equivalently with ambiguity comes less predictability on that Big Date that we may be managed to.  However I believe we have an agreement that the date isn’t usually an important feature so we should stop managing to that.  And get some of that continuous delivery going with more releases more often.

Aside: Are dates reasonable?

Is the original question reasonable?  Of course it is.  A CEO wants to make a deal and needs to know if something is possible.  A managing director needs to move forward with a product and needs to know if a certain client can be supported in a certain timeframe.  A managing director needs to be able to calculate the ROI for two different streams of work and hence needs to know the relative sizes of the work.  These really aren’t unreasonable business requirements.  But they all need to be weighed up and balanced to ensure that in the timeframe required we can deliver the highest priority items for the business.  That should always be driven by the business.  But the business should still ask us before committing.  And we still can’t do more work than we’re capable of.

Step 8: To close…

This is how I’ve been planning.  Continuously feeding back on the progress.  Get the client / boss to understand what that means.   And fight the fight when it comes to a head that the “date” isn’t in the planning – or hit the date because the velocity calculations make the delivery vastly more predictable than other methods so you can actually get the date right – or even deliver early.  Rather do that than deliver late.

I’ve had really great success delivering predictable releases with the above.  There is no magic.  Just a little maths, a lot of measuring and strong will power to help the client understand.  Hopefully this can help others to be more successful as well.

Lightning Talk retro – Fixed Price Scrum

I did a 10 minute lightning talk at the Cape Town 2011 Scrum Gathering on Fixed Price Scrum.  It went down reasonably well – but if I did it again, I’d do it a bit differently.

The point of the presentation was to point out it was possible and effective to use Scrum to manage a project that traditionally is waterfall and would generally be a rather fast march to and past the end date of the project while blindly hoping you’d still make it.  Scrum allowed us to know early that the date wasn’t possible – which let us manage the client early and hence to come to a compromise and tweak the iron triange of fixed date, time and scope so that we could actually tweak the scope and deliver to the date – and then complete the rest of the scope.  I believe it has been the most successful fixed price project that we have embarked on by far.  So it was successful.  The client did get what they wanted.  (Luckily they did sort of know what they wanted as it was replacing an existing system.)

After the talk I realised that I had given some hope to some people that you could succeed in doing Scrum in their environment.  That was the point.

However I did start to feel guilty afterwards.  I did a talk on waterfall with Scrum helping out.  Yes, the Scrum was pull (mostly) and we inspected and adapted, we tested continously and we did reviews.  These were awesome and effective.  But we took a BRS, chopped it up and turned it on its side and called it a backlog and off we went.  It wasn’t agile.  I did a very non-agile talk at what was in theory an Agile conference…

The project wasn’t Agile in the sense of removing waste or in delivering value at all times.

The BRS generation wasn’t agile.  It was waste.  But the client needed it as a proxy for trust – and a requirement for public tendering.

The review process wasn’t agile.  It was prescriptive – did we meet the BRS.  Value was limited to the BRS and the scope management around that.

But the project was successful.  The delivery was of good quality.  The client is now a long term fixed team client.  We built up trust and a working relationship.

So does it really matter that it wasn’t agile?

If I did it again – I’d point out that we were doing Wagile.  And that there are still perks to using Scrum – even in waterfall.  And doing Scrum in any environment allows you to start learning and growing your understanding so when the opportunity comes to not being in the prescriptive contracting relationship, then you are already primed to embrace more agility as you may have started to figure out what that actually means.  And maybe while they are there – they can work out how to be as agile as they can – around the fixed constraint of the contract – and eliminate waste where they responsibly can.

Will fixed price projects go away in the future?  Unlikely.  Should any company contract in a fixed price arrangment?  Not if they can afford to do otherwise.  But would it be better to go bust and be agile or better to make a profit and be wagile?

Hopefully the next time I present it will be on something more relevantly agile and I’ll feel less exposed to agile frowns.  Or at least remember to point out that I know it is waterfall in agile clothing as I go along.