Retrospective Analytics – Sizing and Bugs

A Problem with Size?

In How do you really know enough I mentioned that one of my teams was gathering data around sizing.  This came out of a retrospective 4 sprints back where the team were a bit frustrated about their sizing.  They decided to track the information to determine if there was anything to learn.  In our last retrospective we looked at this data.

Problem: The team felt that their relative sizing wasn’t accurate enough and that they wanted to take a look at it.

Background: We had just gone through a process of doing a bunch of documentation for a web service.  I suspect this was a key reason for thinking this was an issue along with one or two other outliers.  Basically the team were doing activities they weren’t used to doing and were under estimating the impact of generating the Sandcastle documentation – both from a new technology that they were adopting as well as the back and forth communication that seems to always occur when getting some document to be good enough.  Fundamentally the DoD for the documentation wasn’t as clear as for coding a story and required a lot more communication overhead.

Solution: Gather data and analyse it.  One team member signed up to gather the data and when the team felt they had enough we would then analyse the data.

Outcomes: We analysed the data in our last retrospective.  The team looked at the data and determined that across all the sizing for the last 4 sprints about 70% was spot on and a small percentage was either 1 size down or 1 size up.  There were no clear outliers outside of 1 size difference.  This was interesting and the team now could determine that actually they were doing a pretty good job.

So going back the original problem – the team determined that it was more the outliers of new technologies or new types of work – such as adopting a tool such as Sandcastle or doing work that you’re less used to sizing – such as writing documentation – that was more the root cause of the sizing frustration.  This was also something that we didn’t do much of in the sprints that we were measuring.

End result:  We’re pretty good at sizing and really we shouldn’t be sweating it.  But we need to be aware of underestimating the effort needed for work that we aren’t familiar with.   And only really if we also need to provide date based planning on the velocity/commitment that we hope to acheive.

Buggy or Not Buggy?

In this same retrospective we did a bug analysis to see if we could learn something around our bug count in terms of improvements.  We’ve done this a twice already in the last year.  The first two were useful at generating ideas.  When we did the first one we were in a bit of a state and were trying to understand why exactly.  I was interested to see where this one would go.  Again this was driven by the team wanting to take a look again.

Analysis: We do all our internal bugs as red stickies on the board.  This makes it visible and we don’t waste time in Jira dealing with bugs but rather with the visible board that the entire development team can see.  In order to do a bug analysis we gather the issues from the last X sprints, create several buckets that we think the bugs could fall into (technical debt, UI, functionality, DB, process, etc) and add / remove buckets if needed as the process continues.

We then do the equivalent of affinity estimation / magic estimation for bugs.  Each team member gets a selection of bugs and they place them in the relevant buckets – or create a new one.  We don’t do the silence thing seems as some of the bugs are a little opaque in their wording (or hand writing) so sometimes you need to ask for reminders about the issues.  After the first pass everyone sees where everyone else put the bugs to determine if they agree – and so things move around.

Once equilibrium has been found we then break the buckets into critical and non-critical and generate insights as to what the data is telling us.

Outcomes: As it turns out we have quite a lot of issues that are UI issues – but many of them are changes but we still mark them as bugs.  We resolved to not do that moving forward so that this would be more obvious as to real bugs vs. change that we’re happy to do.

We also determined that we weren’t learning anything new.  One of the outcomes of a previous bug analysis was to do more developer testing to ensure the quality of the story.  This was brought up again and the team resolved to do more of it again – but that is a less quantifiable action – test harder / more.  Sure we’ll try.

End result: We determine that the number of bugs coming up in sprint was an acceptable level.  If none were being found, that might be worrying in terms of what did it mean.  The team were dealing with them and things weren’t being left undone.

We also determined that the more interesting thing would be to look at a bug analysis of bugs that have made it into the wild after our next release.  Those are bugs we did not find – instead of did – and it might be far more interesting to analyse those holes.  The things we have to look forward to 🙂

What? No Actions?

So the retrospective ended with the team feeling good that actually the world was pretty good.  This was great.

The devs then sat down after the retro and generated actions to attempt to get our unit testing in the current module working for the next sprint as that has been an ongoing retro action that isn’t getting traction due to some limitations that take time to work around and the team aren’t seeming to get these solved with baby steps – but more on that later.

So yes – still actions in this self organising team 🙂

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s