Welcome to the newest installment of My Agile Team, my team's ongoing series of misadventures in trying to get better at Agile development. This time around, we're still playing Whack-A-Mole on the list of bugs we've encountered since Go Live. We stomp one out, another pops up.
What I'm going to discuss this time is our approach to trying to fix these critical bugs while maintaining at least a semblance of our Agile nature. It's hard to do a planning meeting and decide on what to work on when you've got new things popping up and old things dragging on. Read on for more on how we're planning in the midst of firefighting production bugs.
Bugs incoming!
We did a lot of user testing. I mean a lot. The app is a billing system for an insurance company and 4-6 members of the billing team worked overtime for months to go over the system with as fine of a toothed comb as they could muster. And still, when we went live we started having problems we've never seen before. Billing in insurance is complicated and it's not until you have to live with the decisions you made in development interacting with real people and real situations that you start seeing some stuff, there's just no way around it.
Planning with fire
The question is, how do we plan for this? We're trying to keep on the Scrum path and plan but as I mentioned last time, our "Top Priority" / Critical list is now 10 items. As soon as we get rid of one of them, another pops up and since it has to do with people's money our Product Owner labels them critical. What we've been doing is using the critical list as our Product Backlog. We're still not in the habit of estimating the size of the items the way we'd like to, but with most of these things we don't know what they are until we get into them anyway. So we basically take the list of critical bugs and split them up based on who originally worked in that area and how much bandwidth people have. Giving people items in the area they worked on is really just to help speed up the process since hopefully the person with the knowledge already in their head will get a faster jump on things. Since these are by definition weird bugs though (if they weren't weird we'd have seen them before) sometimes prior knowledge doesn't help.
The Red Queen's Race
The problem of course comes in between planning meetings when new critical things come up. Since to our Product Owner everything that's critical needs to be fixed now, there's not much we can do to re-prioritize. Some things are obvious but most stuff just goes on the list to fix as fast as we can. This is what I mean by Whack-A-Mole, we knock them down but there's always new ones in their place.
The Finish Line?
At this point we're just hoping the point where we stop finding new critical things comes soon. Our manager took our extremely rough time estimates and told the company that meant we would be done with everything in 2 months so we're all shooting for that goal. Any comments from me about this will have to come over beer. :)
Agile Much?
At this point I realize that this series hasn't exactly been All Agile, All The Time as much as I wish it were. I'm doing my best to keep our team and these articles informative and focused on Agile but it's been difficult. I'm trying to make sure reading about our progress (or lack thereof) isn't as hard as actually living it but I also think it's valuable to see somebody else's story, even when it's a lot of running in place. I really, really hope that we make progress in the coming weeks and I have better things to write about next time. Thank you for reading and if you've kept up with this series, thank you even more.
If you have questions for me or suggestions for something you'd like to see me write about our team, I'd love to hear it in the comments below. Thanks again.
Comments
Interesting. I have a couple
March 16, 2009 by Anonymous (not verified), 2 years 46 weeks ago
Comment id: 2378
Interesting. I have a couple of comments based on the exact same experience.
First, don't get trapped in the "two backlogs" situation, where you're maintaining/prioritizing a defect list along with a backlog of desired business functionality. Use one list. Define the capacity for your sprint. Work with the product owner to fill that bucket of time with work. If the product owner decides to fill your sprint with defects, they're choosing quality over functionality. If they fill it with new features, they're choosing functionality over quality. Simple as that.
You should always estimate the work you're bringing into a sprint. Without estimates you have no way to guage the work against your capacity. More importantly though is that you cannot commit to a set of work for a sprint unless you have a reasonable expectation that it can be completed.
Plan and plan often. Reprioritize your sprint backlog if necessary. There's nothing to say you can't meet as a team and discuss bringing higher priority work into the sprint. It's not ideal and you're modifying what you committed to at the beginning of the sprint, but you shouldn't be afrraid to either. If the PO, Management, or the business is pushing you to bring more work into your sprint, welcome the change -- just ask them what they're willing to give up for it. If it's a 20 hour task you're bring in (here's why you need estimates) then tell them to take 20 hours worth of existing work out of the sprint.
I'm reading this to mean that your PO is prioritizing all defects as "critical" just so they get fixed? Is that correct? If so, the prioritization of defects should be a collaborative effort involving the team and the product owner. All sides should be involved to prevent unreasonable things like this form happening.
If the PO still insists on everything being the same priotity then tell him that for sprint planning, instead of having a meeting you're just going to let the janitor pick the work for the sprint. If it's all the same priority, it shouldn't matter right? Hopefully they get the idea.
This comment is worrisome. Why is a manager talking to the business? The team should be interacting with the product owner, through the ScrumMaster if necessary, but ideally directly. Go back and tell the business that your team is producing all the artficts necessary for them to gauge your progress and when you will be done. Tell them to look at the burndown charts and your velocity. That's where it should come from, not a manager.
I hope you're building a test
March 16, 2009 by PJ (not verified), 2 years 46 weeks ago
Comment id: 2379
I hope you're building a test to trigger each bug before you work on it (so you can know it's fixed). You should if only to prevent regressions in the future.
Is your PO really into scrum?
March 16, 2009 by Henrik Johansson (not verified), 2 years 46 weeks ago
Comment id: 2380
Firstly, I agree with previous poster about the importance of having a prioritized backlog (no duplicate priorities) - otherwise your team can't commit.
I would also like to comment regarding the quality aspect. In our team we have removed the PO and management from dealing with quality. As long as we show good progress we are trusted with judging the quality aspect of the software within the team. After all, it's we who are the software engineers. This means that we reduce the parameters the PO has to play with; from three (time, scope, quality) to time and scope. And time is pretty much defined as "in a sustainable pace" according to scrum.
We started the no-negotiate policy regarding quality just after last summer. We then had to work two sprints only focused on bugs. The goal was and is zero bugs at all times. - No backlog items were finished. We now always fix bugs continously as they appear. By doing this our PO only has to worry about prioritizing and coming up with backlog items, not quality or bugs/tradeoffs.
Working in this way promotes quality, makes it non-negotiable and already estimated for when sprint-planning takes place. (The velocity is the velocity, and as far as we know - there will be as many bugs as last sprint.)
Henrik Johansson
One Suggestion
March 16, 2009 by Darren Hale (not verified), 2 years 46 weeks ago
Comment id: 2381
I manage an agile software development team that has built a credit card processing system. Unfortunately, we've had a few pushes where we played whack-a-mole too. One approach we've tried is to create a "punch list" of work items. Every day at the stand-up meeting, we review what pair is going to work on which item, working our way down the list.
During this time, we push changes to a shared environment as often as possible and interact with the users constantly. The approach is not ideal from a Scrum/Agile perspective, but it does keep with the spirit of agile development: we maintain constant contact with business owners and users; we plan and prioritize often; we release often.
My team has been very fortunate that our product owners do a great job prioritizing critical fixes. It has helped us keep these pushes to a week or two at the most.
Maybe this approach, with a heart-to-heart with your project owner, will get you back to a more sustainable development cycle.
Bug fixing
March 17, 2009 by jackMilunsky, 2 years 46 weeks ago
Comment id: 2383
Oh boy, it certainly sounds like a tough situation to be in. But I believe that the anonymous post above is on the money.
I would suggest that it's ok under these circumstances to just focus on quality and drop everything else. The fact that as you fix bugs you find more would suggest that your code could be fragile or not all the cases have been considered. To deal with this, start writing unit tests especially around the critical aspects of the code. As well, I would highly recommend that you pair program the fixes. I.e. get two developers on each bug. It will make the world of difference.
Frankly I am not surprised you're having issues as it sounds like this team worked lot's of over-time. This is not "kosher" situation.
I agree with the previous reply, definitely start estimating everything, do this the planning poker method.
You have got to get the quality under control, before you can proceed with new feature development.
Hope this helps you
Jack
www.agilebuddy.com
blog.agilebuddy.com
From my point of view using
March 26, 2009 by software development services (not verified), 2 years 44 weeks ago
Comment id: 2413
From my point of view using agile development process for making application is quite reliable and chance of crash down is very low but for those whose work ways is agile must have strong knowledge of each phase of development process.
Post new comment