Finding a Partner to Trust - Using Competitive Sprints to Select an Agile Software Vendor

October 29, 2008 by Peter Stevens

Some people would like to believe that building complex software is like going to the grocery store: pick a candy bar off of the shelf, ask what it costs and decide to buy it. You get no risk and quick gratification. But building custom software is more like building a race car. A special one-off product to meet exactly the needs of its sponsor: win races.

As a customer, what are you really buying when you contract for software development? You may think you are getting a solution. But what you are really getting is an implementation team. And risk is always part of the bargain. So what you really want is a team you can trust to build your product and to minimize the risk of that choice.

In this article, I present a lightweight, lean approach to selecting a software development partner which should dramatically reduce the risk and cost to all parties.

Project Risk

A customer will often invest substantial effort into creating a Request for Proposal (RFP). I've seen RFP's whose size is measured in binders and the effort to produce them measured in Man-Years. A vendor will often invest a lot of effort into winning a big project. On both sides, this effort produces mostly paper, i.e. artifacts of no value except as means to the goal of selecting a vendor.

On the vendor side, this effort has to be amortized during the execution of the project itself. As there is only one winner, the others make a substantial effort and earn no money. So bidding on big projects is expensive and risky.

Competitive bidding can mean offering ruinous prices. When “winning” means producing at a loss (sometimes to the point where the supplier goes out of business [English translation]), even the customer loses. The supplier may have no choice but to play the change-request game to achieve profitability.

How do you maximize trust and minimize risk and cost? Customers are often willing to work with agile companies on a time and materials basis, once the trust has been established. Trust greatly reduces administrative overhead and enables a collaborative approach, so establishing trust must be the first priority in a project. How do you build this trust?

A Lean Approach

There are certainly many answers to this question, and some are better than others. Asking for quotes and having talks with the sales & consulting staff is not really enough to establish trust. Much better are working together with the actual team and evaluating real results. Deferring potentially expensive decisions until their implications are clear is usually advantageous. Here is a 3 step approach:

1. Use Scrum to create a lightweight, agile RFP (see creating the Agile RFP and contents of the Agile RFP)
2. Eliminate uninteresting vendors.
3. Select the ultimate winner through a competition which produces an increment of working software.

Why is this a lean approach? By deferring the final decision until you have actual experience with the candidates, you reduce the likelihood of picking a candidate that “looks good on paper” but cannot really deliver software. You reduce the risk of your decision substantially.

Eliminate Uninteresting Vendors

Identify your candidates and send them the RFP. Ask them questions which will separate the wheat from the chaff, for example:

• Please present the team which will carry out the project. How much experience do they have with Scrum and XP (Extreme Programming)?
• Are you willing to organize the project according to Scrum (as described in the Process chapter)? What experience do you have with agile projects on this scale?
• What is your estimate of the of the overall complexity (size) of the system in Story Points?
• What is your expected team velocity in Story Points per Sprint?
• Given our target budget, how much of the functionality do you think can be realized?
• Are you willing to participate in the competition to select the final vendor?

If the vendors are used to working on an agile basis, they will have no problems with these questions. If they are not, they will probably not even be able to respond, especially if deadlines are kept tight.

You will need to meet with prospective teams for a day or so to answer their questions about the user stories. Afterwards the vendors should be able to size the system and answer your questions quickly.

If there are more than two vendors still in the running, you will need to use the answers and the results of the interviews to trim the field down to two vendors, who then participate in the competition.

Competition

Let us assume that you plan to spend $2.4 Million over 12 months, or $200'000 / Month to develop the software. (You got this figure by calculating what the project is worth to the company, perhaps using a double worst case analysis).

The cost of picking the wrong partner is much higher than the cost of working in parallel for a short period of time.

If you add one month's effort to the budget, that would raise the total by 8%. But productivity differences among teams can be a factor of 3 to 10. Furthermore, the cost of delay while you take three months to pick a partner without producing any usable software will be much larger than the cost of redundant development for a short period.

Running the contest:

Here are the ground rules. There are two vendors. Both are going to start your project according to the process you defined in the RFP. After one month, both players present an increment of working functionality and you will decide which partner you want to continue working with. The winner gets a full renumeration for the month and a contract for the rest of the project. The loser gets 50% of the renumeration for a month.

Here are the steps in the competition.

1. Prioritize the backlog and, working with both vendors, create a set of fine grained stories which should be implemented in within a month.
2. Agree on the definition of done. This probably should include points which confirm that the partner is capable of test driven development and continuous integration. You do not want to incur technical debt. The same definition should apply to both contestants.
3. Agree on other playing rules, in particular the team members, who should be the same as communicated previously. Are non-invoiced staff allowed? Is overtime allowed? Who owns the software and ideas which are produced by the loser? You may need to ensure that quality does not get sacrificed for quantity as you will be producing code which one day will go live.
4. Hold the first sprint planning meeting together with both contestants, so both teams get the same initial briefing. Both teams get one month to implement an increment of functionality.
5. Finish the Sprint with a Sprint Review and Retrospective. This should only include the implementation team and the product owner, as it is part of the Scrum process. When the retrospective is complete, the competition is complete.
6. If the teams want to give a sales demonstration, this should happen after the competitive sprint is finished.

You might want to adjust the sprint duration or the trial period depending on the size of the project.

At the end of month, you have not just qualitative and quantitative data on which to base a decision. You have a month of experience working with your new partner. You have some working software (or not). In short, you have much more information to base your decision on.

And the winner is...

Everybody wins.

The risks and up front costs of producing and responding to the Agile RFP are much lower than the traditional approach. By working with the teams, you build confidence that you can work with them and that they can build the software you need. You can judge the teams based on working software rather than attractive presentations or other artifacts.

By starting to work on a solution early, you reduce your time to market. You will probably get a better solution, because the competition will spur everyone's creative juices. Even the loser will have some good ideas which become part of and improve the final result.

Epilogue

This concludes my series on Agile Project Planning in general and the Agile RFP in particular. This series was largely the result of a customer project. These are the tools we used and this is how far we have come. What (do I think) was novel in our approach?

1. Using Scrum and User Centered Design to create the RFP.
2. Double Worst Case Analysis to do sanity checks on budgeting assumptions
3. Competitive Sprints to select the final vendor.

The proof of the pudding is in the eating, but I as write this, I do not know if the customer will move forward with the project. So we can't taste the pudding (yet).

How would you implement competitive sprints? To what situations would it apply well or poorly? Have you used a similar approach?

Permalink