Guidelines for Agile, Challenge-based Procurement

— By Robert L. Read, PhD, and Henry Poole, CEO of CivicActions

In July of 2015, Chris Cairns, Dave Zvenyach and other leaders of 18F within the General Services Agency (GSA) of the Federal Government revolutionized government IT procurement. They challenged vendors to apply for a coveted position on a blanket purchase agreement by rapidly writing code to prototype a simple task, rather than writing written proposals. In this article we seek to explain and analyze what 18F started by articulating some guidelines for program managers of any government agency to perform Agile, Challenge-based Procurement. This applies to government at the Federal, State and local levels.

In a nutshell, here is how government IT contracting usually works:

We the people, via our elected officials, ask for something to be built.
Because our government believes in fairness, building something is done through competition, so a “Request For Proposal” (RFP) is written describing what the people want as best the government can describe it. This works very well for things like bridges.
But software is not like bridges. Few people know precisely what they want until they see it. With bridges and highways, the physical environment is static, and the technology changes slowly. With information systems, the boundaries are often fluid. It is often impossible to fix the boundaries and content because they are subject to change after the procurement is awarded. So the the RFP does a very bad job of capturing what is desired even if it is written by geniuses.
Many people who are highly technical would rather work in private industry than for government, so the RFP often is written by people who are not professional computer system designers or programmers, even though the the subject matter is computer programming.
Many large firms keep specialists on hand whose job it is answer these RFPs. In order to be successful, most firms develop a core competency in understanding how to interact with the rules of the Federal Acquisition Regulations and habits of the program managers in government. This ongoing expense becomes fixed. It becomes a competitive advantage for the so-called “Beltway Bandits”.
Because small firms don’t have that kind of expertise, they often don’t even bid on government contracts, because it doesn’t seem worth it to deal with red tape when there is plenty of other lucrative work.
The bidders write very long proposals that read like “brag sheets”. We’ve all read them. They are often just a list of previous accomplishments. In some cases, they don’t appear to reflect the specifics of the RFP in any way, as if the firm didn’t even bother to read it.
A small panel of moderately skilled people read through the proposals, each of which is 10 to 25 pages long (and often over 100 pages), and grade them much the way you would evaluate an essay written in High School with one big exception. If a bidder doesn’t meet one requirement — which could be completely administrative — they instantly fail. This one requirement could simply be exceeding the font size in their bid. There can be no communications with the firms during this time. Anything unclear simply remains unclear.
Then a procurement officer takes all of the “grades” of those who have not been disqualified and factors in cost estimates and awards the contract to one or several bidders.
Then, for the first time, possibly a year after the RFP went out, the firms get down to the business of learning what really needs to be done.
Often, the winner (or winners) are now able to provide all of the services for that agency for years to come without open competition.
The process is so cumbersome for the government that once a contract is awarded it is more difficult to re-open the competition than it is to stay with a subpar vendors.

What 18F did with the Agile BPA (Blanket Purchase Agreement) was to stand this on its head, in a method that may be informally called “Show, don’t tell!”:

The firms were to be evaluated not on a document, but on a working prototype and the process by which they designed and coded it.
This prototype had to demonstrate the use of modern technological systems that presumably were close to what was needed for the real tasks at hand.
The firms had to demonstrate a modern software methodology process.
The time from the revelation of the definition of the task to the due date was rather short.
The process was by rule completely open source. The code produced had to be open to the public, reusable by them, and the government.
The process included a budget that was used to evaluate the costs.
The firms that won were placed in a pool that could be given tasks (and thereby money) without further formal competition.

The Agile BPA was revolutionary because it required the demonstration, rather than the assertion, of Agility, competence, and user-centeredness. Although the process, like any beginning, had some rough spots and blemishes, it has already inspired at least three imitators, such as the State of California’s Health and Human Services Agency’s Agile Development Pre-Qualified (ADPQ) Vendor Pool, RFI #75001. Possibly the Department of Homeland Security learned from improved upon this growing competency with its recent DHS-FLASH, which uses a different structure. The State of Mississippi is running a challenge, similar in structure to the original 18F Agile BPA.

Why The Taxpayer Deserves Agile, Challenge-based Procurement

If a program office does a good job making their next procurement Agile and Challenge-based, you and your taxpayers will get the following benefits:

You will get the best firms you can get, because you are creating a meritocracy of computer competence rather than paperwork compliance.
You will gain valuable insight into your project early in your competition. In essence, you will begin the project several months early. Nothing decreases the risk of a software project meeting its deadline more than starting interaction with users early.
User interviews will be created early, and from so many different firms that your biases will automatically be eliminated.
Valuable designs and code will be made available to the whole world, but perhaps more relevantly the winning bidders will have every legal right to use the code produced by the losing bidders (and vice versa.)
You are relieved of the burden of writing a complete, detailed, correct RFP for a major project — -which is impossible. Instead, you will have to teach yourself to work in Agile Methodologies, beginning with the procurement.
Most importantly, you will be de-risking your project by becoming more agile yourself, and obtaining a team that is Agile.

Two Approaches: Who provides the User?

The 18F Agile BPA and DHS-FLASH were structured differently, and it is worth discussing the differences because they strike at the core of Agile and User-centered Design: the relationship of the development team to the user. The difference comes down to this: who provides the user?

In the Agile BPA, firms were expected to find users on their own. This had the advantage that the government did not have to provide a user to a large number of firms. The disadvantage is that the government cannot observe directly how the firm interacts with users, but must rely on artifacts which it expects the firm to provide.

In the DHS-FLASH (which is ongoing at the time of this writing), the Federal government has taken upon themselves to provide a human being to act as a “user representative” — presumably either an actual user or someone familiar with the actual user. In theory, this provides an opportunity to directly test a firm’s ability to interact with the user. However, it seems logistically difficult for the government. It means that the challenge competition is necessarily limited to shorter period of time — -one day at most in this case. Because firms must necessarily be treated fairly this seems to imply that challenges must be run on different days, and that secrecy of the subject matter must be maintained. We find this unfortunate, as there are tremendous advantages to being “open by default” as the USDS Playbook calls out or “open from the first line” as we used to say at 18F.

From a firm’s point of view, each approach also has disadvantages and advantages. If the firm must travel to physically meet with the user on a one-day challenge, this necessarily entails travel expense. If the challenge is extended across multiple days or weeks this means an even greater expenditure of labor, so in both cases rising to a challenge will be a moderate expense for a small firm.

As government contractors who have both the privilege and burden of participating in these challenges, we hope the government will continue to be cognizant of the costs imposed on firms by trying to keep challenges well-organized and short. We recommend that the government choose the “firm provides the user” or “government provides the user” approach based on their specific need to learn about the firms, and what makes sense for the firms. For example, a county government that does not expect participation from non-local firms does not add extra costs to local firms by using the “government provide the user” method. If the program office believes that firms can reasonable find exemplary users given the subject matter of the challenge, it is more reasonable to use the “firm provides the user” method.

Guidelines for Agencies

It takes great creativity to run an agile procurement. The textbook on how to do it has not yet been written. However, limited experience has already revealed some guidelines for government agencies. The guidelines will make your procurement more fair, more effective, less risky, and ultimately save money for the taxpayers. Understand that these are guidelines, not rules, and they represent our own untested opinions in some cases. We have divided these principle to those pertaining to content, process and technology.

Content

Keep it real. Life is too short for empty exercises. Ask your vendors to do something which is small but measurably advances your goal in some way. Act as if you intend to use every scrap of code written by every vendor to help the taxpayer in some way. This may require you to get creative, but remember — computer technology often advances by playing with ideas to formulate the best approach. If you get 10 minimal, tiny versions of your project in response to this RFP, you should potentially be able to learn something meaningful from each one.
Insist on open by default if you are using the “firm provides the user” method. The public is paying you and for this process, and they deserves the right to use, and more realistically, examine and learn from, the work developed in this RFP. You protect yourself by insisting that everything be done in the light. If you are using the “government provides the user”, still make a condition of challenge participation that all code developed by the challenge is in the public domain and will be published after the challenge and may be freely reused by anyone — most especially by the government itself.
Insist on being user-centered. By insisting that firms compete on how user-centered they can be and by working in the light, you in fact are recruiting the firms to begin the process for you — months ahead of a traditional procurement process. If firms compete to show how well they listen to the user, you will be learning from the users immediately, rather than after the award, because everything the firms learn will guide your future work. This is one of the principles of Agile Software Development —“Value Customer collaboration over contract negotiation.”

Process

Limit the time period as much as possible to make it less expensive for firms to compete. If you are using the “firm provides the user” method, absolutely limit the competition to 5 days, and probably 2 or 3 would be better. If you are using the “government provides the user” method limit the coding time to 4 or 5 hours, perhaps with additional non-coding time. This will seem ridiculously short to many people. However, you do the firms a favor by limiting the time they have to invest in attempting to win your business. You may announce the competition as much in advance as you want, but the specifics you are asking for should be revealed at the beginning of the competition. A firm that cannot produce and deploy a prototype of some kind in a short space of times needs to use this challenge as practice and hope to win the next one.
Demand that firms default to open (to you) from the beginning of the competition, not go public at the final hour. The code should become public at the end of the competition no matter who provides the user. You should demand to see the development over the competition period as it burgeons. This includes artifacts such as scrum meeting notes, user interviews, UX sketches, story maps, user stories, and design ideas. Require that the firms use a version control system like Git that shows revision history by contributor.
Build into your evaluation process a reward for firms that work well with other firms. A firm that uses code written by another firm should be rewarded for doing so, if it actually makes their prototype better. Even better is a firm that both uses a different firm’s work, and actively contributes back to another firm in some way. Far from demanding independent work, you should value inter-firm cooperation, although not to such an extent that firms chase that rather than delivering value to the users. Great teamwork is key.
Evaluate firms based on how well they use other people. This includes “social networks” but really means “social networks throughout all time and space”. For example, free and open source software represents person-centuries of human cooperation. The ability to tap into that cooperation is even more key to efficient programming at the time of this writing in 2016 than it was 30 years ago. However, firms must also utilize end-users, experts that they may not have on staff, etc. Reward firms for their connectedness into diverse pools of human knowledge and talent.
Plan to take enough time to evaluate the produced prototypes and code repositories. If you have so many responses that this is a burden, create a formal down-selection process, such as evaluating each submission in a full 20-minute timebox, disqualifying those that seem clearly inferior, to allow a longer round of evaluation of the qualified round.
Don’t create a long competition and then ask for a budget of the money or time spent, because you have no way to verify its accuracy. Instead just require that each human contributor from vendor firms identify each discrete contribution.

Technology

Anything we say about technology will be wrong in a few years. Nonetheless we must not shy away from trying to understanding the rapidly evolving technological landscape — nor may you.

Get help. If you are a program manager or a procurement officer, it is not your job to master all of the latest technologies — -but it is your job to seek out experts who have. At a minimum your RFP should be reviewed by a technical expert, what 18F Consulting (now split into several arms) once called “RFP ghostwriting”.
Expect to learn something from your bidding firms. They will probably use some technology new to you. However, if you really want to see technology X used, go ahead and ask for it to be used in some meaningful way, but only if you understand what you are asking for. Do not ask for a specific technology because of hype and buzz.
Write your RFP to demand capabilities, not technologies, if you can. With luck, you may discover that some solution is possible that you did not imagine. For example, with modern APIs, Ajax, and modern browsers, it is often possible to implement interesting functionality in a serverless manner. An RFP that demanded a particular language or server would close out such innovative possibilities.
Dealing with legacy code is a nearly universal part of the job. If you have a legacy database or API or codebase which can be opened to the firms, do so, and evaluate the effectiveness of their ability to work with/around/through the obsolete system.

Guidelines for Firms

A firm bidding on prototype-based Agile competition has a wonderful, and stressful, opportunity. To make the most of it, follow these guidelines.

Plan your participation to use your technical bench during periods when they would be underutilized on other work.
Prepare by having an Agile team in place ready to hit the ground running.
If you can afford it, rehearse. The shorter the challenge, the more you need to rehearse. Rehearsing three times, as realistically as possible, is not unreasonable for a one-day challenge.
If the government provides the user, rehearse by choosing an outside party to play the role of the government user. Plan for them to do what they honestly believe the government will ask, but also plan for them to throw a “curve ball” to the team. The team needs to rehearse dealing with unexpected situations.
Utilize this as a chance to explore new technology and new processes. This is a chance to sharpen up your software methodology process.
Use this as a chance to build connections between people within your own firm by bringing together people who may not have worked together before.
Divide your short time into at least three but at most six short sprints. Compress the whole sprint cycle down to one hour if you have to, and stick to your compressed process. You can’t afford to be out of control, and you can’t afford not to demo early. If you don’t demo in the first sprint, you will miss the deadline at worst and fail to learn something early from your users at best.
Buy your inner perfectionists a long vacation and block them on social media for the duration. In the short period of the challenge you will be given, whatever you build is going to be tiny and crappy compared to what you can imagine — unless you have no imagination. In the words of Guy Kawasaki in Rules for Revolutionaries: “Don’t worry, be crappy.”
Exert yourself but have fun. Celebrate tiny victories. This will be a race, and no race is ever easy, but you should be able to look back on it and say you were glad you ran the race, whether you win the bid or not.
Plan to learn on each bid. Don’t envision failure — but be prepared to fail and extract learnings from each failure.
Don’t depend on leading the marathon on your first attempt. If you cannot afford to risk a full team’s availability on a procurement, then find form an alliance with another firm that can share your risk and extract good value from learning.

(Robert L. Read is @RobertLeeRead at Twitter.)

(Henry Poole is @HenryPoole at Twitter.)

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMIfamily. We are now accepting submissions and happy to discuss advertising &sponsorship opportunities.

To learn more, read our about page, like/message us on Facebook, or simply, tweet/DM @HackerNoon.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!