#42 - Why government software sucks
The government's laughable vaccine portal
I.
One way to think about management consulting is that it’s prestige arbitrage. You have all these random companies in, say, North Dakota, that would love bright Harvard graduates to do business strategy for them, but these Harvard graduates would hate to work at a no-name company in the middle of North Dakota. So, a McKinsey comes along and says, “You know what? We can get bright Harvard grads to come and ‘work’ for us because we have ~prestige~, and we can then send them off to ‘consult’ for you!”
I can try to make a similar argument for technology consulting for the government. The federal government would love for the best software engineers to write government software, but the best software engineers would hate to do it because the government is bureaucratic, slow, formalistic, low-paying, etc. So, an Accenture or a Deloitte comes along and says, “Let’s do what McKinsey does! We can accumulate ~prestige~, attract the best engineers, and then send these engineers off to build software for the government.”
Except that’s not how things have played out. The best software engineers are not at Accenture or Deloitte but at FAANG and the new hottest startups in Silicon Valley. Why? As a threshold matter, software engineers (in my experience) don’t care nearly as much about prestige as management consultants do. Rather, they care about freedom, moving fast, and building cool and new stuff, and while tech companies have those things in spades, govtech consulting companies do not. So, the mean software engineer at Accenture and Deloitte, while good, is just not as good as the mean software engineer at Google. As a result, funnily enough, it’s actually Google that slowly accumulates the prestige, anyway, not Accenture / Deloitte.
II.
So anyway, here’s the MIT Technology Review a couple weeks ago, describing the failure of the government’s vaccine management software:
[E]arly in the pandemic, the CDC outlined the need for a system that could handle a mass vaccination campaign, once shots were approved. It wanted to streamline the whole thing: sign-ups, scheduling, inventory tracking, and immunization reporting.
In May, it gave the task to consulting company Deloitte, a huge federal contractor, with a $16 million no-bid contract to manage “covid-19 vaccine distribution and administration tracking.” In December, Deloitte snagged another $28 million for the project, again with no competition. The contract specifies that the award could go as high as $32 million, leaving taxpayers with a bill between $44 and $48 million.
However, this system, called the Vaccine Administration Management System (VAMS), sucks:
[I]t was supposed to be a one-stop shop where employers, state officials, clinics, and individuals could manage scheduling, inventory, and reporting for covid shots—and free for anyone to use.
Instead, “VAMS has become a cuss word,” Marshall Taylor, head of South Carolina’s health department, told state lawmakers in January. He went on to describe how the system has badly hurt their immunization efforts so far. Faced with a string of problems and bugs, several states, including South Carolina, are choosing to hack together their own solutions, or pay for private systems instead.
Clinic workers in Connecticut, Virginia, and other states say the system is notorious for randomly canceled appointments, unreliable registration, and problems that lock staff out of the dashboard they’re supposed to use to log records.
I’m honestly not surprised. Government software historically has been extremely expensive to build yet has also clunky, slow, and ridden with bugs. That said, I’m somehow still disappointed. The failure of government software is, of course, overdetermined: There are multiple causes of the failure, any one of which could plausibly be sufficient to account for the failure. My goal is not to outline the kaleidoscope of possible causes but to zero in on a couple that I think hold particular explanatory power: (1) Government culture is rigid; and (2) Government processes are bloated.
III.
Government culture is so rigid that it requires projects to be perfect on launch, and that is just ridiculous.
Part of this rigidity is due to over-planning. The government’s well-documented method of managing software projects is known as “the waterfall”: a central calendar, where highly-specified tasks “cascade” through the phases of planning, analysis, design, building, testing, blah blah blah, with concrete deadlines for every small thing. It’s the Gantt chart to end all Gantt charts. In essence, the government runs software development as a bureaucratic process, with everything being managed to fine degrees of precision.
The government is also polarized, which further contributes to the desire to get everything right on the first try. Here’s the New York Times back in 2013 writing about President Obama’s botched Affordable Care Act portal:
[F]ailure was not an option . . . Nor was rolling out the system in stages or on a smaller scale, as companies like Google typically do so that problems can more easily and quietly be fixed. Former government officials say the White House, which was calling the shots, feared that any backtracking would further embolden Republican critics who were trying to repeal the health care law.
Together, the excessive planning and polarization in government form the cornerstone of the idea that “failure is not an option,” and this idea is imputed to into the govtech consulting companies. This idea, however, is nonsensical in the technology world. Venture capitalists don’t micro-manage (or at least the good ones don’t). Instead, they give the entrepreneur money and pretty much get out of the way until they can be helpful. At a software company, the engineers use “agile” software development. They run experiments and pivot the product based on continual user feedback. Facebook runs experiments on a fraction of a fraction of the population to see how tiny changes (even changing the color of a button) would affect engagement. Contra government culture, failure is always an option in tech culture, and it’s not necessarily a bad one, so long as you “fail fast” and “fail forward.”
I started off by saying that technology consulting companies that build govtech can’t rely on arbitraging prestige. Instead, the most important thing is building a culture that is more tech than gov. One company that has actually made this work is Palantir. Instead of arbitraging on prestige, it emphasizes those other cultural things that software engineers care way more about. Yes, Palantir builds software for the government, but it is not a formalistic, suit-and-tie government consulting company. At its heart, Palantir is a technology company. It has a self-professed “engineering culture” with mantras like “the best idea wins” and “nothing is permanent.” Unfortunately, government culture says the exact opposite.
IV.
Again, the government paid $44 million for an online portal to handle “sign-ups, scheduling, inventory tracking, and immunization reporting.” At face, this sounds … appallingly simple to do? Couldn’t you have paid a few college grads with a few hundred bucks and some pizza to bang this thing out in a weekend?
Unfortunately, when it comes to government software, the simplicity of the end-product requirements belies the complexity of the administrative bloat. The actual software development piece of the contract is a pretty small portion of the overall work while the bulk of the work is spent on things like documenting processes and costs, figuring out all the privacy and security requirements, and performing background verification on your employees.
But the worst thing about government bloat is the fact that developers need to build new software on outdated legacy systems. When working with the government, developers have all these annoying questions they need to answer: Does a federal agency house the data? Which agency? Are there multiple agencies? Are state agencies also involved? How is their data currently structured? Do they have APIs for the developer to hook into? What is the current status of their software systems? And the frustrating answer to these questions usually goes something like: Multiple agencies need you to store the data in so and so format, and oh, by the way, our software sucks, but please figure it out, thanks!
This is nonsensical in the private sector. In startup world, founders are often described as trying to build a plane as they fly it, and that’s a perfect analogy. You’re going super fast, but you’re also starting with a clean slate. Companies like Facebook have “internal tools” teams whose sole purpose is to build, uh, internal software tools for other software engineers to code more quickly and efficiently. Stripe gets so much love from developers because its API is so well-documented, so easy to use, and so frictionless for developers to set up and get going.
Government software, though, is old and decrepit and a pain to work with. It’s like being told to cut fresh sashimi with a rusty hammer. It won’t work, and even if it somehow “does,” it’s gonna’ suck.
V.
As you can tell I’m very cynical about the government ever having great software. But maybe this will change soon? You have organizations and agencies like 18F, the U.S. Digital Service, and Nava that are trying to modernize / standardize government APIs and get more talented technologists building government software. And you also have those heart-warming stories about civic-minded technologists stepping up in the wake of healthcare.gov’s failure, and that’s really great, too. So I guess we’ll see. The failure of government software is wasting people’s money and literally costing people’s lives. We need to do better.
📚 What I’m reading
McKinsey settles for nearly $600m over role in opioid crisis. l o l (New York Times)
The pirate game. (Wikipedia)
The journalistic tattletale and censorship industry suffers several well-deserved blows. (Glenn Greenwald)
Five charts that explain the digital transatlantic relationship. (Politico)
The great unbundling. (Benedict Evans)
Facebook’s Oversight Board has spoken. But it hasn’t solved much. (WIRED)
Three-dimensional search engine Physna wants to be the Google of the physical world. (TechCrunch)
A policy of truth. (Pirate Wires)
Farming fish in the sky. (Hakai Magazine)