America’s Drinking Water Isn't Protected Enough

This story was originally published on ProPublica by Peter Elkind and Jack Gillum; Doris Burke contributed reporting.

On Feb. 16, less than two weeks after a mysterious attacker made headlines around the world by hacking a water treatment plant in Oldsmar, Florida, and nearly generating a mass poisoning, the city’s mayor declared victory.

“This is a success story,” Mayor Eric Seidel told the City Council in Oldsmar, a Tampa suburb of 15,000, after acknowledging “some deficiencies.”

As he put it, “our protocols, monitoring protocols, worked. Our staff executed them to perfection. And as the city manager said, there were other backups. ... We were breached, there’s no question. And we’ll make sure that doesn’t happen again. But it’s a success story.”

Two council members congratulated the mayor, noting his turn at the press conference where the hack was disclosed. “Even on TV, you were fantastic,” said one.

“Success” is not the word that cybersecurity experts use to describe the Oldsmar episode.

They view the breach as a case study in digital ineptitude, a frightening near-miss and an example of how the managers of water systems continue to downplay or ignore years of increasingly dire warnings.

The experts say the sorts of rudimentary vulnerabilities revealed in the breach — including the lack of an internet firewall and the use of shared passwords and outdated software — are common among America’s 151,000 public water systems.

“Frankly, they got very lucky,” said retired Adm. Mark Montgomery, executive director of the federal Cyberspace Solarium Commission, which Congress established in 2018 to upgrade the nation’s defenses against major cyberattacks.

Montgomery likened the Oldsmar outcome to a pilot landing a plane after an engine caught fire during a flight. “They shouldn’t celebrate like Tom Brady winning the Super Bowl,” he said. “They didn’t win a game. They averted a disaster through a lot of good fortune.”

The motive and identity of the hackers, foreign or domestic, remain unknown. But Montgomery and other experts say a more sophisticated hacker than the one in Oldsmar, who attempted to boost the quantity of lye in the drinking water to dangerous levels, could have wreaked havoc.

They’re skeptical of the city’s assurances that “redundant” electronic monitors at the plant protected citizens from any possible harm.

“If the attackers could break into the lye controls,” Montgomery said, “don’t you think they could break into the alarm system and alter the checkpoints? It’s a mistake to think a hacker could not introduce contaminated water into our water systems.”

Oldsmar officials, citing the ongoing investigation, declined ProPublica’s requests for an interview or to address emailed questions about the city’s cybersecurity practices.

The consequences of a major water system breach could be calamitous: thousands sickened from poisoned drinking water; panic over interrupted supplies; widespread flooding; burst pipes and streams of overflowing sewage.

(This is not merely theoretical. In 2000, a former municipal wastewater contractor in Australia, rejected for a city job, remotely manipulated computer control systems to release 264,000 gallons of raw sewage, which poured into public parks, turned creek water black, spilled onto the grounds of a Hyatt Regency Hotel and generated a stench that investigators called “unbearable.” The man was sentenced to two years in prison.)

In congressional testimony on March 10, Eric Goldstein, cybersecurity chief for the federal Cybersecurity and Infrastructure Security Agency, described the Oldsmar incident as illustrating “the gravest risk that CISA sees from a national standpoint.”

He said it should be “a clarion call for this country for the risk that we face from cyberintrusions into these critical systems.”

Grave warnings have sounded for years.

As far back as 2011, a Department of Homeland Security alert advised that hackers could gain access to American water systems using “readily available and generally free” internet search tools.

Such admonitions have abounded in recent years.

Booz Allen Hamilton’s 2019 “Cyber Threat Outlook” called America’s water utilities “a perfect target” for cyberattacks; a 2020 Journal of Environmental Engineering review found “an increase in the frequency, diversity, and complexity of cyberthreats to the water sector”; and the Cyberspace Solarium Commission’s March 2020 report warned that America’s water systems “remain largely ill-prepared to defend their networks from cyber-enabled disruption.”

Despite the warnings, and some high-profile breaches dating back a decade, the federal government has largely left cyberdefense to the water utilities. For years, it relied on voluntary industry measures, dismissing any need for new regulation.

Then, in 2018, Congress included a provision addressing cybersecurity in a 129-page water bill that covered everything from river levee repairs to grants for school water fountains.

The requirements were less than demanding. Every U.S. water system serving more than 3,300 customers was obliged to conduct a self-assessment of the risks and resilience of its physical and electronic systems and prepare an emergency-response plan.

Different-sized utilities got different deadlines; for the smallest covered by the law, such as Oldsmar, the self-assessment must be done by June 30, 2021, more than two and a half years after the law was signed.

(Oldsmar had completed its cybersecurity review by early November but hadn’t yet incorporated its recommendations in the city’s emergency response plan before the February hack, according to a statement provided by the city manager.)

Tens of thousands of U.S. water systems with fewer than 3,300 customers were exempted entirely from the law’s requirements.

Those utilities required to perform a self-assessment were not obliged to submit a report to any government agencies. The utilities merely had to attest to the Environmental Protection Agency that they had conducted the assessment.

The 2018 legislation also provided $30 million for grants to help water districts deal with “risk and resilience” problems, including cyberattacks. But Congress never appropriated that money.

The water provisions fall far short of federal requirements (including penalties for violating those rules) and funding aimed at protecting electricity infrastructure, according to Montgomery.

“An assessment’s a good thing,” he said. “But this is well short of what we require from energy companies. We have developed a tool for self-identification of problems. But if you’re really bad at cybersecurity, I’m not sure your self-identification is going to solve the problem.”

He also pointed to low staffing at the EPA’s Water Security Division. “The water security office is a handful of people, probably three,” Montgomery said.

“It historically has not done much, if any, cybersecurity work. This is the product of 20 years of low prioritization.”

The agency’s most recent report to Congress on “Drinking Water Infrastructure Needs,” submitted in 2018, identified $472.6 billion in long-term priorities, but it didn’t mention the word “cybersecurity” once in its 75 pages.

An EPA official, speaking on the condition of anonymity, agreed that the agency had only “a small team” devoted to water cybersecurity but said Oldsmar “and other recent incidents have highlighted the importance of the priority and the investments we need to make.”

The origins of the problem are clear. The vast majority of the nation’s water systems are small and publicly owned, with limited resources and aging infrastructure.

As they turned to digital systems and monitors to boost efficiency while saving money and staff, they failed to install the safeguards and carry out employee training needed to secure the resulting vulnerabilities.

“Every one of them had one guiding principle over the last 50 years: increased automation to lower the size of the workforce to keep costs down,” Montgomery said.

“Along with that, there should have been an investment in the cybersecurity of the infrastructure. But that did not happen.”

Traditionally focused on physical risks, such as natural hazards, burst pipes and on-site intruders, most water systems also have little or no in-house IT staff.

The pandemic, which encouraged remote management, has only made the problem worse.

In testimony last month to the House Homeland Security Committee, former CISA Director Chris Krebs called Oldsmar’s vulnerability “probably the rule rather than the exception. ... These are municipal facilities that do not have sufficient resources to have robust security programs. That’s just the way it goes.”

The industrial control systems that water districts use to manage valves, pipes and other infrastructure are notoriously open to attack.

A 2018 study by IBM and a private security company found 17 major vulnerabilities in equipment widely deployed in “smart cities,” a term that refers to municipalities that manage a wide array of their systems — anything from water treatment plants to parking meters and street lamps — via the Internet.

Among the security problems: Every product the group examined was still using the default passwords (such as “admin”) they came with in the box, allowing “even the most novice hacker to easily gain access to these devices.”

A 2018 study by the firm Positive Technologies reported that it was able to penetrate nearly three-fourths of industrial organizations it investigated, revealing gaps offering hackers “plenty of opportunity to access critical equipment.”

The most common vulnerabilities: remote-access networks, obvious passwords and software so old that the manufacturer had stopped making fixes to protect against intruders.

The report found that vulnerabilities known for years often “remain untouched, because organizations are afraid to make any changes that might cause downtime.”

These industrial control systems are considered such obvious targets that hacking contests use them as quarry.

At the DEFCON computer security conference, an “ICS Village” let curious programmers try to break into devices set up inside a Las Vegas hotel room — demos not connected to real-life systems — in an effort to expose weaknesses.

At the event in 2018, one water pipe control system, likely used for a commercial building, had its computer screen defaced with graffiti-type messages.

The exact number of attacks on water utilities remains unknown. Many go undetected or unreported, and no federal law requires disclosure, even to regulators or law enforcement.

Michael Arceneaux, managing director of the Water Information Sharing and Analysis Center, an industry group promoting cybersecurity, said water systems often refuse to reveal breaches, even to his group, out of fear that they will somehow reveal their vulnerabilities to other hackers.

“It’s not something members wanted potentially floating around in some database.”

The episodes that have been made public reveal a growing array of threats, from random vandalism and disgruntled employees to identity theft and ransomware.

In Oldsmar, for example, the FBI and the Pinellas County Sheriff’s Office, which are jointly investigating, have already revealed multiple lapses.

The attack took place at the city’s water treatment plant, which purifies groundwater for drinking using filters and chemicals, including small amounts of sodium hydroxide. Commonly known as lye, it is used to reduce the water’s acidity.

(In considerably stronger concentrations, sodium hydroxide is also a chief ingredient in drain cleaner.)

The hack began around 8 a.m. on Feb. 5, when a plant operator noticed someone had remotely accessed the computer system that monitors and controls the chemical levels added to the water. The hackers entered through a remote access software program called TeamViewer.

The city had actually replaced TeamViewer six months earlier, but it never disconnected the program, according to county Sheriff Bob Gualtieri.

Logging into the system remotely was a breeze: The water plant’s computers all used a single shared password, required no two-factor verification and had no firewall in place protecting the controls from the internet, according to FBI findings described in a Massachusetts state advisory.

A final vulnerability: All the computers were still running on Windows 7, a decade-old, discontinued operating system; Microsoft had stopped issuing regular software updates to plug its security vulnerabilities in January 2020.

After noticing the hacker’s morning log-in, Gualtieri later said at the press conference, the plant operator “didn’t think much of it” and didn’t contact anyone since other city employees routinely accessed the system remotely.

(It’s not clear why the attacker’s use of the replaced TeamViewer software didn’t immediately raise concern.)

The hacker reappeared about 1:30 p.m., this time visibly taking over the computer, mousing around for three to five minutes and opening the plant’s control system software.

After ratcheting up the water’s sodium hydroxide level from 100 parts per million to 1,100 parts per million, the intruder departed.

After watching all this, the Oldsmar plant operator quickly lowered the sodium hydroxide level and called his boss. The city contacted the county sheriff’s office nearly three hours later, at 4:17 p.m., according to an incident report on the event.

Oldsmar officials maintained that the public was never in danger.

They noted that it would have taken at least 24 hours for poisoned water to start flowing out of kitchen taps, and that even if the onsite operator hadn’t intervened, the plant had backup systems monitoring the water’s chemical balance that would have sounded alarms long before then.

A small number of other incidents present the nightmarish “what-if” scenarios that scare experts, particularly from so-called state actors. Both Russia and Iran have been implicated in such accounts, according to government reports and legal actions.

One such episode occurred in 2013, when a state-backed hacker sitting at his keyboard in Iran breached the computer controls at the Bowman Dam in suburban Rye, New York, with a presumed plan to open the sluice gates.

The gates happened to have been manually disconnected at the time for maintenance, and the dam was actually just a narrow, 20-foot-high structure holding back a babbling brook.

Federal intelligence officials speculated that the Iranians had actually intended to seize controls at the massive Arthur R. Bowman Dam in Oregon, where similar actions would have flooded thousands of homes.

A federal indictment later charged that the Bowman Dam hacker worked for Iran’s Revolutionary Guard and was part of a seven-man team that successfully breached America’s biggest banks, paralyzing their computer servers and blocking customers from accessing their accounts online.

The hacker remains at large, and on the FBI’s “most wanted” list. In 2019, Revolutionary Guard hackers struck again, deploying malware to launch an ultimately unsuccessful attack on a municipal water system in Israel.

In recent years, three U.S. states — New York, New Jersey and Connecticut — decided to go beyond the federal rules and adopted tougher cybersecurity measures for the water utilities within their borders.

After passing new legislation, New Jersey required all public water systems with internet-connected controls to develop a cybersecurity risk-mitigation plan within 120 days, submit it to the state, create a process for reporting all cyberattacks and join a special state-government clearinghouse promoting strong cybersecurity practices.

Connecticut launched a “Cybersecurity Action Plan” and began holding private annual meetings with each of the state’s largest water (and other) utilities to scrutinize the adequacy of their cyberdefenses.

For its part, New York amended its public health law to require water systems to conduct assessments of their susceptibility to cyberattacks and submit them to the state within a year.

A team at the state comptroller’s office has also conducted seven cybersecurity audits of municipal water systems, in each case posting the audit publicly while reserving some findings for confidential briefings to avoid offering hackers a road map of vulnerabilities.

Its audit of the city of Syracuse’s water system, for example, found shared user passwords and accounts that hadn’t been disabled long after employees left the city.

The Binghamton audit discovered a video on the water department’s own webpage showcasing the treatment plant’s controls.

“There’s a tremendous amount of work that needs to be done to shore up the systems,” said assistant New York state comptroller Randy Partridge, who oversees the water system audits.

Since January 2019, he said, his auditors have issued 239 findings at various municipal facilities (including water systems) regarding weak password security alone.

“It’s a health and safety risk for any resident that lives in our local government. No community can really survive for any length of time without access to potable water.”

Arthur House, who served as Connecticut’s chief cybersecurity risk officer, said: “I hope it doesn’t take the poisoning of a lot of people or a catastrophic shutdown for people to say, ‘Omigosh, this is serious.’ The federal government has to have a role on this. You cannot leave something that would cripple us as a country solely in the hands of 50 different states.”

Photo by Thomas Kinto on Unsplash