True Grid

Greetings

     I've always that one of the least vulnerable of our just-in-time system would be the grid.     However, sometimes I'm not so sure.  Here is some info on the vulnerabilities from the NYT..  Grid called vulnerable

"The electric grid, as government and private experts describe it, is the glass jaw of American industry. If an adversary lands a knockout blow, they fear, it could black out vast areas of the continent for weeks; interrupt supplies of water, gasoline, diesel fuel and fresh food; shut down communications; and create disruptions of a scale that was only hinted at by Hurricane Sandy and the attacks of Sept. 11. "   

 see  NYT:  US worries over grid

 Then below, Stuart Staniford reviews  a recently de- classified book - Terrorism and The Electric Power Delivery System.  Finally Staniford reviews the emergency plan of  for Pittsburgh.  He concludes:

"Thus the picture is that for a power outage of days or less, people have gas in their tanks, food in their cupboards, etc.  But for outages beyond that duration, the means to supply the populace with food, fuel, and water is not survivable and will rapidly degrade, and people will quickly be reduced to "refugee" status - needing either to leave to somewhere better equipped, or to rely on emergency measures like water buffalo trucks and food handouts at central distribution points."

   Better check that emergency water supply! 

Electrical Grid Is Called Vulnerable to Power Shutdown

By NICOLE PERLROTH

Stuart Isett for The New York TimesTwo researchers discovered that they could freeze, or crash, the software that monitors a substation, thereby blinding control center operators from the power grid.

• FACEBOOK
• TWITTER
• GOOGLE+
• SAVE
• E-MAIL
• SHARE
• PRINT

Over the past few months, the discoveries of two engineers have led to a steady trickle of alarms from the Department of Homeland Security concerning a threat to the nation’s power grid. Yet hardly anyone has noticed.

The advisories concern vulnerabilities in the communication protocol used by power and water utilities to remotely monitor control stations around the country. Using those vulnerabilities, an attacker at a single, unmanned power substation could inflict a widespread power outage.

Still, the two engineers who discovered the vulnerability say little is being done.

Adam Crain and Chris Sistrunk do not specialize in security. The engineers say they hardly qualify as security researchers. But seven months ago, Mr. Crain wrote software to look for defects in an open-source software program. The program targeted a very specific communications protocol called DNP3, which is predominantly used by electric and water companies, and plays a crucial role in so-called S.C.A.D.A. (supervisory control and data acquisition) systems. Utility companies use S.C.A.D.A. systems to monitor far-flung power stations from a control center, in part because it allows them to remotely diagnose problems rather than wait for a technician to physically drive out to a station and fix it.

Mr. Crain ran his security test on his open-source DNP3 program and didn’t find anything wrong. Frustrated, he tested a third-party vendor’s program to make sure his software was working. The first program he targeted belonged to Triangle MicroWorks, a Raleigh, North Carolina based company that sells source code to large vendors of S.C.A.D.A. systems. It broke instantly.

Mr. Crain called Mr. Sistrunk, an electrical engineer, to see if he could help Mr. Crain test his program on other systems.

“When Adam told me he broke Triangle, I worried everything else was broken,” said Mr. Sistrunk.

Over the course of one week last April, the two tested Mr. Crain’s software across 16 vendors’ systems. They did not find a single system they couldn’t break.

By the end of the week, the two had compiled a 20-page report replete with vulnerabilities in 16 different system vendors for the Department of Homeland Security’s Industrial Control Systems Cyber Emergency Response Team, I.C.S.-C.E.R.T., which notifies vendors of vulnerabilities and issues public advisories.

And then, they waited. It would take I.C.S.-C.E.R.T. another four months to issue a public advisory for Triangle MicroWorks’ system.

Triangle MicroWorks’ engineering manager Greg Godlevski said that during those four months, the company developed a number of its own tests to look for defects in its software and fix them. Mr. Godlevski said the company waited for confirmation from Mr. Crain that the problem had been fixed, then met with I.C.S.-C.E.R.T. several times to review and comment on the government advisory.

“We take any reported problems discovered in our products very seriously,” Mr. Godlevski said. “We expend a lot of effort adding levels of security to our protocols and ensuring that they comply to the published specifications.”

D.H.S. did not return a request for comment.

Over the course of those four months, Mr. Crain and Mr. Sistrunk  found vulnerabilities in an additional nine vendors’ systems.

Like most security alerts, there are some caveats to this concern for the safety of electric facilities: Mr. Peterson’s company, Digital Bond, sells consulting services to assess and improve the security of S.C.A.D.A. systems.

Mr. Crain also has an interest. In March, he plans to release a free version of his security test, but for now he is charging vendors to use his program. (Mr. Crain would not disclose pricing, since it differed for each vendor based on vendor size, saying only that he charged in the “thousands” though he said he charged far less than commercial services like WurldTech Security, which charges tens of thousands of dollars for similar programs.)

“We haven’t found anything we haven’t broken yet,” Mr. Crain said in an interview. At minimum, the two discovered that they could freeze, or crash, the software that monitors a substation, thereby blinding control center operators from the power grid. Mr. Crain likened that capability to “a bank robber being in a bank vault with the camera frozen.”

In the case of one vendor, Mr. Crain found that he could actually infiltrate a power station’s control center from afar. An attacker could use that capability to insert malware to take over the system, and like Stuxnet, the computer worm that took out 20 percent of Iran’s centrifuges, inflict actual physical harm.

“This is low-hanging fruit,” said Mr. Crain. “It doesn’t require some kind of hacker mastermind to understand the protocol and do this.”

What makes the vulnerabilities particularly troubling, experts say, is that traditional firewalls are ill-equipped to stop them. “When the master crashes it can no longer monitor or control any and all of the substations,” said Dale Peterson, a former N.S.A. employee who founded Digital Bond, a security firm that focuses on infrastructure. “There is no way to stop this with a firewall and other perimeter security device today. You have to let DNP3 responses through.”

Even more troubling, Mr. Peterson said, is that most DNP3 communications aren’t regulated. The original version of DNP3 worked on serial communications — a way of transmitting data usually found in things like coaxial cables — and is still widely deployed in large systems, particularly substations around the country. But current cybersecurity regulations, governed by the North American Electric Reliability Corporation’s (N.E.R.C.) Critical Infrastructure Protection Committee (C.I.P.C.) are focused on Internet Protocols, or I.P. protocols, and specifically exclude serial communications and the equipment that uses them from meeting any security requirements.

“Why isn’t D.H.S., N.E.R.C., and the DNP3 committee telling vendors they need to fix this now and utility owners they need to get this patched A.S.A.P.?” Mr. Peterson said.

To date, D.H.S. has posted nine advisories, several of them for software used by major players in the electric sector.

“This is a systemic problem,” Mr. Crain said. “Most of the top five utilities use this software and just because a patch is available, doesn’t necessarily mean that utilities are applying them.”

---------

Terrorism and the Electric Power Delivery System

I am currently reading Terrorism and the Electric Power Delivery System.  From the summary:

The electric power delivery system that carries electricity from large central generators to customers could be severely damaged by a small number of well-informed attackers. The system is inherently vulnerable because transmission lines may span hundreds of miles, and many key facilities are unguarded. This vulnerability is exacerbated by the fact that the power grid, most of which was originally designed to meet the needs of individual vertically integrated utilities, is now being used to move power between regions to support the needs of new competitive markets for power generation. Primarily because of ambiguities introduced as a result of recent restructuring of the industry and cost pressures from consumers and regulators, investment to strengthen and upgrade the grid has lagged, with the result that many parts of the bulk high-voltage system are heavily stressed.

A terrorist attack on the power system would lack the dramatic impact of the attacks in New York, Madrid, or London. It would not immediately kill many people or make for spectacular television footage of bloody destruction. But if it were carried out in a carefully planned way, by people who knew what they were doing, it could deny large regions of the country access to bulk system power for weeks or even months. An event of this magnitude and duration could lead to turmoil, widespread public fear, and an image of helpless- ness that would play directly into the hands of the terrorists. If such large extended outages were to occur during times of extreme weather, they could also result in hundreds or even thousands of deaths due to heat stress or extended exposure to extreme cold.

The largest power system disruptions experienced to date in the United States have caused high economic impacts. Considering that a systematically designed and executed terrorist attack could cause disruptions that were even more widespread and of longer duration, it is no stretch of the imagination to think that such attacks could entail costs of hundreds of billions of dollars—that is, perhaps as much as a few percent of the U.S. gross domestic product (GDP), which is currently about $12.5 trillion.

Electric systems are not designed to withstand or quickly recover from damage inflicted simultaneously on multiple components. Such an attack could be carried out by knowl- edgeable attackers with little risk of detection or interdiction. Further well-planned and coordinated attacks by terrorists could leave the electric power system in a large region of the country at least partially disabled for a very long time.

Alert readers may note that the $12.5 trillion figure for GDP is inconsistent with the 2012 publication date of this National Academies of Science report.  Apparently it was written in the 2004-2007 timeframe, but then classified until last year.

http://earlywarn.blogspot.com/2013/05/terrorism-and-electric-power-delivery.html

____

Sustaining a City in a Long-Term Power Outage

A few comments on this fascinating study from Pittsburgh (site of Carnegie Mellon, which is a center of excellence at studying critical infrastructure issues).  The key theme that emerges for me is the interaction of the liquid fuel system (particularly diesel) and the electricity system.  In a short outage, lots of critical infrastructure has diesel generator backup, and so the hospitals, 911-call centers, and so on can continue to operate.  However, they typically have limited fuel storage capacity (if for no other reason than that diesel doesn't keep indefinitely), and so in a long outage, the availability of diesel becomes critical to keeping everything together.

To illustrate the time factors, consider the water situation in Pittsburgh:

The Pittsburgh Water and Sewer Authority is responsible for providing the city of Pittsburgh with clean water for household and business use. Most of the electricity required at the Aspinwall Water Treatment Plant (WTP) is consumed pumping water from the river. From the treatment plant, water is pumped to the three primary reservoirs. About half of the water from the primary reservoirs is delivered directly to homes and businesses. The other half is pumped to a series of smaller reservoirs, tanks, towers, and standpipes around the city. In this report the main reservoirs are referred to as ‘primary storage’ and the smaller storage facilities as ‘secondary storage’.

Pumping into storage facilities is usually activated when water levels in the facility drop below a certain level. Storage facilities are normally kept full, but may drop to 80% in the evenings. Electricity is only needed to pump water into storage facilities. Once water is stored at a high point in a reservoir or a tower it can flow by gravity to any customer located below it.

During the course of this study, we found that immediately following a blackout, water supplies will be unaffected. In the absence of any backup generation, after one day of power outage, as many as 15% of customers could expect to lose water as secondary storage is depleted. All secondary storage is likely to be depleted after three days, leaving 50% of the population without water, increasing the load on primary storage and depleting the first of the primary storage reservoirs within about nine days. The last water storage will be depleted after two weeks.

So a short outage is no big deal, but between the first few days and two weeks, things start to go really bad, until the point where everyone is dependent on emergency measures:

Current emergency plans include distribution of water by tanker trucks (called water buffalos). Emergency response plans at the city and county level include steps to acquire these trucks from local governments and agencies. With a typical capacity of 2,500 gallons, these trucks would only be practical or providing minimal supplies of water. To provide all 370,000 people in Pittsburgh with an emergency one gallon ration of water per day of water would require 15 trucks working 18 hour days. To provide even 10% of normal drinking water supply would require 240 trucks

The water system itself at the time of the study (2004) lacked emergency power backup, but note that the water buffalo trucks require diesel.  The fuel supply in an extended outage would be very uncertain (as we saw during the damage from Sandy also):

Gas stations become more critical to the citizens of Pittsburgh as a blackout endures. Initially, most people can rely on the gas already in the tank. But over time, the demand for gas will grow, as people will want to leave their homes to procure needed items, or to just “get out.”

There is little incentive, however, for gas station operators to install generators. The probability of a long outage is sufficiently low that the owner will likely not recover the cost of a back-up generator over its lifetime. Thus, if gas stations were to be made more survivable, the government would likely have to step in. For example, is it feasible to designate a few fueling stations around Pittsburgh as “emergency” gas stations and provide incentive to install backup generators?

Pittsburgh is a distribution hub for the fuel system, so there is a lot of fuel there, but it cannot be accessed without electric power:

The second component is the distribution within the city to points of need. We are confident that there are enough trucks to supply fuel to all the critical services outlined in this report. However, the pumps that pump fuel from the large storage tanks are vulnerable to electricity outages. We recommend that this issue be studied further to determine if this dependency is acceptable.

So it sounds like a lot of diesel backup generators would be likely rendered useless for lack of fuel before too long.

The theme of critical infrastructure is in private hands with differing incentives also shows up in the context of grocery stores:

Giant Eagle is the dominant player in the Pittsburgh grocery market, with twelve stores within the city limits. Most have generators to power critical equipment such as emergency lights, but they do not have backup capacity for refrigeration equipment. Pittsburgh has relatively reliable power, and Giant Eagle has decided that large backup is not economically attractive or necessary. On the other hand, Giant Eagle stores in the Cleveland area typically have complete backup capacity, since power there is less reliable.

Thus the picture is that for a power outage of days or less, people have gas in their tanks, food in their cupboards, etc.  But for outages beyond that duration, the means to supply the populace with food, fuel, and water is not survivable and will rapidly degrade, and people will quickly be reduced to "refugee" status - needing either to leave to somewhere better equipped, or to rely on emergency measures like water buffalo trucks and food handouts at central distribution points.

http://earlywarn.blogspot.com/2013/06/sustaining-city-in-long-term-power.html