This puts me in mind of the journalism of Greg Palast, who documented in “Vultures Picnic” that the backup generators in nuclear power plants are very fragile. They are designed after the model of ship power plants, huge pieces of tech which cannot be started quickly. So when a backup generator is put in service in the case of a nuclear emergency, it cracks the crankshaft in half. Don’t believe it? I admit it is unbelievable. Check the source yourself, I’m not here to do your research.

Backup generators and automated switches represent mature equipment. A lot of sites have them and they work very well. The problem is that the hospitals don’t consider the equipment key in their mission so they neglect them. They should be cycled and tested at least once a month on load. Not doing that is stupid if not criminal for hospitals. They check on all sorts of medical equipment and put stickers on them. I hope they are doing a better job at that than with the generators.

Our generators are even used by the local utility in a program that kicks in when there is high load on the grid such as high Summer. They can turn them on remotely. The problem with these at the hospitals is that they don’t take the systems seriously and then they crap out when they have an emergency.

We rarely learn from the mistakes of others.  In 2001 Tropical Storm Allison parked over Houston and did as much damage, or more, than several hurricanes before or since (stalled storm = tons of rain fall, over 24 inches in some parts of the city, couldn’t drive around the city for at least two days).  One of the major issues: backup generators in hospital basements.  More than one hospital had a high profile evacuation of patients carried on gurneys down stairs by staff - some hand pumping oxygen.  This wasn’t last year or two years ago - this was 11 years ago!

Now watch the blame game begin.  More here:  http://runningahospital.blogspot.com/2012/10/who-are-they-going-to-blame.html

Generators that are just cycled occassionly and that are not run under a loaded condition can cause the Generator’s fuel valves to be what is called “Wet Stacking”. Where the valves stay saturated with fuel and after time dry and clogg or restrict the fuel injectors so when the Generator is required to run at full speed and under loaded condition The Generator may Fail.

Unfortunately, there’s no way to test a hospital generator thoroughly without basically ‘pulling the plug’ on incoming power.  The monthly and annual tests do the best they can, but they aren’t perfect. 

And if anyone really believes that these things should be perfect, the last perfect thing I heard of walked on water, and we aren’t it.  These are systems designed and run by humans - we all do the best we can and pray it’s enough. 

Many hospitals are running in the red, and moving generators up several floors in an existing hospital is a $1M+ project.  Add the need to remove patients from those floors during the construction and then add the disruption to the surrounding neighborhood during this process, well, it gets complicated and ugly, fast.

Haviing a workable electric backup system is not rocket science .. and when you have human lives at risk using terms like generators were compliant is a lame excuse when dealing with terminally sick people. Instead of spending billions of dollars on Obama care we should first make sure we can keep the lights on in our hopitals.. We seem to be regressing to a 3rd world standard of compliancy.. Wake up you pin heads..

Even more recently in 2011, we saw how critical cooling water pumps at Fukishima failed because they were located below the flood water line.

Basements are not ideal locations for any backup equipment.
Actually having them work when needed is another aspect…especially if rarely used for long periods of time.

Wonder how much it costs for a major metropolitan hospital to close and be forced to evacuate numerous patients vs the cost of relocating critical backup equipment. Which bottom line is being scrutinized?

My company used to make custom fuel tanks for generators and the company we made them for begged me to make sure there was no welding slag or grit in the tanks. We had viewing ports to blow out the residue on the tanks and also vacume them clean. I was told that when a new generator was run the filters should be replaced every so many hours until they were sure the tanks were clean.  Most tanks are mass produced and the cleaning is not complete.  If they are not clean when they have an emergency they will only run until the filter clogs up and then stop running.  Clean tanks are a must

Another equally life-critical hospital infrastructure system that is often take for granted is the medical gas system.  The systems provide piped medical gases - Oxygen, Medical Grade Air, Nitrous Oxide and Surgical Vacuum, and are supposed to be maintained and inspected to rigorous standards since these gases are directly supplied to patients.  They are often misunderstood and neglected by hospital physical plant departments.

I have a doctorate in education from Harvard and I grew up in Miami. I’ve survived many hurricanes. I founded a non profit named Be Ready Bear (copyright 2008) which uses a friendly cartoon bear to educate adults and children about disaster preparedness. Educating about disaster is harder than educating about anything else. Racism, gender equity, homophobia, smoking, alcoholism - I’ve never before encountered more resistance! I need more help with this project. Be Ready Bear puppets can be dressed in scrubs and tend a toy generator. Believe me, its the most effective approach!

I hold bachelors degrees in electrical and mechanical engineering.  Generators must be started, and run for a half hour or so, every month or so, not only to access their functionality, but to extend their lifespans.  The periodic operations lubricates the engine, dries out the lubricating oil, cleans old fuel out of the fuel system, dries the externals of the equipment, turns the bearings, charges batteries.  ect.  I’d also recommend they run the engine an once a week, until the oil temp got to operating temperatures, since operation is so critical.  It should go without saying, there should be an electrical load applied, so that that part of the system is tested too.
  These hospitals have the engine-alternator set anyway, that is the bit expense.  Having someone in maintenance, start, and run the equipment, and the cost of the few gallons of fuel is small.  In fact, doing this lowers the cost of maintenance, since it will decrease the numbers of repairs.  Imagine the costs of the wrongful death suits.

@ Mike W

That would explain why there has NEVER been a incident at a nuclear power plant caused by a failure of its diesel backups during a loss of offsite power event.

There are thousands of emergency diesel generators operating today in the affected region and around the county at hospitals and other critical facilities.  You won’t hear about those because these units are working exactly as intended. 

However, as this story points out, there can be failures. and there is an important distinction to be made here in the notion of “generator failure.”

From what we’re seeing and hearing from the NY Region– most reported failures in NY/NJ are due to flooding of the generator rooms—as opposed to the mechanical or system failure of the units to start and operate properly (the other kind of failure). 

In this regard, Dr. Kellermann’s comment in this story was spot on.

The other situation that is also common in the high-rise world of NYC- the primary fuel supply and associated pumps are located at ground level/below grade and the generators are located at rooftop or higher floor, and the fuel supply is pumped to the generators.

When the main fuel supply room and the associated pumps go underwater, the generator units on the roof will not be able to function long before their smaller supply tank is depleted

Other commenters generally have it right here—exercising the units on a regular weekly basis, along with proper servicing and test operating under load conditions a few times a year—assures reaediness.
Design and specification of units, their location and assuring necessary redundancy and of couse good maintenance all contribute to systems doing their job under any cicumstances. 

Those are no doubt issues we’ll be hearing more about in days to come.

http://www.dieselforum.org

Another issue during and after Katrina is the depletion of on-site fuel stores. Working in the boiler industry, I see the lack of adequate fuel supplies at all the hotels and hospitals. None of these have more than two or at most three day supply of fuel. One university has a gas turbine co-gen unit which can carry most of the campus—but it burns five gallons a minute, 300 gallons an hour, 7200 gallons every 24 hours!! With a 60 hour reserve this one place needs at least 14,000 gallons every other day to keep going. How is this much fuel to be delivered in the middle of an on going crisis?? It can’t. Many places in New Orleans still have their engineering areas 35 feet below sea level!!! One place we work has been flooded “Katrina like” twice due to broken water mains in the street.

Just testing systems when the weather is nice etc. is not adequate preparation for emergencies
There needs to be analysis for every major component in a system, and what it can be affected by.
Yes the fuel tanks, the fail-over switches (They were in the basement ay NYU?), proximity to roads and loading docks. Etc.
Failure analysis is a profession, but rarely a requirement for builders and contractors.
Without explicit requirements the lowest bidder wins.

There is an old tale of an emergency generator, in a basement, well protected against flooding, with the electrical switchgear also well protected. When the power failed, the generator kicked in immediately, with power supplied to all essential circuits. Unfortunately someone decided that the sump pump was not essential. It cut out and the inevitable leaks gradually flooded the generator. Moral: stupidity trumps everything.

If the generators and their maintenance complied with the regulations, the regulations were inadequate. Why were the regulations not changed? Grandfathering to allow inadequate installations to remain is no solution (although modifying installations to safeguard the generators and switchgear might be enough). And what happened to “good engineering practice”?

See, there’s a conceptual problem, here:  A generator, like anything, is something you only notice—only know if it’ll do the job—seconds after you need it.

Yes, tests are great at picking up some issues, and I don’t know (and probably won’t learn) what happened to these generators.  But still, no test (as Paul points out above) can ever show the performance under the required, persistent demand except to run the backup live as the primary system, which would be a huge waste and probably end up endangering patients routinely.

@Mike H
Sorry, you’ve not done your homework. Check the source I mentioned. Why on earth would you make a claim like this with zero evidence to support it?

Hospitals will test the generators as soon as they can find someone to bill for the time.

I don’t think people know what actually happened here? I keep reading alot of stuff about generators in the basement and such. NYU Langone had all of it’s systems on the roof(generators, transfer switches, distribution switchgear). this was done several years ago. Although one thing that needs to happen is the main fuel tanks have to be on the ground floor, no other way to store large quantities of fuel. The fuel then goes through pumps up to the roof and stored in day tanks which provide a minimum 2 hour run time per generator. the fuel pump room is in a flood resistant room in the basement. When the forcefull flood of water entered the basement it made it’s way into the fuel pump room and tripped a safety feature meant to shut off the fuel pumps in case of a fuel spill. the pumps shut off and stopped delivering fuel at this point. the generators ran until they no longer had fuel in the day tank and shut off. by the time maintanance staff made there way to the fuel pump room to reset the fuel pump controller, they had already made the decision to evacuate the hospital.

There are three other caveats for hospitals to ponder; 1) everyone thinks their special piece of equipment is vital and essential, and they will find ways to connect it to the emergency power circuit. When the crunch comes and the emergency generator tries to spin up, the number of KWhr may be so much more than the designed operating specs when the system was put in (which likely pre-dated the PC explosion) that either the generator fails of the time-to-full-power is stretched out to the point where connected gear operates abnormally. 2) every piece of new hospital equipment that requires electricity to work should be vetted prior to purchase to determine whether transient loss of power results in return to the settings prior to power loss, return to function using “default” settings or return to idle/awaiting setting input. Surprisingly different manufacturers of medical gear have not adopted a common convention on this. Finally 3) if your institution’s engineering staff contracts out periodic maintenance of the emergency generators they need to check whether the generators are left in “auto-on-with-line power loss” mode, “require manual start if line power is lost”, or “off”. I personally had to assist the dialysis staff return the blood of two dialysis patients using cellphone backlighting because the Engineering staff at our hospital failed to verify that the emergency generators were left in “auto-on” after PM.

Load testing for prolonged periods of time is the best way to insure that these systems will actually work in a real emergency. Case in point: I had a facility that had the generator on the OR installed and tested as required over a 7 year period. Then there was an actual power interruption after a tornado. After about 5 hours of run time a low fuel alarm occurred. We found that the transfer pump from the main fuel tank to the belly tank on the generator set was not connected to an emergency power source. In the past when testing occurred the fuel used was replaced after the generator test period when city power was restored. The electrical contractor wired the unit according to the plans. There was an error on the plans. We were able to temporarily correct the problem before we ran out of fuel. We promptly corrected this once power was restored.

Imagine if Sandy happened when the NYU medical center prepared and cooking the food for 5 NYC hospitals including NYU medical center, Mount Sanai Hospital, NYU hospital for joint diseases, NYU downtown and NYU queens located in Astoria. 19500 meals a day
Who will do the work today to feed the patients of the oders hospital?
more centralisation is good in economics books not in reality and especialy in crisis

I see a lot of comments about load testing.  But if the NYU generators failed because of salt water infiltration into the fuel supply room how would this have been tested for?