What Really Happened During the 2003 Blackout?

🏭 Watch more of my power grid videos here: https://www.youtube.com/watch?v=v1BMWczn7JM&list=PLTZM4MrZKfW-ftqKGSbO-DwDiOGqNmq53&index=2 📫 Big announcement coming in March! Stay up to date here: practical.engineering/email-list

The power plant in the very first frame, next to the United Nations building in Manhattan, that's where I was... in the control room as a very young Engineer. I was actually giving a tour of our control room and flipping through some system graphics, when all 3 units tripped offline at nearly the same time, and the plant went completely dark, which before then was thought to be impossible. One of the operators actually yelled at me because he thought it was my fault, lol... I was relieved when the system operator called a few minutes later, and told us to, "Standby, we just lost the entire Northeast." We overcame a lot of challenges that evening, and for a young Engineer, it was exciting. I slept on the roof that night because it was too hot inside. I remember looking up at the night sky and seeing the stars, and thinking... when was the last time anyone in Manhattan looked up at the night sky and saw the stars like this? Great job on the video, and thanks for bringing back such great memories.

Excellent summary! I'm a retired power engineer and am familiar with that final report. There were some additional aspects of this event that your viewers might find interesting: - Utilities such as First Energy maintain a regular program of tree trimming to maintain adequate clearance below overhead transmission lines. These programs typically operate on a five-year cycle, and 2003 was the fifth year of the cycle. In fact, a team of arborists working in the area witnessed one of the tree strikes. - The manufacturer of the First Energy control system had warned that there was an alarm buffer problem in their system. However, rather than implementing the fix recommended by that supplier, they opted instead to purchase a new system from a different supplier. The replacement system was supposed to have been installed by the time of the event, but that second supplier encountered delivery delays. - The MISO state estimator was taken off line in the morning to fix the input problem. After the problem was fixed, the IT technicians who had been working on it went to lunch - but they forgot to put it back on line until after they returned - which was too late. In the 1990's. there was a trend to deregulate the electric utility industry in the US, and most of the northeastern part of the US had advanced pretty far in that direction when this event occurred. So naturally, one of the first questions asked was whether the event had been caused by deregulation. The traditional regulated monopoly model for electric utilities included an 'obligation to serve' stipulation that translated into an intense focus on reliability. Translating the vague notion of grid reliability into something that can be addressed by competitive markets is difficult, and the local grid operator, MISO, had only commenced operations a few months before this event. So while the official answer was that this event was unrelated to the move to deregulation, I think it influenced some of the factors that contributed to this event Finally, this event demonstrated very dramatically two key points about the electrical grid. First, if failures occur, they are the result of coincident occurrence of multiple factors and not one single cause. That makes it very difficult for system planners to anticipate every possible combination of factors that could lead to a potential outage, especially over long periods of time. Second, if perfect reliability is not possible (and it isn't), then the system must be resilient. That means that system protection must engage soon enough and react in a way to prevent damage to system components so that the system can be quickly restarted.

Damn...I just remembered how amazing that outage was. A bunch of us kids in the neighborhood had sleepovers, people were just going to each other's houses. We were playing in the streets. There was such a strange and unique coming together of humans where I was. As soon as the power came back, people started to go on about their usual business and that special moment was gone

The power went out above my head as I was boarding a plane in the Toronto airport. Ended up sitting on the tarmac for two hours, not knowing what the cause of the delay was. We were shuffled off the plane and I, a thirteen year old about to embark on their first solo journey, was left alone in an airport. I was too young for a cell phone at that time and the pay phones weren't always working. Managed to get a hold of my parents who brought me home. I can remember the sweltering heat. My elderly dog ended up passing away that night because of this heat. No vets were open because of the blackout. We didn't get power back in my house until 7am the next morning. It was...definitely a bad day for me.

This happened when I was 4 and I thought I caused it by watching too much TV. I've lived with the guilt for many years, thank you for explaining what really happened and for clearing my conscience.

When you learn about the complexity of managing the power grid, it feels like a miracle that we consumers are actually allowed to do whatever we want and use as much power in whatever way we want in our homes. Something to be thankful for, since all the heavy lifting and problem solving is being done by someone else on our behalf.

I loved this day. I lived beside a valley with a river. After so many hours, and seeing that everything was down, i set up a campsite beside the river. Will never forget how bright the stars were. I was actually sad when the city glow returned.

I was not even 10 at the time. My dad had just walked in the door with 2 weeks worth of groceries when it tripped. I can still remember his “You have GOT to be kidding me!” As we hoped nothing would go to waste.

The Lift Pump Station located near Ludington, Michigan helped save the entire eastern half of the US and Canadian power grids. This facility maintained grid frequency by absorbing most of the nearly 5 GW load imbalance as the disconnects around Lake Erie engaged. The pump station filled itself to the brim, well past its safe design water surface elevation, then ran itself nearly dry, again outside of safe operating levels and at great risk of causing cavitation in the turbines to provide power afterwards. These actions helped prevent catastrophic damage cascading all the way to Denver and gave some time for other stations to reconnect. If a power station is scrammed, a rapid shutdown, and depending on its type it can take days if not weeks to get a power station back up and running. A pump station can provide nearly immediate power generation. Thank you to all of the station operators, line workers and service crews who struggled on that day, your actions and quick thinking, kept the lights on east of the Rockies and prevented a far darker outcome. Both the 2003 and the Texas outages could have been much worse if the load imbalances had been allowed to progress a few minutes or even seconds longer. The control systems in place now are night and day more robust than what was present in 2003.

I was a kid in Brooklyn during this blackout. It was one of the most memorable nights of my life because we got a full night sky due to no light pollution. You usually never see more than 2-3 stars at night in my neighborhood.

I remember this blackout in Jersey City and NYC. Surprisingly, the blackout took so many people by surprise that there were zero reports of crime because everything came to a standstill. Instead, everyone tried to help each other when everyone tried to figure out what happened.

For additional context to anyone who isn’t hip to this story. The companies who were responsible to manage things (FirstEnergy and others) also had plenty of shady activities going on internally. So as you follow along with this wonderful walkthrough, keep in the back of your head the vision of your neighbor who has that old lawnmower that barely works. But it works so he won’t replace it even though he spends 5 hours every time just to get it running. Apply that exact mental visual to the scale of a company that is responsible for managing the grids as you listen to him talk about how these tiny pockets of problematic issues slowly domino effect

I was working at a 2000 MW power plant that day. I recall hearing air blast breakers operating frantically in the switchyard just after 4 pm. Lights blinked and then moments later the plant was black. From my office, I looked across the river and saw two other large power plants also fully tripped. Our plant generators were spinning, but there was no grid to connect to. We hoped to keep the units ready to resynch, but we could not keep up with demin water consumption, and one by one, lost the plant. The grid was partially restored by the next day when we were able to fire our units and resynch, thus rebuilding the grid. Certainly a day etched in my mind!

I'm not gonna lie, as someone who was in NYC during this, Manhattan specifically, it was actually really fun. Gas service was still going so restaurants were cooking and effectively giving away all the food they had built up, everyone was out on the street ( who wasn't trudging across a bridge) was drinking. Driving...yes driving...was actually really easy because traffic was self moderating. It was a very surreal experience, all told. Spent the day after that first night out in Brooklyn by the water drinking with friends and then hung out at my friend's place on the west side and watched them switch the super blocks back on one by one. Looking up 8th avenue, it was kind of amazing. From the horizon you could see entire sections of Manhattan switch on, I wanna say like 10 block squares at a time coming towards where we were down in the 30's.

I remember this day very very well and I was so proud of my friends group. We had a contingency plan that if anything ever happened to the point where it was a panicky situation that required everybody to meet up at one place they would all meet at my place. I want to go pick up my friend Lindsey because she was the only one without a car and when I came home there was literally about 20 of them just waiting outside. They had generators and gas and propane generators and they brought their grills and their coolers and we started just cooking up all the food that was going to be going bad eventually and we basically had a three day long party. When we were on the third day we started siphoning gas from our cars so that we can continue to power our generators so that we can keep people's medications cold that needed it. And ever since then we all now use propane generators and we usually have around six propane tanks, each, at all times.

I remember this quite well. I was working in Solon, OH at the time, and the power tripped off at around 4:00. Some phone calls from our facilities manager confirmed it wasn't just us, but pretty much everyone in NE Ohio. They sent us home since we couldn't do much except twiddle our thumbs and it was clear it wasn't coming back on anytime soon. Got home, pulled the rope to open my garage door by hand, turned on the camp lantern for light when it got dark, and kept listening to the battery powered radio for news. Lit my gas stove with a lighter and made dinner, and read a book by candlelight. Quite peaceful, actually. When I heard that water pressure was going down I filled my bathtub so I had some water. Power came back on around 10 PM. Next day, I found out this was huge problem for lots of folks. I didn't have a generator back then, but it was certainly eye opening.

When first learning grid operations it was explained to me thus: Imagine the ceiling in a room has a wide variety of hooks embedded. Each hook can only support a different weight and will pull out if overstressed. Now imagine you have a wide variety of weights that you must suspend from these hooks. You attach them with rubber bands of varying strengths. You link a bunch of hooks and weights with bands. But if one hook pulls out, the weights stretch rubber bands and shift and you need to keep things suspended. Now, plan things out so you never drop a weight, regardless if a hook pulls out, or a band breaks.

My friends in Toronto recounted stories of how traffic during rush-hour became total chaos. There were several people standing in the middle of intersections marshalling traffic for hours on end. I've always wondered who relieved these samaritans when they became tired and needed to go home.

Moral of this story, every time you flip the light switch on and the lightbulb illuminates, appreciate the hundreds of engineers and operators that make that little thing possible!