This morning my neighborhood lost power. Unfortunately, the phones died also.

At about 7:45 this morning, May 3, 2001, my bedroom UPS woke me up. No big deal, the power's out. Then my alarm company called on my cell phone to say that they had lost contact with my alarm panel. "is your phone line dead?" Sure enough, no dial tone. This isn't good.  I take dial tone failures quite seriously. In addition to making house alarms useless, it also effectively shuts down 911 in the area. The local cell site may be out, or at least overloaded.   So started my search for information about phone network power failures.  

In the old days, each telephone line went to each home, each on it's own copper pair of wires. Everything was powered from the huge bank of batteries at the Telephone Company Central Office (CO). These batteries are charged all the time using rectifiers connected to the power company. When the power went out, the batteries could run the whole phone system for hours. But within seconds a huge generator kicks in and starts charging the batteries again. It's just like a huge UPS. That is why the phone network was considered one of the most reliable networks in the world.

Well, things are different now. Over the last 10-15 years, BellSouth has become very dependent on utility power in the field.

I luckily know where the BellSouth SLCs (Subscriber Line Carrier) are in my neighborhood. These are where they change the fiber optic lines from the Central Office into copper pairs to carry the phone lines into your home. They can be big beige cabinets above ground, or like a hatch on a small hill. Below this hatch is an underground Controlled Environment (Air conditioned, dry) Vault (CEV). All of this requires utility power. There is NO standby generator at these sites. That's something I want to change.

Here's a couple CEVs in a local park.  Believe it or not, there's a large room with air conditioning and a lot of high tech equipment just below the ground:

Here's are a couple of the above ground cousin of the CEV, an SLC Cabinet.  No Air conditioning, or sump pumps.  Fans keep everything cool.  Circuit boards are behind the ribbed doors.   You can see the power meter in the middle:

A couple Bell service techs were parked there, so I assumed the situation was well under control. Wrong. I asked them if they knew the phone system was down. No, they were there for other work. After I told them I thought the power outage had caused the SLCs to fail, they went down into the three of them and found alarms all over the place. The batteries that are SUPPOSED to run the equipment for 4 to 8 hours had failed on the racks that had the multiplexers. That meant that all connectivity toward the rest of the network was dead. The other batteries for the equipment toward the homes was working. Apparently, battery failure is quite common on these systems.

After the first service tech made some calls on his cell phone, the parking lot started to fill up with BellSouth vehicles. 10 at one point. There seemed to be some confusion about what to do. After some discussion, they ordered two generator trailers to come to the location. Before they arrived, FPL restored the power.    Just wondering... what would the BellSouth tech do if the cell phones weren't working?   There are no 2-way radios in the service vans.   Is there a copper delivered phone line at each SLC in case the SLC is down?

The Big Picture

Here's why I'm sharing this with you... in case of a large, prolonged power failure, this could be happening all over Miami-Dade County. There must be thousands of these SLCs around just Dade. Does anyone remember the day long power failures all over South Florida when one relay somewhere at FP&L failed? In my case today, the phone failure only effected maybe 1000 homes in West Miami Lakes. A few miles south, it would have effected a hospital. Maybe a 911 center a few miles East. It wouldn't be a bad idea to know where your neighborhood's SLCs are located.  Use the pictures above to find them in your neighborhood.   If you find these within a half mile or so, then there's a very good chance that the only thing between you and a phone line failure is a batter.  You might not be able to do much, but just confirming that the FPL power is out, by looking at the power meter, will help.  If it's an old meter, is the big disk stopped? If it's the type with the LCD display, is the display blank? If either is true, the SLC isn't getting FPL power. It may or may not be on battery power. Call 305-780-2355 (not 611) on your cell phone and report your line as being out, but also mention that the local SLC has no power. That MAY trigger the right people to get to your site faster.   And I guess there's nothing to prevent you from calling the power company yourself.   You'll need the meter number and/or the street address for the SLC.  The address is usually attached to the outside of the SLC enclosure, with reflective orange and black stickers.  

Here's a spec sheet for a CEV: http://www.hartfordconcrete.com/cev.asp Remember, most of it is buried. Just being there to point to the CEV the stopped meter could take an hour or more off of the repair time. Suggest that they order the generator NOW. If it's a huge outage, and the limited generators are already in use, and you know an electrician in your neighborhood, you might be able to work together to set up a temporary feed from a building with a back up generator. Lots of things become possible after a disaster.

Here's a power meter for an older CEV.  The meter is on the right, the connector for the generator on the left :

Here's a newer style.   This one feeds a CEV that has IFITL equipment:

At another SLC failure a few years ago, I actually went down in the CEV with the BellSouth service tech and bypassed a blown out circuit breaker panel with a few extension cords. That got the site going again. That time, even the generator being there didn't help. They didn't have an electrician available, so they needed help understanding what to do when the wiring in the CEV failed. Once they were stumped, they were very willing to accept suggestions from this sidewalk supervisor. One small quirk in the CEV. The AC plugs on the equipment are 20 Amp, not the 15 Amp that we're used to. That's means one of the blades is rotated 90 degrees. Most extension cords can't handle that. But most receptacles can. If you make your own extension cords or quad boxes, use the 20 Amp receptacles. The left slot will look like this: -|

Other issues to consider

As mentioned before, the batteries in a SLC are designed to supply power to the equipment for four hours.   Some sites may last longer.   However, the electronics at the SLC site are only part of the equation.   Air conditioning, blowers, lights and sump pumps operate only on AC power.  So, in a very tightly packed, very active CEV, will the equipment tolerate the higher temperatures until the back up generator arrives?   

Many people wrongly assume that cell phones are a good backup plan in the event of a phone network failure.   Some cell sites are linked with microwave.   However, there is a finite limit to the number of microwave channels that can co-exist in any area.   Many cell sites, especially the newer ones, are linked with Telco supplied T1 circuits, which are also derived from fiber, in the same SLC enclosures that carry your phone line.   SLC down?  Local cell site is down also.   

CLECS, or Competitive Local Exchange Carriers, like SupraTelecom, usually use the same exact facilities as the Incumbent Local Exchange Carrier, BellSouth in our area.   So just switching phone companies doesn't help a bit.   In some cases, the CLEC may have their own facilities, but these are usually available to businesses in high bandwidth circuits, like T1s.   

Disaster planning issues

So, what is the master plan in case of an disaster effecting power?   As it has been explained to me, BellSouth owns a fleet of generators on trailers that will be distributed to each SLC location having a power failure.   Questions I have that I think need review:

Is all this something related to ham radio? Not really. But, to me, its just another piece of information we might need during a communication emergency. More on what to do when all the phones are dead is available in the page about dealing with Phone Outages. 

New as of 12/16/2004:

A very good article about the effect of the hurricanes in Florida.  Notice that the point out the weakness of T1s powered by batteries.   http://mrtmag.com/mag/radio_miracle/

BellSouth employees: Feel free to correct me on any of this. I know you know it better than I do. I want to get all of this on the SFFMA web page tech section for future reference. I'll also add some pictures.   E-mail me here:  Ray J. Vaughan

More info on SLC: http://www.telco.com/products_solutions/WhitePapers/digital/page1.html

Personal Preparation:  If you have nothing but cordless phones at home, or any other type of advanced phone that needs AC power to work, no one can reach you in a power failure. Have at least one good old fashioned phone you can hear hooked up. Could be very important in a 'shelter in place' scenario.   

Here is some information I found on the Internet about SLC power failures.   It would appear that I'm not the only one with these concerns.  

From:  http://www.cctec.com/maillists/nanog/current/msg00606.html

To: nanog@merit.edu (nanog list)
Subject: Re: SLC power failures
From: David Lesher <wb8foz@nrk.com>
Date: Wed, 3 Jan 2001 22:52:21 -0500 (EST)
Delivered-To: nanog-outgoing@merit.edu
In-Reply-To: <20010104033803.14631.cpmta@c004.sfo.cp.net> from "Sean Donelan" at Jan 03, 2001 07:38:03 PM
Reply-To: wb8foz@nrk.com
Sender: owner-nanog@merit.edu

Unnamed Administration sources reported that Sean Donelan said:
>
> As power is being restored throughout the midwestern part of the US,  one interesting feature showed up. A lot of telephone companies are using Subcriber Loop Carriers (SLCs) to provide service. Some phone  companies are using local power sources with limited battery backup for some SLCs. With the extended power outages, central offices with generators maintained service; however, the remote equipment didn't have local generators. This resulted in people losing their local loops and telephone service as the power outages lasted for days.

Yep... The theory is, the RBOC has portable generators to deploy as needed. There's a male plug on each to receive same. That works fine for local failures (truck hits power line..). But in a widespread debacle, say Hurricane Andrew, earthquake, ice storm, etc... they have nowhere near enough, nor can they get them deployed fast enough if they did.

("re: Some" above; I've never seen a generator installation at a SLC...)

Subject: Re: SLC power failures
From: Chris Adams (cmadams@hiwaay.net)
Date: Wed Jan 03 2001 - 23:19:21 EST

Reply: Chris Adams: "Re: SLC power failures"

Once upon a time, Sean Donelan <sean@donelan.com> said:
> As power is being restored throughout the midwestern part of the US, one interesting feature showed up. A lot of telephone companies are using Subscriber Loop Carriers (SLCs) to provide service. Some phone companies are using local power sources with limited battery backup for some SLCs. With the extended power outages, central offices with generators maintained service; however, the remote equipment didn't have local generators. This resulted in people losing their local loops and telephone service as the power outages lasted for days.

I've noticed that all the newer BellSouth SLCs in this area (Huntsville, AL) appear to have natural gas hookups. I have been wondering if this is for some kind of backup power. Anyone else seen this? 

Other reading:

Here's some data about Sources of Failure in the Public Switched Telephone Network

Sources of Failure in the Public Switched Telephone Network, pp. 31-36

Nat'l Academy Press, Trust in Cyberspace (1999), Contents

Comprehensive Communications for Emergency Medical Vehicles

Information about premature battery failures in SLCs due to heat

An outdated FCC report about Telco reliability

Florida PSC

Since posting this page, I've had some feedback I would like to share with you:

5/14/2001

As an engineer involved in the design and installation of this equipment I heartily agree with your analysis. Furthermore cell phone site would overload in such a failure even with backup and the whole communications within an area would co down including maybe fire and police who rely on telephone circuits to control their radio systems.

Manufacturers do provide backup battery for estimated times. Telcos should get at least the minimum required by the manufacturer and this requirement must be enforced by both the FCC and the local public service commission.

<Ray> I know this works on a site by site basis, but I'm afraid that most areas could never handle losing power at all location. I don't see any practical way to deploy thousands of generators in 4 to 8 hours. But I'm willing to bet that the first 48 hours will be spent waiting for the rest of the generators to show up. Maybe I'm just a pessimist, but I just don't see how there can be enough generators ready to go. And I wouldn't put money on the extra ones you're planning on will actually be there. Highest bidder gets the power.

<Ray> Do you keep the required special connectors and cables on-hand for these third party generators?

Sure do.  Usually standard electrical supply depending on current requirements but the site must have its backup batteries in place and working and checked regularly. 4-8 hours is a good time usually more is not necessary and a backup generator could get to a site before the batteries go dead You would not need 1 for generators as outages usually are in a small area and most COs only have 2 or 3 remote sites as you show in your web page

Robert Schoenfeld

05/14/2001

Hi Ray, 

Your page is very interesting and hits home on quite a few notes. I work for a small ILEC in the North East. We see our share of power failures due to both natural and man made events. In use in our network are OPM's (Off Premises Modules) which are very similar to SLiC's. They consist of a pad mounted box (no vaults ) with equipment and batteries contained. We have a battery replacement schedule that allows all of these OPM's to have at least 8 hours of standby time under full load. In addition, we have enough portable generators to cover about 50% of these units ready to go at all times. We also have commitments from various local vendors to supply more as needed in an emergency situation. A cross section of other outside employees have been trained in deployment and operations of these units. We have letters of cooperation with electric companies serving our areas making the PSTN a priority in restoration. The equipment housed in buildings is being backed up by a combination of auto-start and manual generators. These buildings also have batteries with larger capacities than we can fit in the OPM's that can last in some cases several days. During the Y2K make ready, we estimated we could run the current backup system for about 2 weeks with out any outside help (except for gasoline) and months if necessary with outside help. I hope this shows you not all phone companies take this matter lightly. Maybe being as small as we are makes it easier logistically to deal with problems. Please feel free to contact me if you have any questions.

Name withheld

An Australian Communications Authority (ACA) fact sheet about power and telephones.  Almost all of this information would apply in the US.   

http://www.aca.gov.au/consumer/fsheets/consumer/fsc46.htm

BellSouth explains the need for local power for ISDN circuits:

Does ISDN need its own power back-up-

A simple explanation about how regular phones might work in a power failure.

Why does the phone still work when the electricity goes out?

Have anything you would like to add to this web page?   E-mail me, Ray J. Vaughan  Please let me know if I can add your information to this web page, and if I should use your name or not.

Thanks.