Once upon a time, a large teleport operator was hit by a major power outage. Uninterruptible power supplies immediately kicked in and the NOC staff waited for the generator to start. Nothing happened. So, they switched to the backup generator. Still nothing. Work crews rushed to trouble-shoot the generator failures while the countdown timer on the UPS batteries ticked downward…and downward…and still downward. Just minutes from total loss of power, the work crews got one of the generators started and customers suffered no loss of service.
That did not stop the operator for buying two more generators to serve as triple and quadruple backups.
Service continuity is the core deliverable to customers. Most other aspects of service can be managed, explained or finessed, but not whether the service is off instead of on, severely degraded rather than healthy. Not surprisingly, it is the issue to which teleport engineers and managers devote most of their time, energy and money.
And yet, as the story shows, it is always the hidden issue, the weak link left unnoticed, that threatens the continuity of service, even at well-run facilities with proper procedures in place. In 2015, WTA introduced its Teleport Certification program to help operators find the strengths and weaknesses in their facilities, technology and procedures and to receive independent, standards-based certification at one of four Tiers, from Tier 4 at the top to Tier 1. To date, WTA has certified 46 teleports operating in the US, the UK, France, Germany, Italy, Norway, Cyprus, Switzerland, the UAE, Australia, Hong Kong, Singapore, Mexico, Bulgaria, Colombia and Brazil.
Teleport Certification has also generated substantial data on teleport operators from its 270-question survey instrument. In a recent report, we looked at what the data reveals about the most common under-appreciated issues that can have an impact on service continuity, and how teleport operators can best resolve them.
Why Service Continuity?
It is fundamental, for one thing: the most basic value delivered by a telecommunications service provider. Of the 270 questions in the survey instrument, nearly 20% are about service continuity and many more cover topics with an impact on it. But it is also an area, particularly where operating procedures are concerned, where our sample of teleport operators showed some subtle but meaningful weaknesses.
Finding the Weak Links
Much of it comes down to the procedures that govern management of service continuity. The best operators make sure that their approach is based on accepted international standards, so that they get the advantage of the best thinking on the planet. They take a hard look at the continuity scenarios in their planning and make sure they align with the real risks. And they test relentlessly: the continuity scenarios and the automated failure and recovery systems that are supposed to prevent loss of service.
They also keep a sharp eye on redundancy. Deciding how much redundancy is needed in every part of the operation is a judgment call that balances probability and expense. The research showed that combiners, LNBs and site diversity are areas where even well-managed facilities come up short. Redundancy becomes especially important as more teleports are controlled by remote NOCS. Communication between the NOC and teleport can be a critical point of failure. While saving money through centralization, the best operators make sure they have enough redundancy that they will not regret the decision to centralize.
Every service provider must make decisions about how bullet-proof to make their facilities, infrastructure and facilities. The Teleport Certification program (www.worldteleport.org/certification) helps to make these decisions visible, so that operators can make better-informed decisions and deliver the price-performance that their customers expect.
-------------------------------
Robert Bell is Executive Director of the World Teleport Association, which represents the world’s most innovative teleport operators, carriers and technology providers in 46 nations. He can be reached at rbell@worldteleport.org. High-Performance: Service Continuity is available free to members and for sale to non-members at https://www.worldteleport.org/store/default.aspx?.
