A 7-Point Checklist to Prevent Routing Outage and Secure Network Connectivity

Network outages continue to be a major challenge, disrupting businesses, individual lives, and major communication platforms. The significant impact of such disruptions is highlighted by one such incident, the Australian telecom outage.

The widespread disruption caused by the outage lasted for several hours, affecting Australian businesses, critical services, and everyday lives.

This example showcases the intricate structure of contemporary telecommunications systems and the possibility of interruptions taking place.

Even if the infrastructure is highly advanced and redundancy measures are strong, unexpected events like software malfunctions, hardware issues, or natural disasters can cause network outages.

Network interruptions can occur unexpectedly. So, let’s explore the reasons behind these disruptions and ways to protect your network from them.

Behind the Australian Outage: A Closer Look

The outage was caused by a combination of technical issues, mainly related to a software upgrade and the large amount of routing information it brought.

• Too much routing data disrupts BGP stability – The outage was caused by modifications made during a regular software update. These modifications accidentally disconnected a central router, leading to an overflow of routing information in the telecom network. This overflow caused instability in the BGP.
• Overloaded routers and safety limits – The significant burden on crucial routers in the telecom provider’s network was caused by the routing issue. These routers were overloaded and surpassed the predetermined safety limits as they handled the extensive routing data. These limits establish the limits within which the network’s routers can handle routing data.
• Router Default Settings and Protections – In light of the safety thresholds being surpassed, approximately 90 impacted provider edge (PE) routers implemented a default protective measure from the vendor, leading to their disconnection from the telecom provider’s IP core network. This isolation method effectively cut off the routers’ capability to engage in routing data, leading to a disruption in network connectivity.
• Cascading failure affects the entire network infrastructure – The failure of these essential routers, especially those in charge of core network routing, led to a chain reaction of problems that resulted in extensive disruption throughout the telecom infrastructure.

What prolongs network downtime?

Restoring extensive network outages can be a complicated and time-consuming task. Key elements that can worsen incidents such as the Australian telecom outage and extend the recovery process consist of:

• Lack of resilience: In order to avoid overwhelming the routers with a large amount of routing information, networks must have adequate protections in place.
• Lack of proper monitoring: Without efficient network monitoring systems to promptly identify the problem, network administrators may experience delays in pinpointing the underlying cause and implementing necessary corrective solutions.
• Manual repair: Restoring the impacted routers without configuration management tools may necessitate manual reconfiguration, resulting in a time-consuming and labor-intensive procedure.

7 essential practices to prevent network from outage incidents

While it is unfortunate that network outages occur, there are measures that both individuals and organizations can implement to reduce their consequences. Here are seven important factors to keep in mind:

1) Implement a reliable system for monitoring networks:

A holistic network monitoring system offers complete visibility and management of your network infrastructure. It allows you to track network performance, detect possible problems, and address them quickly.

2) Establish well-defined procedures for configuration management:

This involves version control, change management, and documentation. Properly managing configurations is crucial to prevent unauthorized changes and uphold consistency throughout the network.

You need to be aware of the pre-set configurations of your routers from the vendor and take necessary steps to prevent any problems when updates are implemented in your network setup.

For example, network administrators can establish compliance rules in ManageEngine Network Configuration Manager to guarantee that if the maximum prefix configuration (i.e., safety threshold) is exceeded, only a warning message is generated instead of isolating the router entirely.

3) Traffic engineering and capacity planning:

Use traffic engineering methods to control network traffic efficiently and guarantee routers are capable of handling high volumes and sudden increases in data traffic.

This includes examining traffic flow, pinpointing possible areas of congestion, and implementing measures to manage congestion.

Conducting capacity planning exercises is essential to guarantee that the network infrastructure can accommodate projected growth and traffic requirements.

4) Implement a thorough plan for backing up and recovering data:

This guarantees that you can rapidly bring back your network to a functional state in case of an outage or disaster. This plan should involve backing up important data regularly, establishing protocols for restoring network settings and automation, and implementing a method for testing your recovery processes.

5) BGP configuration and troubleshooting:

Implement strict configuration management procedures for BGP to guarantee correct route redistribution, prevent loops, and filter communities effectively. Stay informed about the latest BGP vulnerabilities and apply the necessary measures to prevent routing attacks.

6) Redundant network setup:

Design and implement a robust, redundant network infrastructure that includes multiple core routers. This will significantly enhance network resilience, minimizing the impact of potential failures and ensuring rapid recovery in the event of outages.

This involves having backup systems at the device, link, and path levels to maintain connectivity even during hardware or network outages. Network admins should utilize multiple, independent carriers for network management and communication to reduce the risk of service disruptions.

7) Conduct routine network evaluations and vulnerability scans:

Regular network assessments and vulnerability scans conducted on a routine basis can uncover any weaknesses or vulnerabilities in your network infrastructure that may be targeted by malicious attackers or result in unintended disruptions. These evaluations must guarantee that both the physical and logical security aspects of your network are addressed.

Final Words

Even top-performing networks can experience routing and configuration problems, as demonstrated by the recent Australian telecom outage.

It is crucial for businesses to strengthen their network infrastructure in order to protect against vulnerabilities present in modern network infrastructures.

This can be achieved via a thorough network monitoring system, well-defined configuration management procedures, traffic engineering, and capacity planning.

One effective way to improve network resilience and reduce risks is by using ManageEngine OpManager Plus. Ensure continual connectivity and rapid recovery from unforeseen obstacles. Contact our product specialists for a brief demonstration of capabilities today.

Author Name: Sandhya Saravanan

About the Author: Sandhya Saravanan is a Product Marketer at ManageEngine. She creates user-friendly content that drives awareness around advanced network monitoring, observability, and AIOps. Beyond work, she’s an art enthusiast and volunteers at a non-governmental organization.