How do you navigate a scenario where external vendors are causing network disruptions?

Know your network, inside and out. No vendor will know and understand your network better than you. It is why you are there, even though you may give your vendor an entire floor in your building. They are there to maximize their invoices toward you. They are not there to care as much as you do.

A good security practice is network segmentation, this can help to limit the extent to which network intrusion can affect the network at large. If a vendor causes issues in your network only where he has access to, the issue remains there and doesn’t spill to the other parts of the network

A utiliza??o de ferramentas, como firewall, Mikrotik e segmenta??o de rede por VLAN, ajuda a combater essa invas?o de fornecedores na rede, porém, a seguran?a n?o se torna efetiva sem as boas práticas de seguran?a por parte dos usuários.

Handling network disruptions caused by external vendors can be challenging, but here are some steps to navigate such scenarios: Assess the Impact: - Determine the severity of the disruption. Is it affecting critical services or just inconveniencing users? - Identify which vendor(s) are involved. Communication: - Notify relevant stakeholders, including management, IT teams, and affected users. - Contact the vendor(s) promptly. Clearly describe the issue and its impact. Isolate the Issue: - Use network monitoring tools to pinpoint the affected area. - Check if other internal components (servers, switches, etc.) are functioning correctly. Maintaining open communication and a collaborative approach with vendors is crucial.

Opening TAC cases is one of the most painful experiences an engineer can endure, especially if you are more experienced than the TAC engineer (which happens more often than people realize). If you can communicate your issue via your SE, but keeping the TAC team in the loop, it will help, because the SE can reduce the initial friction at the start of every TAC case.

Along with effective communication, I’ve found that establishing a clear escalation path within the vendor’s organization is crucial. Knowing who to contact at different levels of urgency can speed up the resolution process. Additionally, maintaining a communication log, including timestamps and details of interactions, helps ensure accountability and provides a reference for future discussions.

Diálogo é talvez a melhor forma de entender o problema e resolvê-lo. Um vez avaliada a situa??o, pode-se aplicar as práticas de seguran?a na rede em contribui??o ao fornecedor.

There's more than one kind of vendor and each can cause outages of varying severity. Know which vendor-provided services will cause the most harm to your company when they go down. If possible, have a fail-over service present. Plan in advance for outages. When one does happen, and it will, keep your fellow employees and customers informed about what is happening and when they can expect the interruption to be resolved.

It's always best to have a regular vendor communication including in person visits even when everything is running smoothly in order to build a personal relationship and to make sure the vendor or his service team will be there when needed.

I’m not sure that In-House mitigation should be the third step. I think it should either be step 2 or a part of step one after the initial analysis and should be done before contacting the vendor. This will mitigate the impact to services that rely on this connection and drastically shorten the outage time (albeit internally). After the traffic has been rerouted away from the impacted circuit, then the vendor should be contacted.

We can block external interfaces which accept traffic to standard ports using efficient security policy implementation, also use security policy groups to detect vulnerabilities. Rerouting of traffic can be done as well.

En mi experiencia, he enfrentado interrupciones en la red causadas por proveedores externos varias veces. Utilizamos herramientas de monitoreo avanzadas para detectar problemas y actuar de inmediato. Desarrollamos un plan de contingencia con rutas alternativas y sistemas redundantes, crucial para mantener la continuidad del servicio. Designamos un equipo especializado en resolver problemas de red rápidamente y mantenemos registros detallados de todas las interrupciones. Informamos a los usuarios sobre el estado de la red y las medidas correctivas. Revisamos y mejoramos continuamente nuestros procesos e infraestructura. Estos pasos nos han permitido estar mejor preparados y minimizar el impacto de las interrupciones en nuestra empresa.

Internal mitigation strategies play a crucial role in minimizing the impact of network disruptions caused by external vendors. Traffic Redirection: If possible, reroute critical traffic through alternative paths or backup connections. Prioritize essential services to ensure continuity. Failover Systems: Implement failover mechanisms for critical components (e.g., redundant servers, load balancers). Test failover regularly to ensure seamless transitions. Workarounds: Deploy temporary solutions while the vendor resolves the issue. Communicate these workarounds to affected users promptly. Emergency Protocols: Train your staff on emergency procedures. Ensure everyone knows their roles during disruptions.

Be extremely conversant with the technology and/or protocol being implemented by your routing/switching vendor. That way, you quickly understand whether the failure is due to an incorrect feature implementation by the vendor, or a misconfiguration by you, the operator. Remember that the people that write the code for the vendor do not always understand networking, even if they work for a networking vendor.

Unless you are a very large operator, it is unlikely a vendor will compensate you for a bug-related outage. I mean, that is why you are paying for support... because the equipment and software are developed and offered with no performance guarantees. That said, do maintain pristine correspondence with the vendor about the issue, because it will be handy when colleagues and customers ask for an RFO (Reason for Outage).

Documentation review is critical to dealing with network outages caused by external vendors. 1. **Monitoring and Alerts:** - Set up alerts to monitor supplier performance in real time. - Be aware of the key performance indicators (KPIs) defined in the SLAs. 2. **Proactive Communication:** - Maintain regular communication with the supplier. - Inform them of any problems or concerns immediately. 3. **Risk and Impact Assessment:** - Assess the impact of the disruption on your business. - Consider financial and operational risks. 4. **Detailed Documentation:** - Record all communications, agreements and actions taken. Remember, transparency and proper documentation are essential to effectively resolving outages.

Reviewing documentation should also include an evaluation of the vendor's historical performance and reliability. Tracking past incidents and their resolutions can provide valuable context and help with strategic planning. Additionally, understanding the financial implications of downtime and having a clear strategy for cost recovery can strengthen your position in discussions with the vendor.

All the service contracts must be reviewed annually to assess if the vendor has fulfilled his SLA obligation and what if any more improvements he can make to his SLA.

Reviewing documentation is key when dealing with vendor-caused network issues. Check the SLAs to know the terms of service and any compensation for downtime. This helps you understand the expected resolution times and hold the vendor accountable if they fall short. Keep all communications documented to support any future disputes or claims. Being prepared helps you manage disruptions effectively and maintain a strong vendor relationship.

Bug-related outages are a fact of life. They are not going anywhere despite how cheaply or expensively you pay for equipment and support. Make friends with your SE (Sales Engineers), your AM (Account Manager), and the PLM (Product Line Manager) of the equipment you are using, so that you can have them on speed-dial when the next issue happens.

Unless the issue was the cause of some gross negligence. The Resolution Follow-up should not be looked at being punitive. It is strictly a learning period for each side to discover the ways of being able to mitigate the same type of issue in the future. Every company should have a lab in which they can test all software and hardware and evaluate the behavior before production implementation. Even 0day fixes can be tested and should be before taking down the cashbox to company.

In addition to thorough testing, I recommend implementing a post-mortem analysis after the issue is resolved. This involves a detailed review of the disruption, including what went wrong, how it was fixed, and what preventive measures can be implemented. Sharing this analysis with the vendor and internal teams can foster a collaborative approach to future improvements.

After the vendor fixes the issue, follow up to ensure the solution is stable. Test and monitor the network closely. Have a debrief with the vendor to understand the cause and resolution. This helps prevent future problems and strengthens your relationship, fostering trust and better collaboration.

After the disruption has been resolved, follow-up actions are essential to ensure that the solution is effective and that the vendor has met their commitments. Here's how to proceed: Verify Resolution: Confirm that the network is functioning normally and that all issues have been resolved. Conduct thorough testing to ensure that the solution is stable. Feedback to Vendor: Provide feedback to the vendor regarding their response and resolution efforts. Highlight areas where they performed well and areas that need improvement. Assess Impact: Evaluate the impact of the disruption on business operations and identify any long-term effects that need to be addressed.

Finally, use this experience to update your policies and procedures. Adjust vendor management strategies and add safeguards against future disruptions. Consider exploring alternative vendors for redundancy. Learning from this and adapting your strategies will make your system more resilient, ensuring smoother operations and better handling of future challenges.

Wherever possible, operate two different vendors (may be harder for an optical network, but easier for a routing/switching network). If you are proficient with the protocols and technologies, you can become an asset to your vendor when you identify and engage in bug resolution with them. This gives you plenty of credibility with them, and can put you at the top of every priority queue when your network hits an issue.

From my experience, most of the times when I have found that problems from external providers impact the operation of a client's network is due to the fact that the Service levels agreement and quality assurance process were not properly specified when carrying out the signing of the contract of services. As We may know all telecommunications, cloud and network as a service providers maintain standard products which are sold to hundreds or thousands of clients and it is a joint effort between the client and the service provider to customize the solution for the Final client.

To prevent similar issues in the future, it may be necessary to adjust policies and procedures. Here’s how to approach this: Update Vendor Policies: Revise vendor management policies to include clearer guidelines and expectations for network-related activities. Ensure that vendors are aware of these policies. Enhance Monitoring: Consider implementing more robust monitoring and alert systems to detect and respond to disruptions more quickly. Review Security Measures: Evaluate and update network security measures to prevent unauthorized changes or access by external parties.

In any role I’ve ever had, it’s rarely a deliberate action by a single individual that’s to ultimately blame for any given service disruption but rather it’s a flaw or oversight in the planning and execution of providing the service that’s at fault. Once these issues come to light then it’s imperitave that the relevant stakeholders determine the necessary policy adjustments that need to take place in order to prevent like disruptions from ever occuring again.

In addition to the steps outlined above, consider these additional points to strengthen your approach to managing vendor-related disruptions: Vendor Training: Offer training sessions or resources to vendors to ensure they understand your network's specific requirements and protocols. Regular Audits: Conduct regular audits of vendor activities and network configurations to identify potential risks before they lead to disruptions. Build Strong Relationships: Foster strong relationships with vendors based on trust and collaboration. A good relationship can lead to more effective communication and quicker resolutions during disruptions.

After all it falls back to your governance maturity level. Governance maturity level can be gauged by how well risk management is being exercised. A good risk management will allow us to map risks proportionately, include incident response. In case of incident, we will immediately get a grasp of severity and recovery procedures.

Join a public mailing list dedicated to the vendor's equipment. There, you will learn and share knowledge about all manner of failure and resolution modes you and others may have never heard of or ever imagined. Vendor engineers also lurk on those mailing lists, so it is a good chance to provide direct feedback to them on how they can better develop their products.

When considering networks of any size, always plan for failure, even if it comes down to having a 4G uplink and a secondary router in standby in the office. Downtime and network issues are one of the most costly problems to deal with.