FireNet: Critical Tropical Communications

Broward County, Florida sits directly at the business end of Hurricane Alley.

Named storms are the New Norms, and 26 inches of rain per hour is not just a theoretical thousand-year event. We call that Last Year, and likely next year too. Under these conditions even the most margin-rich microwave link is subject to rain fade, and fiber users wish they only had backhoe fade.

This is our tropical world of critical communications under all its versions of ugly.

===

Fire Dispatch Communications are, by most measures, the most time-sensitive and critical of all Public Safety applications. When fire apparatus are expected out the door in 60 seconds, each call really can be a life in the balance.

With over 300,000 calls for service annually, the Broward County Fire Dispatch system is one of the largest, busiest, and most important life-critical communications systems in the world. We built FireNet to deliver those calls to every Fire Station across the 1200 square mile county.

We decided early (and learned often) to approach this challenge with multiple layers, multiple paths within them, and multiple end-point retries, each optimized to increase the probability of data delivery success. To make it play, come what may.

When we designed FireNet we knew that this was the last stop between dispatch and disaster. We had excellent suppliers including Cambium, General Electric and US Digital Designs (now Honeywell), but when the storms roll in, the vendors stay home.

I tell our team to think about response time while you're holding your breath, because -- on a daily basis -- somebody is not breathing and needs Fire Rescue.

As the Joint Venture partner responsible for FireNet, I made a personal commitment to make the very best of all available resources, and to build the most reliable network possible. This is what we delivered and how we did it.

About FireNet

FireNet provides fire dispatch communications on a wide-area licensed 4.9 GHz network to 107 Fire Stations throughout Broward County, Florida. ? Multiple UHF sites provide overlapping layered backup coverage. FireNet is essentially a multiband, multilayered, Public Safety?WISP with OCD-grade backup reliability.

A traditional 15-hop 6/11 GHz microwave ring is enhanced by overlaying hub-and-spoke ancillary MPLS/L2 connectivity at 7 key sites. The network supports 44 high profile 4.9 GHz quadrant panels on 11 County and co-located tower sites.

Reliability: Above and Beyond

The red numbered blocks below denote the base design with associated test points. This shows the first of three phased field deployments.

The nature of environmental outages became more clear over time, and we overlaid a hub-and-spoke alternate route network, shown above in red. We optimized our delivery process by distinct edge, network and application grooming.

Edge Grooming

One poignant example of the need for edge grooming was discovered the hard way, and at a very bad time.

It turned out that, in times of trouble, the Telco would sometimes split the IP route to and from a destination. This is not an unknown transport/bearer network anomaly, but it is certainly unexpected, and there is no cure to that chaos this side of an AT&T engineer. When this occurred in the middle of a serious storm, our Solarwinds queue was already brimming with alarms, and we learned how to groom the edge for that obtuse uncontrolled external variable in-flight; fun times...

We similarly found that Cisco UDLD protocol was quite helpful on bouncy ports when the world is crashing all around.

Most of our edge grooming accounts for truly fantastical tropical field conditions and might not be required in some environments. But as Broward "tag-along tenants" in 16 RF shelters controlled by Others, we do not have custody of site conditions or even visibility of adjacent issues. Power problems, grounding issues, lightning strikes, walls of water, and human errors all happen.

We looked at the reliability challenges in total, and we faced them squarely with our customers and suppliers, together as one team. We were determined to make FireNet as resilient as possible and together we accomplished that by designing to: Tolerate simultaneous failures, Degrade gracefully, and Provide self-healing end-to-end connectivity with (unique!) managed RF resource routing.

Application Grooming

We are fortunate to have US Digital Designs/Honeywell as a major supplier for FireNet. Their excellent Communications Gateway provides a well-implemented delivery algorithm that we used to optimize individual sites specifically for our harsh tropical environment. The USDD Gateway's ability to support and optimize multiple outbound paths for each station uniquely within the application environment allows us to measure and tune the performance to best suit our tropical network conditions.

During our deployment USDD also developed and provided an important site proxy server which optimized and synchronized the outbound channel so we could re-use a single frequency pair from four disparate sites. This allows us to provide seamless geographically distributed overlapping coverage.

Even our backups have backups.

Personally I don't know of any other vendor who addresses this as well, and in my experience the USDD team understands and delivers; I highly recommend them.

Network Optimization

Storm Season used to fall between Mother's Day and Thanksgiving; these days it's pretty much continuous. Even when there is not a hurricane (we're overdue) we experience weeks on end of severe storms rolling through Broward County, often inflicting multiple sequential and simultaneous outage events.

Even AT&T dispatch-grade lines bounce hard, and regularly. After considering and trial testing many network options, we implemented OSPF (Open Shortest Path First). We added an overlay, leveraging Broward's MPLS connectivity, providing an additional layer of enhanced fault tolerance.

OSPF readily detects outage conditions and reroutes around them, any which way it possibly can. By adding path weights to strategic links we can tune the network as necessary and appropriate.

One of my great geek-joys now is watching the network reconverge from my dashboard afar, fully confident that, no matter which links fail, or why, OSPF will do its job and communications will be optimally restored; it just works.

Each Fire Station is uniquely designed with at least two, and often three or four RF delivery paths. This increases our reliability substantially, and we have yet to miss a call going into our fourth storm season.

Storms line up in a grim sequence rolling through South Florida with tropical force winds and torrential rains punctuated by unfathomable gigajoules of lightning, often on a daily basis, for weeks on end. If you want to talk outages, lets!

Mitigating simultaneous broad outage conditions was our initial goal and remains our highest challenge. This has been one of the most intriguing problems of my career, and I am grateful for the opportunity every day.

Lessons Learned

We learned many lessons along the way, some harder than others. And we learned something new just last week. Without overstaying my welcome (vigorous writing is still concise):

• Design For Change. System changes. Staffing changes. Customer changes. All. The. Time. That is not generally a measurable bid spec and it was not here, but presents significant ongoing design, test, documentation and maintenance challenges.
• Create "Solutionship." They're more than solutions, not quite relationships, but they're needed within our circle. As we've come into more information we need to socialize it, add to our Cookbook of standard operating and technical procedures, announce to users and otherwise deal. Extending this reach to 24/7/365/15-years puts a generational shine on documentation and training. We Rehearse, Review and Revise as part of our standard maintenance procedures. "Solutionship" may be one too many words for any language, but the community intention is important.
• Be adroit. This is never easy in a production level life-critical dispatch environment. When failure can mean death, success requires careful planning, incremental testing, and the courage of one's convictions.
• FAIL GRACEFULLY. Fall back, don't fall out. When things go downhill, slow down. Seek for root cause(s) like your very life depends on it, because somebody else's does. Plan and practice the Art of Graceful Degradation.

===

FireNet is more than a project now. It is a maintenance and migration continuum, and will be for the next 15 years on a 24/7/365 two-hour basis. This tier of maintenance response requires more than a break-fix service approach.

Fortuitously we are contractually responsible to Identify Fire Dispatch Failure Root Causes, regardless of the source or supplier. This allows everyone to avoid all manner of finger pointing and balderdash accordingly. (And, there is little quite so satisfying as the truth when it's somebody else's problem!)

FireNet is our bridge Between Dispatch and Disaster and our team stands ready, capable and equipped.

We did the best we could, on-schedule with zero dollars of change orders. I am honored to present FireNet to my peers; please hit the LIKE button and tell all your friends (unless you think it stinks, in which case: Bring It!)

Please hit that Like button; we need the KPIs. And the validation. But mostly KPIs.