"Once you create a metric, it ceases to have value as a metric" is closely related to Goodhart’s Law, which posits that when a measure becomes a target, it often stops being a good measure. This dynamic has specific implications for the rail industry, leading to unintended consequences.
1. Incentives to Game the System
When specific performance metrics are turned into targets, rail operators may manipulate their operations to meet those targets, sometimes at the cost of overall quality and safety. For example:
- Punctuality Metrics: The UK's rail system emphasises "on-time" performance. In response, some train operators have been known to bypass stops, reduce the number of scheduled stops, or adjust timetables to appear more punctual, even if it inconveniences passengers or reduces service coverage. For instance, trains may skip smaller stations or implement "skip-stopping" practices, where they don’t serve low-priority stops to ensure they reach the final destination on time.
- Safety Protocol Shortcuts: In Japan's infamous 2005 Amagasaki derailment, an intense focus on punctuality contributed to a crash that killed over 100 people. Drivers faced disciplinary actions for delays of as little as a minute, which pressured them to speed excessively and cut corners on safety. This highlights how safety can be compromised when the "on-time performance" metric is prioritized without considering broader operational risks.
2. Short-Term Gains Over Long-Term Sustainability
Metrics can incentivise short-term performance over long-term investments critical for the rail system’s sustainability. For instance:
- Deferred Maintenance in the US Rail Industry: To meet financial performance metrics, many U.S. freight rail companies have deferred maintenance and reduced investments in infrastructure, leading to a backlog of necessary repairs. The result is an increase in track failures and derailments, such as the 2013 Lac-Mégantic disaster in Canada, where cost-cutting led to inadequate train safety protocols, resulting in an explosion that killed 47 people.
- Staffing Cuts for Cost Efficiency: Operators like Southern Rail faced severe criticism for understaffing trains and platforms to improve labour cost metrics in the UK. These cuts resulted in significant service disruptions, overcrowded trains, and safety concerns for passengers and staff, revealing that labour efficiency targets can negatively impact service quality.
3. Loss of a Holistic View
Rail operations are complex and interconnected. Focusing on singular metrics can lead to a narrow view, disregarding essential trade-offs. For example:
- Overemphasis on Revenue Per Mile in India: Indian Railways often uses metrics like "revenue per train-kilometre," which can push operators to run fewer, longer-distance trains that maximise revenue while neglecting local or short-distance connectivity. This compromises the accessibility and inclusiveness of the rail network, reducing service to smaller towns or regions where it may not be as profitable.
- Freight vs. Passenger Prioritization: In the U.S., metrics prioritising freight rail profitability often come at the expense of passenger service. Amtrak, the national passenger rail service, frequently experiences delays because freight operators have priority on tracks. This leads to lower passenger satisfaction and ridership declines, demonstrating how one-sided performance metrics can harm the overall system.
4. Metrics Can Become Detached from Reality
When metrics become targets, their real-world relevance can diminish. For example:
- "Right Time" Departure vs. Passenger Convenience: Many European rail operators measure performance by whether trains leave at the "right time," which is often defined as within a few minutes of the scheduled departure. However, this metric ignores the impact on passengers who may struggle to make tight connections or experience frequent schedule changes that, while technically "on time," are inconvenient and unreliable.
- Empty Trains to Meet Operational Targets: In Russia, performance metrics based on the number of kilometres travelled by passenger trains can result in trains running nearly empty to meet target distances. While this may improve statistical performance, it wastes resources and needs to reflect actual demand or passenger needs.
5. Reduced Innovation and Adaptability
Rigid adherence to metrics can encourage rail operators to adopt new technologies or improve services. For example:
- Resistance to Autonomous Train Operations: In cities like Paris, the RATP (Paris Metro) has faced resistance to automation despite its potential to improve efficiency and reliability. This hesitation often stems from fear of disrupting existing performance metrics that are already being met, even if automation could provide better overall service and safety.
- Reluctance to Adopt Modern Signaling Systems: In countries like Germany, the adoption of new signaling technologies, such as the European Train Control System (ETCS), has faced delays partly because legacy systems are meeting current performance metrics. This reluctance to invest in improvements due to metric satisfaction limits the rail network's capacity and reliability.
6. Pressure to Show "Quick Wins" Over Systemic Improvement
Rail operators may seek to show rapid metric improvements to satisfy stakeholders, but these quick fixes can have more profound consequences. For example:
- Service Reductions to Improve On-Time Performance in the UK: Following severe delays, operators like Northern Rail have sometimes reduced service frequencies to meet on-time performance metrics. While this improves punctuality on paper, it leads to overcrowded trains, longer wait times, and decreased passenger satisfaction.
- Cost-Cutting Leading to Lower Safety Standards: After privatisation in many countries, such as Argentina in the 1990s, private rail companies focused on reducing costs to improve financial performance metrics. This led to decreased investment in safety and infrastructure, culminating in incidents like the 2012 Buenos Aires train crash, which killed 51 people and injured hundreds due to neglected maintenance.
Conclusion
While metrics are crucial for measuring performance and guiding improvement in the rail industry, turning them into rigid targets often has negative consequences. They can encourage gaming the system, short-term thinking, a lack of holistic improvement, and resistance to necessary innovation. To mitigate these issues, rail operators must design metrics thoughtfully and ensure they incentivise comprehensive enhancements in safety, reliability, sustainability, and customer satisfaction rather than merely meeting isolated targets.
