How We Fixed a Desktop App Running in 4,000 Pharmacies

?????? We all have our startup war stories. I'll tell you mine, but what's yours?

My war story today involves one senior engineer being onboarded to contribute to a pharmacy app. The app was running in 4,000 pharmacies across the whole US, with an automated system to push out new settings.

On his first day, the poor lad accidentally deleted the settings across all 4,000 pharmacies, rendering the app completely useless. There was this script that could update the settings, and he just was in the wrong place at the wrong time. The script didn't have enough protections built in, and the documentation could have been better.

Instead of displaying valuable notifications that would integrate into the pharmacists' daily workflow, the app instantly became dead wood. That was a bummer, because thousands of man-hours of creative marketing and customer support work had gone into getting this app installed into so many pharmacies ??

???? But it wasn't the first time we were faced with an impossible challenge. Because that's what happens when you go from #mvp to #productmarketfit

We immediately formed a war room with the most experienced engineers in the team who worked on the app previously. Meanwhile, our project manager was keeping the customer informed continuously, giving the engineers space to figure things out. Our engineers were brave and didn't cave to the immense pressure.

A few hours in, they came up with a creative solution to write a special Python server that impersonated an ElasticSearch cluster. This special server would intercept the broken requests from the pharmacy apps and reply with the correct information. 36 hours later, everything was fully functional again. They had written, tested and deployed to production the server, and healed the vast majority of the instances of the application.

What I loved most about this?

1. ????? Our team was brave, and dealt with the pressure, immediately entering "fight mode"
2. ?? Communications with the customer were clear and transparent
3. ? Once the fire was over, we had a good laugh and started learning lessons without blaming people or pointing fingers

The "poor lad" is now one of the most valuable team members ??