My first postmortem!

Ok, so this is my first postmortem. And I'm writing this after a very interesting case of a server outage after, literally, a typo. This event occurred on Wednesday 27, 2021 and lasted from 16:15 to 17:20 GMT -5. Any get requests to the server returned a 500 error. The cause of this outage was a typo on line 137 of the wp-settings file calling for a non-existent file "class-wp-locale.phpp". I hope the typo is pretty clear.

Timeline (all GMT -5)

• 16:15: Outage begins.
• 16:20: Investigating root cause.
• 16:45: Root cause discovered. Typo in wp-settings file.
• 17:00: Typo fixed. Checking server.
• 17:20: All systems back online.

Resolution and recovery

At 16:20 I decided to check on the processes the server was running to see if there was any anomaly and to check on the PID of the Apache we server. Then I used the strace command on that process and call again a GET request to the server and saw why the process immediately failed: some file was trying to open a "class-wp-locale.phpp". Curious, I looked if the file existed, and it existed, but without the typo. Then, using the find command I wanted to look which file was the cause of the typo, and then, it showed me: the wp-settings.php file. By 17:20, all systems went back online after fixing the typo.

Corrective and preventive measures

As a preventive measure before deploying the server, it's best to double check typos right before saving the file, this will more likely prevent any outages due to (dumb) human errors...