Case Study: How I Restored a Down Website and Optimized AWS Infrastructure for an E-commerce Client in 20 Hrs

Introduction:

• Client/Project: An e-commerce business that had been facing significant downtime due to server resource issues and an infected WordPress website.
• Timeline: The project took 20 hours to complete.
• Role: DevOps
• Hourly Rate: $30/hour.

Problem/Challenge:

• The client was frustrated as their website was completely down, and previous DevOps help had become unresponsive. They only knew that the website was down, but they didn’t understand the technical aspects behind it.
• Upon diagnosing, I discovered that the EC2 instance for the WordPress website (Bitnami) was running on a t2.small instance, while the actual billing reflected t2.large. Additionally, the website was infected with malware, causing server overloads and gateway timeouts.
• The setup included an EC2 instance for the WordPress Bitnami installation, an RDS database (5GB size), and an Elastic Load Balancer with Auto Scaling, which were either misconfigured or not being utilized effectively. The site was experiencing high traffic (50k users/month), and this exacerbated the resource issues.

Solution:

With 14+ years of experience in the tech industry, I took on the dual role of DevOps Engineer and Project Manager, ensuring that the client understood every technical decision and its impact on their budget. I provided a clear, transparent consultation on the best way to optimize the website, despite slightly increasing AWS costs due to necessary resource upgrades.

Here’s how I approached it:

• Backup and Restore: I immediately performed multiple backups of the WordPress files and RDS database to ensure we had working copies for restoration if needed. This was the first step to safeguard the client’s data while troubleshooting the site.
• In-depth Troubleshooting: After performing PHP and Apache optimizations, removing unwanted plugins, and cleaning the infection using Wordfence, I restored the files and database several times. However, it became clear that the primary issue was resource-related. The EC2 instance size (t2.small) was insufficient to handle the traffic load, and the auto-scaling setup was misconfigured.
• Transparent AWS Consultation: I explained to the client the benefits of scaling their EC2 instance from t2.small to t2.large and upgrading the RDS instance from t3.small to t3.large. I made sure to balance performance needs with budget concerns, ensuring the client would only pay for resources they truly needed.
• Optimizing AWS Resources: I upgraded the EC2 instance to t2.large for better performance and reconfigured the Auto Scaling setup, so the system could dynamically adjust based on traffic loads. Additionally, I terminated unnecessary EC2 and RDS instances, helping to reduce their monthly AWS bill.

Implementation:

• Backup & Restore: I used the following Bitnami WordPress commands to back up and restore the website:
• Optimization of PHP and Apache Configurations: I configured the following to optimize PHP and Apache for WordPress:
• Scaling & Cost Reduction: After optimizing the server, I explained each AWS change to the client, like upgrading EC2 instances and tuning Auto Scaling and Load Balancer configurations, making sure all unnecessary resources were terminated.
• Final Outcome: The website was restored and optimized to handle up to 50k users without any issues, with faster load times. Although the AWS cost slightly increased due to the necessary resource upgrades, I terminated unused instances and optimized settings to minimize extra spending.

For those who want to understand how and what steps were performed, I have prepared a tutorial or steps based on actions taken. Here's a complete step-by-step list of all the commands, including permissions setup, AWS configuration, backup and restore commands, file operations, and more. I have organized them by functionality for easier reference:

1. AWS Configuration

Configure AWS CLI:

aws configure        

This command will prompt you to input your AWS Access Key, Secret Key, Region, and Output format.

Pull a file from AWS S3 Bucket:

aws s3 cp s3://bucket-name/wp-content_datetime.zip /tmp/wp-content_datetime.zip        

This copies the wp-content_datetime.zip file from your S3 bucket to the /tmp directory.

2. WordPress File Permissions

Set Ownership for WordPress Files:

Make sure that the ownership of the WordPress files is set correctly:

sudo chown -R bitnami:daemon /opt/bitnami/wordpress/        

Set Correct Permissions for Files and Directories:

• Directories: Set the permissions for all directories to 755:
• Files: Set the permissions for all files to 644:

Set Correct Permissions for wp-config.php:

• Set ownership:
• Set permissions to 640 to protect it:

3. Backup and Restore WordPress

Remove All Files from wp-content Directory:

sudo rm -rf /opt/bitnami/wordpress/wp-content/*        

Extract a WordPress Backup Zip File:

To extract the wp-content_datetime.zip file from /tmp to the wp-content directory:

sudo unzip /tmp/wp-content_datetime.zip -d /opt/bitnami/wordpress/wp-content/        

Backup WordPress Plugins Directory:

To copy all plugins from a backup directory (bk_plugins) to the active plugins directory:

sudo cp -r /path/to/bk_plugins/* /path/to/plugins/        

Zip WordPress Directory and Move to Temp:

To create a zip file of the WordPress directory and move it to the /tmp directory:

sudo zip -r /tmp/wordpress_backup.zip /path/to/wordpress/        

4. Swap Configuration

Create Swap File:

Create a 2 GB swap file:

sudo fallocate -l 2G /swapfile        

If fallocate is not available:

sudo dd if=/dev/zero of=/swapfile bs=1M count=2048        

Set Correct Permissions for Swap File:

sudo chmod 600 /swapfile        

Set Up the Swap File:

sudo mkswap /swapfile        

Activate Swap File:

sudo swapon /swapfile        

Make Swap Permanent (Add to /etc/fstab):

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab        

Optional: Adjust Swappiness (System Tuning):

sudo sysctl vm.swappiness=10        

To make the swappiness setting permanent:

echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf        

6. Installing Software (htop)

Install htop on Bitnami VPS:

1. Update package list:
2. Install htop:
3. Run htop:

7. Useful Monitoring Commands

Check CPU and Memory Usage:

top        

or use htop for a better user interface:

htop        

Check Free Memory and Swap:

free -h        

8. Tunneling for phpMyAdmin Access

Create an SSH Tunnel with PuTTY on Windows:

1. Open PuTTY and go to SSH -> Tunnels.
2. Source Port: 8888 (or any unused local port).
3. Destination: localhost:80.
4. Click Add.
5. Now connect to the server with PuTTY.

Once connected, you can open phpMyAdmin in your browser by visiting:

https://127.0.0.1:8888/phpmyadmin        

Login to phpMyAdmin Using RDS Database:

Since you're using an RDS instance, you can use the database credentials from your wp-config.php file to log into phpMyAdmin.

9. Additional WordPress Commands

Deactivate All WordPress Plugins:

sudo wp plugin deactivate --all --path=/path/to/wordpress/        

These commands should cover all the steps required for managing your Bitnami WordPress instance on AWS, including backups, restores, file management, and database optimization. Let me know if you need any further assistance or if additional steps need to be included!

Results:

Quantifiable Outcomes:

• Increased Performance: wp-admin load times improved drastically, with a 50% reduction in response time.
• Stability Restored: Website downtime was eliminated, with the server now able to handle 50k monthly visitors without issues.
• AWS Cost: Although AWS costs slightly increased due to resource upgrades, the system now performed reliably and was scalable for future growth.

Client Impact: The client’s site was fully restored, with significantly faster admin functionality. Although their AWS bill increased, the benefits far outweighed the costs, as the site now runs smoothly and efficiently.

Challenges Faced & Overcome:

• Infected WordPress Files: The malware required multiple cleanups and backups before the site could be fully restored.
• Server Resource Bottlenecks: Despite optimizing the code, the sluggish wp-admin area and gateway timeouts persisted until the EC2 and RDS instances were upgraded. This solution involved transparent decision-making and budget management with the client.

Conclusion & Key Takeaways:

• Lessons Learned: Sometimes a slight increase in costs is necessary to ensure optimal performance, especially for high-traffic websites. Balancing performance, security, and budget constraints is key to delivering a successful project.
• Future Potential: With the Auto Scaling and Elastic Load Balancer now properly configured, the system is well-prepared for any future traffic surges, ensuring the client’s infrastructure remains robust and scalable.

Summary:

• Role: DevOps Engineer & Project Manager.
• Total Time Consumed: 20 hours.
• Hourly Rate: $30/hour.
• Total Project Cost: $600.
• Outcome Achieved: Restored the website with 50% faster wp-admin response times. Handled traffic without downtime. Slight AWS cost increase for better scalability and performance.

Now for the important question:

So, guys, what do you think? Did I do a good job? ?? I’d love to hear your thoughts!