High traffic periods can pose a challenge for any system administrator. How do you ensure that your servers, applications, and networks can handle the increased load without compromising performance, reliability, or security? In this article, we will explore some of the best practices and tools for maintaining system performance during high traffic periods.
The first step to maintaining system performance is to monitor your system regularly and proactively. You need to have a clear picture of how your system behaves under normal and peak conditions, and identify any potential bottlenecks, errors, or anomalies. You can use various tools, such as top , ps , vmstat , netstat , ping , and traceroute , to monitor different aspects of your system, such as CPU, memory, disk, network, and process usage. You can also use more advanced tools, such as htop , glances , nmon , sar , iostat , and tcpdump , to get more detailed and comprehensive information. You should also set up alerts and notifications for any critical or abnormal events, such as high CPU load, low disk space, network congestion, or service failures.
The second step to maintaining system performance is to optimize your system based on the monitoring data and your performance goals. You need to tune your system parameters, such as kernel, file system, network, and application settings, to improve the efficiency and responsiveness of your system. You can use tools, such as sysctl , tuned , ulimit , sysfs , and /etc/security/limits.conf , to adjust various system settings, such as buffer sizes, swap space, process limits, and scheduling policies. You should also optimize your applications, such as web servers, databases, and caches, by configuring them properly, using caching and compression techniques, and minimizing resource-intensive operations.
The third step to maintaining system performance is to scale your system according to the demand and capacity of your system. You need to plan ahead and anticipate the possible scenarios and challenges that your system may face during high traffic periods, such as seasonal spikes, flash sales, viral events, or cyberattacks. You can use tools, such as ab , siege , locust , and jmeter , to perform load testing and benchmarking on your system, and measure its performance, throughput, latency, and availability. You can also use tools, such as ansible , puppet , chef , and terraform , to automate the provisioning and deployment of your system resources, such as servers, storage, and networks. You can use different scaling strategies, such as vertical scaling, horizontal scaling, load balancing, and clustering, to increase or decrease your system resources as needed, and distribute the load evenly and efficiently across your system.
Adding more resources to scale your system is a valid approach. But there are few systems that are infinitely scalable and also scalability is not always linear. Doubling the resources does not mean your system will be able to cope with double the workload. Load test your system before hand, have a plan on how to manage peak and monitor the system in live are all good things to do.
In my experience , get the peak load of system before hand and performance test prior to identify bottlenecks. After resolving the bottlenecks, get recommended scalability configurations and configure accordingly in production systems.
Question: Do you know the profile of the scaling activity over the course of the peak day? Do you understand the workload mix? Have you validated any assumptions in your workload model? If the answer is no to any of these questions then go away and check it out before you consider scaling options.
The fourth step to maintaining system performance is to secure your system from any external or internal threats that may compromise your system integrity, availability, or confidentiality. You need to implement various security measures, such as firewalls, encryption, authentication, authorization, logging, auditing, and backup, to protect your system from unauthorized access, data breaches, denial-of-service attacks, malware infections, or accidental errors. You can use tools, such as iptables , sshd , openssl , fail2ban , rsyslog , auditd , and rsync , to manage different aspects of your system security, such as network filtering, remote access, data encryption, intrusion detection, event logging, system auditing, and data backup. You should also follow the best practices and guidelines for system security, such as using strong passwords, updating your system regularly, disabling unused services, and limiting user privileges.
The fifth step to maintaining system performance is to document your system thoroughly and consistently. You need to have a clear and updated record of your system configuration, architecture, performance, and issues, and share it with your team and stakeholders. You can use tools, such as lshw , lspci , lsblk , uname , and df , to collect basic information about your system hardware, software, and storage. You can also use tools, such as git , svn , markdown , and sphinx , to create and manage your system documentation, such as code, scripts, manuals, reports, and diagrams. You should also use tools, such as issue trackers , wikis , forums , and chat apps , to communicate and collaborate with your team and stakeholders, and keep them informed and engaged about your system performance and issues.
The sixth and final step to maintaining system performance is to learn from your system and improve it continuously. You need to analyze your system data and feedback, and identify any strengths, weaknesses, opportunities, or threats that your system may have. You can use tools, such as awk , sed , grep , sort , uniq , and cut , to manipulate and process your system data, such as logs, metrics, and reports. You can also use tools, such as gnuplot , matplotlib , pandas , and numpy , to visualize and analyze your system data, and generate graphs, charts, and statistics. You should also use tools, such as feedback forms , surveys , reviews , and ratings , to collect and evaluate your system feedback, and measure your system satisfaction, quality, and impact. You should also use tools, such as blogs , podcasts , webinars , and courses , to learn and update your system knowledge, skills, and trends.
Do it in Disney Hotstar way. During ICC Men’s WC 2023, it scaled up to 5.3 Crore (53 Millions) eye balls. They scaled down on the features no one would use during the live streaming.
Think about what can be disabled during busy periods to allow more capacity to handle the peak traffic. For example some batch processes could be delayed or even some application will restrict functionality during busy periods to ensure core functionality is maintained.
Consider conducting a business continuity plan based on the organization’s size and structure in compliance with relevant regulations and industry standards.
Where possible use workload prioritisation to ensure your important workloads get the resources they require - for example using IBM’s Workload Manager
Consider leveraging Content Delivery Networks (CDNs) to boost system performance during high traffic periods. CDNs cache and serve static content from strategically placed servers worldwide, reducing the load on your origin server and ensuring faster content delivery to users. CDNs also offer DDoS protection and load balancing. Providers like Cloudflare, Akamai, and Amazon CloudFront can enhance your system's performance during traffic spikes, delivering a smoother user experience, especially for global audiences during peak times.