The Ins and Outs of Performance Testing - Part 2

In part 1 ( here ) , I talked about the core aspects of SVP testing, some of key concepts and some fundamental metrics to monitor in your results.

Now, let's talk about some important considerations when it comes to designing and executing your test strategy.

Potential Tripwires

Here are some key considerations that have significant impact on the efficacy of your SVP tests and whether you will actually get usable metrics and SLAs or just an academic number with no practical value. 

SLAs and NFRs

SLAs and NFRs define the range of acceptable operational parameters for your systems.

If you're building an enhancement on an existing app, or indeed if there are additional pre-existing systems in your end to end solution , you will need to retrieve SLAs without fail. i.e. any previous benchmarks of performance of these systems . While this should be straightforward, It should not surprise you if you are unable to retrieve these. 

SLAs and NFRs for performance typically need to be finalised jointly with IT ( Infra, UX, UI at the very least ) and business, and product owners. That the system should be resilient enough to support infinite load and work perfectly is not a valid requirement. Organisations need to get down to practical numbers of how many customers they are looking to support today and into the future and actually pen those numbers down.

An easy way to get this information from existing system log analysis and seeing the trend of traffic and response time over the last year and extrapolating into the future.

If the system is brand new, without previous logs to analyse, then, ideally, the business case must define some level of anticipated customer engagement, which would need to be translated into traffic and system built accordingly. Then, crucially SVP testing will then establish the SLA for the new system for the first time.

Log access

This should go without saying, but you must ensure you have access to application and network logs to do analysis for each key node in the system that is of significance in serving an incoming request. 

Not only are they crucial to establishing SLAs for existing systems, but also key to effective result analysis.

While you may not need to analyse each node’s ( system ) log for every test or project, it is extremely important to have the capability to isolate and get at them easily in case there are abnormal results.

Network logs can be harder to get than application log depending on who manages either ( e.g. if the network infra is managed by a corporate shared service and not your project, you may need to account extra time and funding for those resource for log retrieval / debugging )

Payload Size vs SLAs

This one especially applicable to CMS, where the payload of the final output delivered to end customers (e.g. a web page) can vary depending on what authors put on the page.

SLAs need to be defined w.r.t. to an acceptable page / payload size. E.g. a general SLA of page response times of 500MS will fail if authors put up several high res images on the page.

A more reasonable SLA is "page load times of 500MS or less for an average page size of 500KB"

Page Views vs Concurrent Requests

This is more geared towards web based systems, loading HTML based pages.

SVP SLAs typically deal with a system being able to support a certain number of concurrent users or allow certain number of concurrent page views. This number is always lower than the actual number of requests servers are expected to handle

Each page that is loaded may involve multiple static assets, e.g. images, javascripts, CSS files - each asset will result in an individual request back to the server for processing.

Thus when analyzing the logs either isolate your requests to server per page ( something which can be quite tricky depending on the tool for log analysis ) OR have an empirical ratio of page views to requests processed to help make sense of your results - e.g. assuming an average of requests per page view

Environment

You must conduct SVP in a production like environment. This includes all necessary network connections and system integrations that are needed in providing the experience to the a live end user . All systems and applications must sit on infrastructure that is production like in capacity

Note: SVP environments are not cheap – thus any capability that allows you stand up and terminate SVP environments on demand will be immensely useful. This is generally served by a mature DevOps team, if available

Permutations of Execution

There can be multiple configurations and setups in which you can conduct an SVP. It is not necessary to test all such configurations ( e.g. see below , browser cache on or off ). However it IS necessary to simulate real world conditions of traffic as much as possible including spike loads and DDOS attacks , to measure the true limit of your application.

Headless CMS testing

A headless CMS generally has the CMS providing raw content while another application renders the experience. An end to end SVP benchmark is the most relevant for the end user, however given the number of systems involved, it can be beneficial to do performance tests for CMS and rendering systems separately

Headless CMS systems typically have multiple hops from fetching the content to final render and this break up of benchmarks can support identification bottlenecks in the pipe. This is also beneficial if different systems have different support agreements and thus even through individual SLAs may be fine but the overall end to end SLA for the consumer fails.

E.g. If both , the rendering system and CMS has an SLA of response time 300 MS and if the end to end response time SLA needs to be 500ms , then there can be opportunities where all systems are operating within their parameters ( e.g. headless CMS takes 270ms and rendering systems takes 300ms ) and still delivering a poor user experience ( final response times = 570ms )

Cloud based applications

If your application is on a cloud, and if you have a CDN in front ( Content Delivery Network e.g. Akamai or Amazon CloudFront ) , an SVP test directly onto the public end point as it will appear in production will yield little value.  In this scenario you will be testing the performance of an industrial strength CDN network – You will invariably get a fantastic result but no insight into the performance limits of your applications.

A true determination of performance will require you to disable your CDN cache and allow all requests to go through to the application servers.

While this is especially applicable for a transactional application, It is also important for static websites, to help you plan out how your static websites can grow in the future especially if there are plans for growth into dynamic content.

Browser Cache Considerations

Many load testing tools give you an option to simulate browser cache on or off. This has a direct impact on the number of requests reaching the application vs the no. of users. If off, the results you get will be lower than the actual real world performance as typically normal users will have their browser cache enabled by default

Server cache 

Web applications typically have a web server cache . e.g. in AEM , the dispatcher has the cache. It is generally favourable to run a test with both server cache off and then on , to measure the capacity and benchmarks of both the Web Server Tier and the Application Server Tier

Specific to CMS - Author Performance

SVP focuses on the capacity and resilience of the system in relation to end user experience. Quite often people overlook the fact that authors of a CMS system are also consumers and thus there should be consideration of performance testing no the author facing systems.

Author environments typically perform a lot more write operations, and must be able to support reasonable number of concurrent authors without breaking down ( e.g. saving content changes starts to take prohibitively long as more and more users log on )

Miss this aspect at the risk of usability of your entire setup.

Conclusion

SVP testing is a crucial part of not only understanding how your application will fare in the real world but also how you can plan for growth of your system.

While simple in concept - " hit the system as hard as you can and see if it breaks" , there are many considerations to keep in mind when designing and executing your test strategy to get usable results

That's it for this series on SVP testing, and I hope some of my learning from my engagement provide some form of reference or guidance

Let me know how you go in the comments, or indeed if there are areas where you'd like me to elaborate further.

Until next time!