Why Performance Testing Your Applications Is Not Enough: Performance Engineering Best Practices

Performance Testing, while a substantial contributor to the process, is considered a subset of Performance Engineering. The primary goal of Performance Engineering is the reduction and avoidance of unplanned infrastructure costs, development remediation costs, service interruptions and production tuning efforts. Performance Testing is an important part of any systems certification effort, but alone, it is not enough for enterprise wide system certification of mission critical software and hardware.

Performance engineering is practiced at every phase of the Systems Development Life Cycle which ensures that a solution will be designed, implemented, and operationally supported to meet the non-functional performance requirements defined. The goal is to detect problems early in development while using measurable methods to support cost-benefit analysis of hardware vs. software solutions. This critical point is one reason why performance testing alone is not enough.

Performance engineering begins at the design stage, with infrastructure and design review of the solution, proceeds through the execution phase with code and infrastructure, and continues until the product leaves final certification. When the project is released to production, continuous monitoring of the application's performance and the application’s capacity to operate as designed is verified. All of these activities are defined as performance engineering; they comprise activities that go well beyond testing alone.

The following is a set of best practices I have found to be helpful to establishing a true Performance Engineering capability.

Get Commitment Of Performance Engineering From All Levels

The successful adoption of performance engineering requires commitment at all levels of the organization. Developers are usually keen to do whatever it takes to improve the quality of their product; upper and middle managers are faced with numerous conflicting goals which must be satisfied. Articulation of the value of performance engineering activities as well as identification of risks associated with only adopting a testing purview must be shared. Value and risk can in part be derived from failure models: what happens when performance is sub-par (loss of clients, business)? The value of these early engineering activities may be harder to defend without some solid historical data from within your organization's portfolio. If you don't have any, there are plenty of sources outside that will help you tap into industry data around the value of performance testing and risk related to performance dis-satisfaction or outright failure.

Integrate Performance Engineering Into The Software Development Process And Project Schedule

To be effective performance engineering should not be an “Add On” service, it should be integrated into the way in which software development is approached. Integrating performance engineering into the development process avoids two problems: over reliance on individuals by a project and performance being treated as a discretionary budget item rather than an essential development process.

When an individual is relied upon by a project, rather than a process, the knowledge of that individual leaves when they leave. Processes are standardized and not dependent upon an individual.

Making performance engineering an integral part of the development process ensures that performance is not left behind as the project’s schedule or budget slips during development. Performance problems are not always apparent; managers or developers are frequently tempted to omit performance engineering in favor of meeting project milestones.

Calculate Early Estimate of Performance Risk

Identify Risks - New Technology, Architecture, Schedule, Team Skills, Market Factors, etc. It is important to understand the level of performance risk and to make those risks known. Get them published early and circulated at the beginning of the project. Within the discipline of performance engineering, risk is anything that has the possibility of endangering success of the project. These risks can include new technologies, evolutionary flexibility of the solution architecture, market factors, project schedule and other influences. You can use the same risk models you used above for obtaining management buy-in.

It is important to document and justify the financial value and benefits of the application of performance engineering efforts. Such costs will include performance specialists, technology, tools, and testing facilities. The benefits are usually avoidance of costs due to poor performance, reactive performance tuning, lost revenue, system outages, support costs and contract penalties.

Architecture Assessment

While decisions made at each phase of development are important, architectural decisions have the greatest impact on quality attributes (re-usability, reliability, performance, etc.). While a good architecture cannot guarantee performance, a poor architecture can prevent the achievement of good performance. Evaluating those decisions early in the context of potential performance issues will go a long way to informing how you approach your early performance certification tasks once you have some code to test.

Identify Critical Business Use Cases or Scenarios

Use cases describe the behavior of the system or its sub systems. They document the user’s view of what the system is supposed to do. Critical use cases are those that are important to the responsiveness as seen by users or are deemed a performance risk. That is, critical use cases are those for which the system will fail or be less than successful, if performance goals are not met. You should push for those types of use cases early on; it may be too late to circle back to the user community or the business sponsors mid-way through the project because they may be tied up with other competing activities.

Establish Precise, Measurable Performance Objectives

Precise, measurable performance objectives help to control performance by explicitly stating the required performance in a format that can be used to quantitatively determine whether the software meets the objective. Well defined performance objectives also help in the evaluation of architectural and design alternatives and trade-offs. It aids in the selection of the best methods to meet performance and quality requirements.

Performance engineering is an amalgamation of practices under a single discipline. The discipline integrates practices and processes with the SDLC and can also be applied to agile methods as well. Performance engineering influences all phases of development from design through release to production.

Establishing a strong, repeatable performance engineering process will bring greater success to your overall quality efforts. It will ensure a happier user community through fewer performance related disruptions. While performance testing alone is an important step, performance engineering will provide your organization with a more reliable and well performing product.