How to Shift from Performance Testing to Performance Engineering and Why You Should Do So

Slow product performance can lead to a large number of negative consequences, from user dissatisfaction to loss of customers. If you provide critical services, performance glitches may even worsen the quality of life for your users.

For example, in 2021, the NHS App went down for several hours and users couldn’t access their COVID passes. Thus, NHS clients couldn’t access public services requiring proof of vaccination and even couldn’t board flights.

Performance testing is often not enough to uncover all hidden issues before a product’s release. But switching to performance engineering — an approach that prioritizes performance from the start of product development — can significantly improve software quality.

In this article, we examine the cycle and benefits of performance engineering and offer practices to ensure its smooth adoption. This post will be helpful for development and QA team leaders, project managers, and product owners who want to improve their system’s productivity.

What is performance testing and why is it not enough?

Software performance testing is a quality assurance (QA) practice designed to examine the speed, scalability, and stability of your solution. Unlike other QA practices, performance testing isn’t aimed at detecting bugs in code. Its goal is to make sure that a user will be comfortable interacting with your product despite heavy loads.

Running performance tests before release is a common practice, especially for projects that rely on Waterfall development models. Performance tests help you to:

• Validate that software satisfies performance requirements by emulating the load specified in requirements and checking software behavior
• Identify key performance issues and bottlenecks due to extensive testing under various conditions and loads to discover cases of system crashes, unavailability, unexpected behavior, and slow responses
• Improve user satisfaction with the end product by checking that users will not experience issues with software functionality
• Avoid interference with other development stages, since performance testing is isolated from the rest of the software development lifecycle and is conducted at the end of the development cycle

So if performance testing is so efficient, why do many solutions still freeze when we need them the most? Partly it’s because of errors in the performance tests themselves. But performance testing also has drawbacks and limitations you can’t fix. Here are the key ones:

Late discovery of performance issues. Performance issues are usually discovered when the product is already developed and functional testing is completed. To fix such issues, your team will need to rework part of the code and all of its dependencies. In some cases, they may even need to change the software infrastructure or architecture. Substantial code changes at this stage will dramatically increase the cost of your project.

Limited time for in-depth testing due to release dates. The late start of performance testing often means the product already has a release date and the QA team has limited time to test it. Because of that, some performance issues may be overlooked or not get enough attention.

Risk of false positives and testing errors going unnoticed. Being a one-time activity, performance testing provides little room for double-checking the results of previous tests and code after fixing issues. QA specialists aren’t immune to human errors, and the absence of a safety net in the form of additional testing may allow some errors may allow some errors into the final product.

Possible inconvenience for alpha users. If your product relies on an iterative development methodology, you can share it with early adopters before the release of the final version. Performance tests can negatively influence the experience of these early adopters. You can avoid this issue by planning tests for hours with minimum load (e.g. nights or weekends), but you’ll need to cover overtime of your QA specialists and additional infrastructure costs.

Because of these limitations, at Apriorit we combine performance testing and performance engineering to ensure the best result. Combined performance testing and performance engineering recently helped us discover performance limitations that could have resulted in the loss of users for one of our clients. The client wanted to integrate a particular SMTP server into their application to send important information to users via emails directly from the application.

We conducted load testing and discovered that this server could only process 40% of the emails before it freezes, and the rest were simply lost. We fixed this issue before the release of the new version of the app and helped our client avoid a problematic situation.

Now let’s see how performance engineering works and why lots of businesses shift to it from simple performance testing.

What is performance engineering?

Software performance engineering is an approach to software development and testing that introduces continuous performance assessment and improvement processes throughout the product development cycle.

The key difference between performance testing and performance engineering is that the latter focuses on development with performance in mind from the moment your team starts working on the product. Performance engineering fits well in Agile development and shift-left testing. Both of these methodologies implement the idea that each development iteration must end with a ready-to-use product version.

The performance engineering process consists of four key stages:

1. Architecture and modeling

QA experts and business analysts start software development by gathering and analyzing non-functional requirements for future software. They need to understand the expected system load, the estimated number of users, the methods of interacting with databases, integrated third-party services, etc.

This information helps BAs predict possible productivity issues and bottlenecks and design the system architecture with performance in mind. In-depth research of requirements and architecture modeling can save your development teams hours of bug fixing at later development stages.

2. Performance diagnostics and profiling

Thorough research of performance requirements doesn’t mean new performance issues won’t arise during development. Consider using diagnostic tools to analyze product performance during each iteration. They will help you discover newly emerging issues and fix them during the iteration when they appear.

Diagnostic tools create baseline performance parameters for common system events, such as requests to the database and responses to certain buttons. Then they can detect cases of abnormal system performance and notify the development team.

There are plenty of diagnostics tools out there that help to measure a particular system parameter. Your team can also use the built-in functionality of Microsoft Visual Studio to measure code performance.

3. Performance testing

When included in the performance engineering process, performance testing mostly works the way we described earlier. But there’s a bigger emphasis on load testing, since performance diagnostics helps to eliminate most of the issues.

Performance testing should be conducted at each iteration that brings changes to the product’s functionality. Before the release, it’s best to conduct an additional round of testing to make sure that product performance satisfies the requirements and allows for a smooth UX.

At this stage, QA specialists emulate user activity in the system to see how it behaves under expected and higher-than-expected loads. They also examine how the system scales under load and analyze the user experience (UX).

4. Monitoring system capacity

When your software is released, the product team monitors its performance and infrastructure capacity with dedicated tools to ensure that the system works as expected. Similarly to the performance diagnostics and profiling stage, the product team needs to determine baseline parameters and look for unusual spikes of activity to predict user experience issues or system malfunctions.

At this stage, your team can use Microsoft Azure Monitor, Amazon CloudWatch, Windows Performance Monitor, and some tools used during the performance diagnostics and profiling stage.

This final stage of performance engineering helps to ensure a smooth UX, detect and fix issues before they become critical, and collect data to set up automatic system scaling in the future. The choice of tools depends on the monitoring level and the place of product deployment.

Such a long and continuous process takes a lot more time and resources from a project team than one-time performance testing. Let’s see why performance engineering is becoming more popular and how it can benefit your project.

What are the benefits of implementing performance engineering?

Being a complex approach to ensuring software performance, performance engineering helps businesses catch performance-related issues early and fix them at a relatively low cost. Here’s how performance engineering can help your business:

1. Continuous improvement of the user experience. Performance engineering allows your team to control product performance during all development stages and after the software release. Thanks to that, all product users will encounter fewer performance issues, even if they started using the product during the alpha release.

2. Ability to save money for fixing productivity issues. The cost of fixing the same bug varies depending on the time when the bug is discovered. To fix a bug or a performance issue during the requirements elicitation and architecture design stages, you only need to change your plans. At the development stage, your team will need to rewrite code and maybe even change the architecture. Doing so requires you to allocate more time and resources for bug fixing.

Performance engineering helps your team discover and address productivity issues early in the development cycle. This way, they will be able to fix issues quicker and at a lower cost.

3. Reduced infrastructure costs. Performance engineering helps you predict the load on your system and determine which resources it will need to scale and when it will need them. Using this information, your development team can fine-tune scaling processes and reduce the use of unneeded resources. Such fine-tuning is especially important if your system relies on cloud resources to scale, since cloud expenses are closely tied to cloud usage.

4. Possibility to use modern development approaches. You can easily fit performance engineering into the continuous integration and continuous development (CI/CD) lifecycle, DevSecOps, Agile and Lean development, and other approaches that are gaining popularity in the development community.

These approaches allow businesses to be more flexible and iterative with the software development process as well as deliver a working version of the product with each iteration. Adopting performance engineering can help you ensure each iteration is usable.

5. Fewer performance issues after release. The release of the final product is often a stressful time for the project team. No matter how well you prepare for it, there always will be some bugs to urgently fix and issues among the first users. Performance engineering helps to ensure that your product will not go down under the growing load after the release and will scale smoothly to provide users with a quality user experience.

Adopting performance engineering and achieving these benefits will take you a long time, especially if your development team hasn’t used any iterative development approaches before. Let’s take a look at five practices that will help you move from performance testing to performance engineering smoothly and quickly.

How to adopt performance engineering?

It’s best to implement performance engineering right at the start of a new project so you can add its principles to the foundation of your software. Adding it in the middle or the end of your project’s SDLC may bring more challenges than benefits.

When starting a new project, you can introduce performance engineering with these five steps:

1. Research non-functional software performance requirements

Software performance and scalability are the key elements of non-functional requirements. They describe:

• Average number of users interacting with the system over a certain period of time
• Maximum expected number of users
• Planned application scalability
• Number of requests various parts of the system must be able to process
• Expected response time for various user actions

Usually, business analysts (BAs) are in charge of gathering and eliciting all project requirements. When Apriorit’s BAs start working on a project, they interview various project stakeholders, collect existing project documentation, analyze existing solutions on the market, and prepare use cases.

The results of business analysts’ work allow the rest of your team to outline performance metrics and criteria for future performance testing. These criteria will also be useful when designing the software architecture.

2. Assemble the performance engineering team

Assign team members within your project team that will be in charge of performance engineering. Depending on the size of your project, they can be dedicated only to performance engineering or have other responsibilities as well.

At Apriorit, we include the following roles in any performance engineering team:

• Business analyst. The business analyst has to define requirements for the product’s performance and scalability in the manner discussed above.
• Software developer. Keeping performance requirements in mind makes developers change their working routines, tools, approaches, and coding practices. That’s why it’s best to designate one or several developers in your team that will specialize in performance improvement. They can oversee performance-specific development tasks and run tests, while the rest of the team will focus on coding.
• Performance engineer. This specialist is involved in all stages of performance engineering. Their role is to create a performance engineering strategy and oversee that it’s executed as planned. Depending on the project specifics and the development stage, a performance engineer can facilitate requirements gathering, write various QA tests, research ways to improve product performance, etc.
• QA specialist. This team member is in charge of the performance testing stage. They write tests to check the product’s ability to scale, perform under various loads, respond to user requests, and so on. Their goal is to make sure that each product iteration delivers a satisfying user experience.

3. Address internal pushback when shifting to performance engineering

Adopting performance engineering can be a mental as well as a work-related challenge. It differs greatly from Waterfall-style development approaches, where performance testing is done at the end of the SDLC. Shifting performance checks left in the development process makes every product team member change their mental gears and keep an eye on software performance and functionalities.

Another mental challenge is the constant need to look for new and novel solutions to problems that can arise when implementing performance engineering. Since this approach is relatively new, your team may face challenges that are not yet discussed on StackOverflow and Quora.

To help your team adopt the new approach, explain why you want to try it out and how it may improve your team’s work. Also, provide the team more time to prepare for the project before they switch to performance engineering. Alternatively, you can hire outsourced performance engineers that can share their expertise with your team.

4. Select the right tools

The adoption of performance engineering requires your project team to broaden their toolset for measuring and improving software productivity. Your performance engineering team not only needs to be able to test the system and fix discovered issues but also:

• Continuously monitor the system’s work
• Measure lots of metrics
• Analyze measurements
• Integrate performance engineering tools into the CI/CD pipeline

The good news is your team doesn’t have to do all of this manually. They can select dedicated tools that fit your project and CI/CD pipeline. Performance engineering tools also help to automate a lot of processes.

At Apriorit, we use Azure Monitor and Amazon CloudWatch for performance monitoring. They can watch system performance, collect actionable insights, and notify us when there’s an issue that requires a developer’s attention.

5. Focus on engineering instead of fine-tuning

Performance engineering ultimately teaches any product team to look past pass/fail tests and focus on improving their system.

Say that developers discover your system slowly processes a bunch of user requests. If your team simply adds more computing power, it might help for some time but won’t resolve the underlying issue.

However, if your team follows performance engineering practices, they will first take a look at request processing mechanisms and analyze what causes the delays. The reason may be a lack of computing resources, inefficient work with databases, cluttered code, use of outdated development practices, etc.

Figuring out the true reason behind poor performance will take your team more time but will allow them to deliver a solution of higher quality, fix the issue permanently, and even reduce infrastructure costs.

Conclusion

Performance engineering is aimed at detecting and resolving performance-related issues at the early stages of the SDLC or avoiding them altogether. Shifting from performance testing to performance engineering helps businesses deliver high-quality software, ensure a smooth user experience, and reduce the cost of development and system infrastructure.

Compared to performance testing, engineering requires a product team to consider system performance at the start of the project, even before writing a single line of code. Your team has to keep system performance at the top of their priorities throughout the whole development process. Whether you choose performance testing or performance engineering, adopting these practices is both a procedural and mental challenge.

QA experts and developers at Apriorit pay a lot of attention to ensuring the performance of our solutions. We carefully elicit all performance-related requirements, take them into account when designing the system architecture, and constantly watch out for arising issues.

Reach out to us and start improving the performance of your system with this modern engineering approach!