How your software project can benefit from test automation

Test automation in software development can bring considerable savings to projects. Helping to reduce release cycle times, build solid foundations for traceability, and improving project status visibility. Freeing up testers (and some development time) enables them to invest time in activities that add extra value to your project.

But what is automated testing, and how does it help you? Read on to find out.

What is it, and why would you want it?

As a company, we fully embrace Lean-Agile, TDD, and any technique that improves quality and project success. Test automation can play a vital part in this. It gives these things the edge. When you automate your test suite, there are significant benefits across the project.

Most developers already create automated unit tests, but you can also automate typically manual tests. Examples include Behaviour-Driven Development (BDD) tests, white-box user interface tests, verification tests, and performance tests.

For us, automated testing is essential for creating and maintaining high-quality software, and it has many benefits:

Quickly discover regression failures, reducing risk

It is always tempting to dive right in with writing the code, adding feature after feature. Your developers are excited about what you are working on. Your team has time pressures and deadlines. The customers are desperate for that functionality. But what if changes inadvertently break a different feature? Would you realise? These regression failures can often happen if multiple people work on the same project and merge code changes. But don’t worry; test automation has your back.

Any developer who regularly runs their suite of unit tests will tell you the relief when they spot a new regression failure. (Admittedly, there is initially a moment of frustration with any defect!) But quickly, the “at least I know now” relief from realising the test suite alerted you early. If you had none, you might not know about a new defect until much later or even after release. It is then harder to unpick all the changes to determine when and how the bug crept in. And you will be less familiar with the code you wrote months ago. Fixing it late can be more frustrating and take even longer after more complex code has evolved on top. It is easier to fix it earlier. And having to fix bugs late means your project deadline is more at risk.

“Tests don’t break your code, they break your illusions about the quality of that code”
– Maaret Pyhäjärvi

Visibility and insight into the system quality

Everyone involved with a project can benefit from seeing the “health” of the software by looking at the test results. Project managers (or product-level people) can see the overall system status—which features are working or are not yet there. This reassures them of the software quality and progress. Developers quickly see what they need to fix. Testers who see several failures in one area can use the knowledge to focus their exploratory efforts.

Tip:  use human-readable test names that explain what the test does.

Improved quality

We know that doing Test-Driven Development improves software quality significantly. Microsoft and IBM performed a study in 2008 (link opens PDF) that directly compared test-first with test-last approaches. The study found that:

“[the] pre-release defect density of the four products decreased between 40% and 90% relative to similar projects that did not use the TDD practice.”
Source: Microsoft and IBM

We recommend starting by writing a BDD test that fails. Use the TDD cycle until that BDD test passes, and then automate the test suite produced. This gives quick quality feedback when making code changes.

If you use test automation, fewer defects will sneak into a release version.

Code coverage

Code coverage metrics are a bit of a freebie when you have a test suite. It identifies areas of code that don’t get tested at all. You might not want 100% code coverage, but you can ensure the riskiest or most complex areas have associated tests. We aim for 80-95% code coverage for embedded software development, depending on the specific project. However, fewer better-quality tests can be better than poor-quality high-coverage tests.

Manual testing reduced

Manual tests are slow to perform because there is only so fast a person can do something. The testing pyramid shows that it is more effective if the bulk of your tests are automated. Automating a test reduces the need to test it manually, saving time (and cost). With fewer testing delays following a releasable version, there is the potential for shorter release cycles.

Automated testing can be more thorough than a human; it can test all the inputs, and press more buttons, more accurately and quickly. For that reason, they can be “cheap” to run once automated.

More complex automated tests, such as GUI tests covering multiple functionalities or user interfaces, can be harder to maintain. So, choose these tests carefully.

Testers will always be valuable on a project. Automated testing does not remove the need for testers—in fact specialist testers are actively involved in implementing automated manual tests. Instead of repeatedly manually testing “basic” things that have passed since day one, it frees them to test other, more exciting things. Instead, they could do more exploratory, usability, performance, or security testing. Or focus on complex areas or try different platforms.

Option for automated requirements and documentation

A specification (requirement) that is automated is called an executable specification. These can be your test scenarios when doing Behaviour-Driven Development (BDD). If BDD tests are automated, you have a living requirement document that auto-verifies itself. And you will know that it matches the software implementation. Automated documentation can be especially beneficial for regulated or safety-critical software. It’s ideal for sharing knowledge and visibility of the project across the team and to product-level people.

What types of testing could you automate?

Test-Driven Development (TDD)

TDD often takes the form of developer unit tests. Developers write tests before they start to code. The test passes when a developer adds code. Writing in this manner focuses the developer on the intention of the feature before they begin. It encourages them only to add enough code to make the test pass before refactoring to make it more maintainable or efficient. All our development teams work in a TDD style.

Behaviour-Driven Development

BDD boosts communication. Stakeholders and the development teams collaboratively discuss the system’s required behaviours and write the BDD tests together. As part of the conversations, questions arise, and details clarify the intention. Writing these tests in a human-readable format (not code) ensures more people can understand them. BDD tests can form part of the requirements, and automating them gives stakeholders clear insight into the product status.

Automated testing works best in Agile and Lean-Agile software teams

Many Agile or Lean-Agile development practices work well with test automation. You might already do some of these things.

But what do we mean by Agile or Lean-Agile? You can visualise Agile as being three layers, with Lean software development included:

• Principles: Lean-Agile principles puts adaption over prediction. The seven core ideas include empowering the team, reducing waste, and amplifying learning. Test automation can also help you build integrity in and deliver as fast as possible.
• Project management practices: Agile teams are already familiar with the common Agile ceremonies, such as sprints, tracking work in user stories and Kanban, and retrospectives.
• Technical practices: Automated testing extends Agile technical practices focusing on high-quality and effective software development. Techniques that are a good fit for automation include TDD, BDD, and continuous integration and delivery.

When you’re already doing Agile testing activities that improve quality, the automation of those is the natural evolution.

What’s the investment like to set up automated testing?

We often see that customers who have had some automated testing experience already know its value. So, they are happy to invest the time at the start. You could implement it in-house or outsource to involve people with experience and toolkits already.

Unbelievably, for TDD, having more automated testing efforts will not slow down development overall. The Microsoft and IBM study mentioned above found that TDD did not slow them overall. The estimated 10-15% extra effort at the start saved time later in the project. The study states:

“From an efficacy perspective this increase in development time is offset by the by the reduced maintenance costs due to the improvement in quality (Erdogmus and Williams 2003), an observation that was backed up the product teams at Microsoft and IBM.”

And once you have a test suite, it is not a big step to automating it. It could be as simple as adding it to a build script or batch file to run automatically after doing code merges.

Working in a BDD-way, you are already creating behaviour tests. Tools also exist to automate each step of these scenarios. However, the effort required to automate might depend on the toolchain decisions.

Are there any commonly used test automation frameworks, script-based, that you recommend?

Framework choice often depends on the hardware and software you will use in the project. At Bluefruit, we often use Python-based Behave for the BDD tests. Behave acts as a test runner for human-readable behaviour tests, written in Gherkin and stored in feature files. We might also use Selenium for web testing, Squish and Funq for Qt, and Appium for WPF (C#). Your choice will depend on your project, platform, and technology.

Tests rock

“If you’ve got the code for your project and the tests, what would you rather lose if you had to lose one? I’d lose the code because if I’ve got the tests, I know I can build that code again.”
– Yan, Senior Software Developer, Bluefruit Software

Are you looking to bring test automation to your next project? Bluefruit Software has helped several clients automate testing for their products. Get in touch today to see how our teams can help you.

Written by Jane Orme.