Executable Specifications in software development

Written by Jane Orme. Estimated reading time: 6 minutes 

Sometimes our clients struggle with managing lengthy requirement documents that might only be partially implemented by the latest version of the product. Managing requirements and verifying that those requirements are implemented and functioning can be a large task with lots of time spent manually testing. Some clients need a list of requirements and regression tests but are starting with none. 

Executable specifications are a test-first approach to adding new requirements and their features to a product. They are an alternative to storing them in large requirement documents that might not accurately reflect the implementation. 

What is test-first? At Bluefruit, we do a lot of Test-Driven Development (TDD), when coding – where we write a unit test before and then add the code to pass it. Writing tests first is a technique that improves quality and keeps code more efficient. We can treat requirements in the same way. 

Contents: 

• What are executable specifications? 
• Why are they helpful? 
• How do you do them? 
• Top 5 tips
• Executable specifications for better requirements management 

What are executable specifications? 

Executable specifications are requirements written as tests that are automated. The test title becomes the requirement text, and the steps describe the functionality. These tests are commonly written in a plain-language format such as Gherkin and referred to as scenarios. 

In the following scenario, the requirement text is “The Home screen shall transition to the Recording screen”, and the Given/When/Then lines below it make up the test steps that will verify that the requirement functionality works: 

A step handler matches the step text with a function that executes some lines of code to simulate the action of that step. 

To automate these scenarios, each step is implemented by what is called a Step Handler. A Step Handler matches the text of each step to a specific function that performs the actions for that step.  

In the Behave Step Handler (for Python), the code that matches our steps to the method and performs the actions looks like this: 

When the test is run, it gives a pass or fail result, so you would know if the requirement was implemented and working correctly or not. 

Why are they helpful? 

Executable specifications can help give you a clear picture of what is implemented and working within the product. They can save time and improve quality and collaboration. They: 

Improve communication 

Getting the group of people together to discuss and decide on the requirement title and steps naturally improves collaboration between the development and product teams. The development team better understand what the requirement is for and should do. Questions and clarifications arise that development teams might only find once deep in the code. This saves time and ensures they are implementing the right thing. 

Create an automated regression suite 

Having all your requirements automated means you can run them regularly: 

• When the developers submit code changes, catching new accidental failures early. 
• At the end of each day or sprint. 
• When you have a version ready to release and want reassurance that your top requirements are still working. 
• When you need to revisit an older version and want to see which requirements were implemented and passed at that point. 

Testers are freed up to do other sorts of testing, such as integration, usability, or exploratory. They can focus on high-value manual testing, letting the automation handle more of the “grunt work”, such as batches of similar tests with small variations.  

Automated regression suites get run much more frequently, from “almost never” to every day or sprint. This gives development teams more confidence in the quality of the project. 

When tests are automated, development teams can catch more bugs and find improvements in the project. 

Time saved on testing means you can create releases faster. 

Live alongside the source code so you know the requirements match the implementation version 

Code doesn’t always get developed in a linear manner. Sometimes code branches are created to test out ideas or add features that never make it into the final product. Tracking this can be not easy if you only have a single external place where requirements are stored. Storing requirements as tests alongside the code means that your requirements are always up to date and have more flexibility to experiment and innovate before choosing the best option to keep. 

Better documentation 

Some test frameworks create test reports for you when the Step Handler runs the tests. Sharing this document in an accessible place means that everyone can see the latest status. This feeds into what is known as living documentation. Living documentation is the practice of having documents that are iterated upon and always up to date—ideally, automated as much as possible to reduce team effort and improve collaboration between teams and stakeholders. 

Simplify verification and validation (V&V) for compliant product requirements 

If you happen to be working in a compliant sector such as medical or automotive, you will know that it is essential to trace requirements to tests. As the tests are coupled with the requirements, this part is done for you and saves a lot of time! 

How do you do them? 

Collaboratively write a test in a Behaviour-Driven Development (BDD) style at the start of a sprint. 

As a group that includes the development team, requirement author, and any stakeholders: 

1. Decide on the requirement text. This becomes the title of your Gherkin test. 

2. Write the steps for the test that will verify it works. 

The development team then: 

3. Write the Step Handler implementation for each step.  

4. Run the test. It should fail because the development hasn’t taken place yet. Seeing it fail proves that the test is valid. 

5. Write the code to add the functionality to the software. This will often be implemented using Test-Driven Development (TDD). 

6. Rerun the test. It should now pass. 

Once the tests are run, share the documentation that contains the results. 

This integrates with the usual TDD process for making code changes. When the code is added, the TDD test will pass, and so will the BDD test: 

Top 5 tips

We recommend you follow these top 5 tips to write good executable specifications: 

Ensure a diverse team attends the initial planning meetings 

Having a broad range of roles and experience present when writing the executable specifications initially can ensure that they are fit for purpose and correct: 

• The development team will know what is feasible for the test steps. 
• Testers will make sure they know how to prove it works.
• Analysts make sure requirements are well written (especially if there is a poor specification or large specification documents) 
• And product-side stakeholders will know the functionality the users need and be able to answer any requirement questions. 

Phrase the steps consistently to enable reuse 

Slight variations on similar step wording can make matching a specific function within the Step Handler awkward. For example, the following step wording variations have the same intent: 

Creating a quality checklist or guidelines for test writing can make it quicker to implement new tests. 

Scenario Outlines with example tables process lots of test values 

You can use the Scenario Outline type of Gherkin test to repeat the same test for different data. These tests substitute each value listed in an Examples table into the scenario, running it once per value in the table. In the example below, the test will run three times, for “Dry”, “Wet”, and “Croup”. 

Automated tests don’t care if you give them large quantities of different values to test a piece of behaviour with, so take advantage of that by testing a wide range of values. For example, different motor speeds on a “motor speed accuracy” test, different date ranges, or temperatures for setting up a user heating schedule.  

Integrate with the build server 

The best living documentation is relevant, up-to-date, and easily accessible. Configure the build server to run the automated tests, generate the living documentation, and then push the test report (or all living documentation) somewhere that the team can view the latest details when needed. 

Use scripts to automate requirement numbers 

Another feature of Gherkin is the ability to give each scenario tags. These tags are prefixed with an @ symbol and are written above the scenario title. These tags can be used to classify the test type (enabling filtering), and we use them to assign a unique ID. For example, @ID-00042. Requirement IDs help with traceability. 

We recommend using a script to auto-assign unique ID tags to your scenarios to ensure each is unique. 

Treat executable specifications like code 

Executable specifications live alongside the code, often in a repository. They can be reviewed by a Product Owner as part of a code review. Drafted tests can be a part of a Definition of Ready for a user story and passing tests a part of a Definition of Done. 

Bonus tip: Use specific automated tests for edge-case testing, soak testing, and stress testing. Automated tests can be left running unattended, so use that to do more kinds of testing and more often! 

Executable specifications for better requirements management 

Teams that struggle with requirement management and testing delays can benefit from using executable specifications. Whilst they are best done up-front, they can be retrospectively added to create a self-verifying requirement document.