V&V Testing: Signs you have a verification & validation issue
Verification and validation (V&V) are vital in medical device software development regardless of where you’re looking to sell your device.
But how do you know if your V&V testing activities could get in the way of software certification? And what can you do about it? Read on to find out.
Coming up:
What are verification and validation?
Imagine you are building a house. Has the house been built? You ask someone to verify that a house has been built—that’s verification. You then ask if the house was built correctly—that it is fit for purpose for someone to live in—that’s validation.
V&V testing is a little like this:
Of course, it’s more complex for verification and validation in medical device software. But reasonable verification and validation processes should answer:
Has the right software been built? And has that software has been built right?
For medical device software, this means:
- Verification ensures that software teams build the correct software according to each action it needs to do, based on the software requirements.
- Validation looks to see if the intended use of the device software fulfils the stated user needs.
It’s worth noting that the definitions above are up for debate. In guidance from both the EU and FDA on V&V, descriptions of the two terms are woolly at best.
Either way, getting V&V right is no small task.
The challenges of getting medical device V&V right
The software verification and validation process aims to ensure full traceability between your device’s software requirements and what your users need.
Getting medical device V&V right is challenging. It can be tough to:
- Maintain traceability;
- Show full verification coverage of the requirements with tests;
- Ensure risk assessments are done and included in the traceability;
- Develop testable software requirements in the first place;
- Find appropriately trained personnel to carry out V&V.
How do you know if your verification and validation processes aren’t working?
Five signs your medical device software V&V needs fixing
If your V&V efforts aren’t effective, you risk software not getting regulatory approval. And with the high cost involved in both time and investment for higher class devices, it makes sense to get V&V right from the get-go.
What follows isn’t a complete list, but here are five signs that your software verification and validation efforts could sabotage your software.
1. Mediocre requirements
Requirements that are ambiguous or hard to understand will bring down your software’s traceability.
How? Because the relationship between requirements and what is in the software and what it does will be hard to see. Why does this feature exist? Why does the software act in this way? Is the software acting in the way it’s meant to?
Poor requirements make it challenging to test the software and how it functions.
2. Not enough test coverage
A knock-on effect of poor software requirements is insufficient test coverage. If you don’t know what to test, there’s no way to have a complete test suite.
Often this happens because product owners or people working on user experience have created requirements in isolation. No one’s engaged the V&V team early to determine if the starting requirements are easily traced or testable.
As our checklist for checking your IEC 62304 readiness points out:
You need to have sufficient testing in place or else there’s a risk of harming a patient down the line.
Processes, team experience, and requirement quality can all affect test coverage.
(It’s worth noting that IEC 62304 only covers verification. For medical device software you’ll be looking to QMS standards for product validation. And for an SaMD, IEC 82304 will cover you for validation.)
3. Requirements aren’t kept up to date
When you don’t keep requirements up to date, you end up with misalignment between requirements and what’s being coded and tested.
Developers implement something that’s not needed or in a way they shouldn’t. Testers test it and find that a discussed change hasn’t happened. A true red flag moment.
That misalignment wastes time and resources. Keeping requirements up to date ensures shared understanding, which helps everyone stay on the right track. But too often, there isn’t someone responsible for overseeing and maintaining requirements who also has sight of changes discussed and agreed upon.
4. Leaving verification and validation until the end
If you leave it to the end of development to see if the right thing has been built correctly—it will cost more to fix.
Keeping on top of it from the beginning, taking small vertical slices, can help the team stay on top of things and reduce surprises. It’s well known that fixing things later can get expensive.
In the case of making a medical device, it’s not just the change that needs to happen in the software. Everything then must be documented and put through V&V again.
5. Using tools like Word or Excel to record requirements
Sure, word processors and spreadsheets are easily accessible pieces of software. Non-technical people can use them. But with them, it is far too easy to break traceability because there’s no:
- Automation;
- Strict version control or review process;
- Unique auto-numbering.
Manual tools like these aren’t built for the level of rigour most medical device software projects need.
And when something does go wrong with requirements recorded using a tool like this, it’ll be tough to fix. The team member tasked with that must make sure they don’t mess up any other requirements and manually check for duplicates. That person’s time would be better spent updating requirements with wanted changes rather than having to fix what’s already there carefully.
What you can do to improve in-house V&V
You can take many steps to ensure that your verification and validation efforts don’t hold your product back. The following isn’t exhaustive, but it’ll help with many of the issues raised above in V&V testing.
1. Think verification and validation from the start
Ensuring you have fit for purpose software requirements can happen if you involve software testers and analysts during their creation. With usable requirements, teams will be able to get test cases right.
And having a software analyst involved early means you can have someone in from the start who’s helping to keep requirements in check. Software analysts can help alleviate challenges, including version control and creating activities for verification plans and tests. They are especially good at assisting with documentation for IEC 62304.
2. Write requirements in a standardised way
Consistently written requirements help everyone involved in implementing and checking them to be sure of what they’re doing. It makes them easier to understand, helping to build understanding between everyone.
You could take an approach such as using Behaviour-Driven Development (BDD). Here, user stories can be turned into requirements (often written in a Given When Then format such as Gherkin).
Or requirements could be written using an Easy Approach to Requirements Syntax (EARS). Here, the aim is to reduce the issues introduced using natural language, which can be imprecise and unclear.
3. Keep your feedback loops short and frequent
Through tools, effective processes, and rapid feedback, verification and validation activities don’t need to slow projects down.
You can keep a project on track by including it as part of an overall “Evolutionary Lifecycle”, where it’s done little and often. This feedback loop can use new knowledge from testing (from QA to user feedback) and development to continuously feed into the product.
What does an Evolutionary Lifecycle look like? According to the FDA, who put it forward in TIR45 (their idea of how to bring Agile to medical device software development), it looks like this:
In a lifecycle like this, it’s assumed you’re working in an Agile or Lean-Agile way. With software developed and tested in small increments, such as sprints.
During the main body here, V&V activities can fit in easily as part of ongoing work alongside other activities like quality assurance. If they uncover discrepancies, teams can feed that knowledge into the development cycle.
4. Use living documentation
Living documentation can help with keeping on top of documentation.
Instead of relying on lots of manual processes and leaving everything to the end, living documentation done right cuts the time spent on documentation tasks. It benefits software development and testing teams who work in Agile or Lean-Agile ways.
Why? It’s to do with many of the technical practices and tools they’ll use. Things like Test-Driven Development (TDD) and BDD are ripe for being pulled in alongside other documentation in a version-controlled and human-readable manner. Like this:
How does this fit into a release cycle? Well, following the ideas put forward in TIR45, living documentation as an automated process happens at the story, sprint, and release stages:
There’s automation of traceability and updates to requirements in a set up like this, with version control and document approvals.
And, of course, living documentation benefits the most from not being used alongside tools like Word or Excel. Tools such as Jama, and others, can help here.
Should you be doing V&V in-house?
One thing worth considering with your V&V testing for your medical device software is whether you should be doing it in-house at all.
For instance, the FDA advises that an independent party should ideally perform verification and validation:
“Validation activities should be conducted using the basic quality assurance precept of ‘independence of review.’ Self-validation is extremely difficult. When possible, an independent evaluation is always better, especially for higher risk applications. Some firms contract out for a third-party independent verification and validation […].”
While the FDA states this can be an internal team, clients often approach Bluefruit if they don’t have sufficient resources (or are not sufficiently experienced) to achieve this. Suppose you’re facing the issues described in this post and cannot make the changes needed to solve them. In that case, you really should consider outsourcing V&V for your medical device software.
Do you need verification and validation for your device software? Talk to us
For over 20 years, Bluefruit Software’s software teams have worked in safety-critical sectors, including medtech. If you’re reconsidering your approach to medical device software V&V testing, our teams can help you.
Further reading
Did you know that we have a monthly newsletter?
If you’d like insights into software development, Lean-Agile practices, advances in technology and more to your inbox once a month—sign up today!
Find out more