Increase reliability, reduce cost, and speed up your project with virtual testing

Executive Summary

Steer73 were developing a new consignment scanning automation solution in a group of high-volume warehouses for a global logistics client. The legacy system was extremely manual and involved staff scanning individual barcodes with a handheld scanner. Steer73 were contracted to build an easy-to-use, reliable system to speed up shipment processing (by over 80%) and reduce error rate.

Due to the extremely expensive scanning hardware involved and COVID-19 restrictions, Steer73 wanted to find a way to reduce or eliminate physical testing using actual hardware.

To remove uncertainty around bugs post-release and ensure a well-tested product was delivered, Steer73 constructed a series of ‘virtual’ test rigs. Working from hardware specifications, drivers, documentation, and a single test device, Steer73 built and validated mock test rigs to enable simulated testing of thousands of shipment scans.

A scripting tool was created to enable the test team to cover all necessary test scenarios, and run over 20X the level of testing that would have been possible on physical hardware. The approach reduced project costs and risks, resulted in a far more reliable product, increased timeline certainty and resulted in a smoother development process.

Full article

Client
An international courier and logistics company

The solution being built
Streamlining shipment scanning for greater speed, accuracy, and traceability. The legacy system was extremely manual and involved staff scanning individual barcodes with a handheld scanner. Steer73 were contracted to build an easy-to-use, reliable system to speed up shipment processing (by over 80%) and reduce error rate.

The problem

Within the solution Steer73 was building, there was a collection of physical scanning devices. However, the devices were large and very expensive, making obtaining a test rig impractical. Furthermore, COVID restrictions meant that the development team were unable to attend site.

This meant that:

1

Steer73 had no conventional means of testing their solution against the integration as they built it.

2

Risk was skewed towards the end of the project when code could be tested on the real hardware.

3

There was a large unknown at the end of the project, leading to uncertain timelines and costs.

The solution

Determined to deliver a robust solution on time and on budget, Steer73 tackled the problem with a two-stage approach:

Stage 1: Specification

Early on in the project, Steer73 agreed firm specifications with all stakeholders about exactly how external dependencies would integrate.

Using the hardware drivers, documentation, and a single test device, Steer73 were able to establish how the devices would communicate with their application.

Liaising closely with the hardware supplier, Steer73 agreed and documented the format of the data that the scanners would return.

Stage 2: Constructing test rigs

Using these specifications, Steer73 were able to create mock instances of the external integrations.

These allowed Steer73 to develop the app against “fake” instances which they could use to simulate the missing component. The Steer73 team made efforts to design these with various outputs in mind and worked with the quality assurance team to capture their test scenarios so that they could also simulate error states.

For the scanning hardware, they created interfaces modelling the services they would need and then created mock services alongside the production ones.

Including a ‘virtual hardware’ mode in the application allowed them to simulate specific hardware events. A simple scripting language extended this further, allowing the team to replay a large number of events sequentially, including with simulated delays representing real-world usage. This enabled both a wide range of different tests, and also a high volume of test runs to be done, resulting in very high confidence in the quality of the solution.

The culmination of this was that the Steer73 team were able to run scripts simulating thousands of scenarios, something that would not have been possible even if they had the physical hardware to test on. In fact, running the same number of tests on a physical test rig would have probably taken over 2 years!

The outcome

In short, when it came to deployment, the Steer73 team approached this stage of the project with huge confidence. They had run an incredible number of tests, even without the physical hardware to test on. As a result, the risk and uncertainty that could have been shifted to the end of the project had they not developed the test rigs had been eliminated.

When it came to swapping the mock services for the genuine service, it was a trivial configuration change. There were zero problems with the hardware integrations.

Benefits for the client

The cost of retrospectively reacting to issues at the end of the development was swapped for a well-known (and smaller) amount of work at the start of the project.

Costs were significantly reduced, as no expensive hardware was needed to test on. The system went through many orders of magnitude more testing, enhancing robustness and reliability.

The timelines for the project were significantly more certain due to the elimination of unknown issues in deployment.

Benefits for Steer73

Significant reduction in project risk.

More accurate time planning and project trajectory.

More flexibility around the scheduling of the development work, including reduced blockages between team members working on related areas.

Fewer interdependencies leading to a more elegant, maintainable implementation.

A happy client and a long-term partnership!

Working with our test rigs meant I didn’t have to keep running manual tests on complex hardware, so I could really focus on the important parts of what I was doing. There would have been significantly increased project risk if we hadn’t found a way to simulate things and run a lot more tests covering all the different real-world scenarios we had to cater for and working with other team members would have been virtually impossible. - Quote from the Steer73 team

Conclusion

The ‘test rig’ model used on this project was a fantastic validation of Steer73’s approach to risk reduction and something they will continue to use across projects.

Post-release QA – which is often the default option for projects of this type – results in increased levels of risk and higher costs. Swapping this for consistent, incremental steps towards a well understood target, improved quality and resulted in a much better outcome for everyone involved.