Blogs

The value and role of Test Beds to enable System Integration, Spiral Upgrades and Rapid Systems Experimentation

31/01/2021

This, in turn, enables them to evolve and create new defence capabilities to counter emerging new threats and get new technology and systems into the hands of war-fighters faster. Previous articles in this series have explored the role of digital threads to build through-life evidence chains, and a vision of a Defence Digital Experimentation Platform to explore concepts and make early design decisions. These all need to fit within the right regime of Test and Evaluation (T&E) to inform the decisions needed at the most appropriate time. A key tool within that regime is a test bed within which to qualify system integration, deliver spiral upgrades and conduct rapid systems experimentation.

Within a complex system, or system of systems, there is rarely a simple one-to-one link between a single component and the wider military capability delivered. Bringing these components together to form such a system is integration. Integration is most effectively achieved when changes are designed to be integrated from the outset, then progressively de-risked, integrated and accepted into service. The pace of complex system change is ever increasing. Some changes could impact in hours or days, perhaps involving agile development methodologies, whilst others may take several years to reach fruition. These changes may be hardware, software and/or critical settings, or a mixture of all these. As threats evolve, as the mission changes, and as equipment becomes obsolete, the whole complex system has to be changed, with various asynchronous lifecycles. And throughout, consideration needs to be given to safety, security and cyber.

As many as a half of the components of a system of systems could be changing at any one time, so making and coordinating these changes takes planning, monitoring and critical decision-making, all being informed by a complex system risk profile. Mitigating and qualifying risks lead to the need for targeted T&E across the testing regime to fill knowledge gaps.

By having ready access to the right T&E regime and evidence, the customer can be flexible to priorities, uncontrollable programme changes and evolving understanding. Using this T&E evidence, the customer can make the decisions and choices to maximise platform availability through knowing what capability can and cannot be fully exercised.

As you will see from our previous article on digital T&E, whilst there will be a logical drive to maximise model-based approaches, these will not completely eradicate the need for testing real systems when seeking to de-risk complex system integration. Hence, a core consideration for T&E capability is an effective test bed or reference integration facility to:

• Gain confidence in an evolving system which is integrating progressively maturing components that are being incrementally de-risked, tested and trialled. This applies both to the system lead and the supplier as they can gain confidence in their sub-systems implementation to meet the host system’s needs (interface, functionality and standards);
• Assess and qualify the evolving impact that the changing component has on the rest of the system;
• Conduct experimentation to evaluate candidate options, or potentially risky solutions, to understand the merits and drawbacks of their progression.

The test bed can be used to represent the combat system of a naval ship system, this could be land-based or even fitted into a floating platform for specific trials purposes. A test bed can be used to represent an aircraft’s sensors, weapons and defensive aids suites this could be either ground-based or fitted into a suitable flying platform. Test beds can also bring together complex weapons and/or communication systems. A representative test bed is a safe and cost effective place to evaluate changes without direct exposure to front line platforms and systems where the risks may be higher and the need for stakeholder assurances much greater.

Through appropriate management and treatment, safe and secure testing can be conducted in a test bed. Changes can be rapidly implemented and tested to explore options and inform further development. It is better, and more cost effective, to find and resolve as many problems in the test bed rather than wait to deploy in a front line platform where additional costs, delays and assurances would be needed. However, there are inevitably some things that can only be tested on a front line system, within a truly realistic environment.

The aim of the test bed is to minimise the necessary testing in the front line system but where testing needs to happen there, understand the best and most cost effective means to undertake.

A test bed does not purely comprise all of the components that make the system; it sits within an appropriate test harness which is typically a synthetic test environment. A synthetic test environment allows a wide range of conditions, circumstances and events to be simulated within the test bed, at lower cost and in a shorter period of time than live trials in realistic conditions, and perhaps with live targets. This synthetic testing can be conducted relentlessly and repeatedly as the host system evolves and matures. A standardised set of system tests can be defined to enable baselining towards qualification. With increasing experience and evolving understanding, those baselining conditions can be progressively matured.

To fully understand the function, behaviour and performance within the test bed, various common tools provide the means to instrument and analyse the system in the test bed. These tools can measure and characterise the whole system and the impact of the changing component. These same tools can be further exploited to instrument and
evaluate real systems and units operating in truly realistic conditions.

Sub-systems could be shared and sent to other suppliers for them to conduct system integration, whilst potentially beneficial for the suppliers involved; across the whole enterprise this drives cost and time into integration. The test bed should be a resource for the customer, system integrator and the supply base and will be most effective if a single capability or facility can satisfy the needs of all to provide coherency and cost effective delivery. Policy, process and standards can be used to increase design understanding across the enterprise and minimise the need for T&E, however there is always scope for misunderstanding or errors for which undertaking T&E within a test bed is the best tool. The customer can use it to gain overall confidence and eventual qualification; this could help to continually populate and update evidence in the digital thread. The supplier can progressively evaluate and test a maturing product or component, in particular, to resolve uncertainties that cannot be assessed in isolation from the host system (such as within the factory).

Within a mature enterprise, a collaborative and open-book approach should be adopted where the customer, system integrator and supply-chain see the outputs from a test bed to drive cost and time benefits into the programme. The system-integrator can provide the customer with confidence in their platform and they can understand priorities and concerns from the customer. The customer can see how each of the evolving parts are progressing and how the whole of the system is evolving and maturing. The right commercial environment ensures that all parties are ‘safe’ and secure, whilst maximising collaboration for customers’ end benefit.

A well designed test bed can also be used for wider testing with other collaborative test beds, or live units (such as in representative or realistic environments). By envisioning the test bed as being an interchangeable force component, it can be used to test or experiment within a real force, or the test bed can provide a more controllable test component within a force. This would enable experimental capabilities to be evaluated in truly realistic conditions. By integrating the synthetic test environment of the test bed and the test capabilities for the live units, compelling and game-changing testing can be conducted to further drive time and cost out of demanding programmes using live, virtual and constructive (LVC) techniques.

Whilst typically designed and built to support the purposes of testing and integration, test beds can also be used for training and familiarisation (for when the real component in not yet in the deployable system). This enables tactics and doctrine to be explored and evaluated alongside the development of the component to ensure that end users usability is considered in any decision and poorer choices discounted as early and safely as possible.

As can be seen through the best design and use of test beds, the customer is empowered to build, test and operate a complex system, making the best possible decisions based upon an informed understanding of risk at the right time, thereby driving cost and time out of programmes. A test bed is a crucial tool in the T&E armoury for rapidly and safely introducing novel technologies into front line service. By evaluating and maturing a potentially complex programme of changes into a system or platform, timely and effective decisions can be made, maximising platform availability and utility for service.