You are currently not logged in. Login

TIGARS

TIGARS (Towards Identifying and closing Gaps in Assurance of autonomous Road vehicleS) was an international research project aimed at investigating the assurance gaps and challenges for first generation autonomous systems and research into techniques and engineering processes for addressing them. The project was joint funded by the Assuring Autonomy International Programme (AAIP) and UK Centre for the Protection of National Infrastructure (CPNI).

The project was led by Adelard in collaboration with the following institutes:

 

Members of the project team pictured at the National Innovation Complex, Nagoya University Japan

Research

We argue that assuring trust and trustworthiness through argument-based mechanisms, specifically, the Claims, Arguments, and Evidence (CAE) framework, allows for the accelerated exploration of novel mechanisms that could lead to advancements in the assurance of disruptive technologies.

This assurance approach is informed by an understanding of engineering processes and technical analysis for developing and assuring autonomous vehicles addressing resilience, formal verification, static analysis, security, and other aspects. Our project

  • Identified current autonomous systems engineering approaches and their assurance gaps.
  • Investigated how to address the assurance gaps with new analysis approaches based on verification of machine learning in both benign and adversarial environments, using simulation and test strategies, and an evaluation of defence in depth.
  • Provided recommendations to regulatory and policy organisations and standards developers on a principles-based framework to address autonomy, as well as on a near-term interpretation of existing standards.

The project also conducted experimental trials with real life demonstrator systems, such as the TEV (TIGARS Experimental Vehicle) and Donkey car.

 

The projected furthered work in the international standardisation area and the team have held regulatory workshops in the UK and Japan.

Outcomes and recommendations

The project concluded the research studies by reporting technical results, guidance and recommendations as a series of TIGARS Topic Notes (TTNs) - short briefing papers (7-12 pages) – on key areas.

The TTNs address the challenges faced in the current landscape regarding the noted attributes for Machine Learning (ML) based autonomous vehicles and systems. Additionally, we discuss potential solutions and recommendations proposed by a varied set of literature as well as preliminary research that we have carried out.

In addition, we also provided an overall summary of our research and longer supporting papers on experimentation, standardisation and regulatory guidance, sociotechnical perspective and SysML stochastic modelling.

For further information, please expand the section headings below.

 

Assurance cases and approaches

Developing an assurance strategy should be a key part of the overall design approach and integrated into the overall lifecycle. The assurance approach should be commensurate with the different risks and consistent across them, e.g., by adopting an outcome-based risk informed approach.

Novel assurance approaches (e.g., articulated using CAE) exclusive to ML and AI-based systems should be developed to identify areas to focus on and establish how they impact both the system and its assurance. It can help define and evaluate the reasoning and evidence needed.

Key claims should address the high-level functional and ethical principles such as those from the EU Expert Group report [9] and the Sherpa project [10]. These principles can be used to shape and define system or service level properties.

An assurance case for autonomous systems should at a minimum address the points below:

  • what the system is and in what environment and ecosystem it will operate in
  • how much trust in a system is needed, considering interdependencies and systemic risks
  • whether it is sufficiently trustworthy to be initially deployed
  • whether it will continue to be trustworthy in the face of environmental changes, threat evolution and failure

Structured argumentation for safety cases (and more generally assurance cases) needs more emphasis on reasoning and evidence, if the cases are to be sufficiently robust and acceptable. We have characterised a new CAE based assurance framework to achieve this, which would utilize evidence extracted from V&V, defence in depth, and diversity techniques.


Resilience and safety analysis

Autonomous vehicles (AV’s) deployed as part of an ecology of systems that deliver services (e.g., mobility service), must appropriately define resilience and safety requirements.

To develop the dependability or resilience of a service, discussions of requirements and assurance should start from a service level, not from systems or components level. Discussions should include vehicle capabilities, infrastructure sensors, cloud systems, etc. Resilience requirements should also be derived at service level, and then assigned to each system or component. The system should have high-level safety, security and resilience requirements. A systems theoretic approach (e.g., using Systems Theoretic Process Analysis - STPA) combined with an impact of variability using Functional Resonance Analysis Method (FRAM), can be useful to addressing these requirements.

Open systems dependability perspective

For AVs to be accepted socially, stakeholders need to have confidence, before they are deployed, in how they are going to adapt to changes post-deployment. This is particularly important when considering security requirements, as AVs are deployed in threat-filled environments that keep changing.

The future behaviours of AVs should be assured systematically through Open Systems Dependability (OSD) deployed on the system’s lifecycle.

Policy makers and regulators should enable and promote adoption of standardised OSD by AV manufacturers, operators and users. They play a key role in ensuring such collaboration is possible across legitimate AVs. Sharing of information, including assurance, is a major concern. Appropriate policies can incentivise AV manufacturers and operators to participate in a transparent evaluation and assurance regime which in turn strengthens users’ confidence in Robotics and Autonomous Systems (RASs’) future behaviours.


Verification and Validation (V&V) techniques

It is crucial to strategize the use of V&V (and defence in depth and diversity techniques) through the lens of an assurance approach, in particular, CAE, to identify the role of such methods and how they complement other approaches. We provide recommendations regarding their use below:

Formal Verification

ML-specific properties such as pointwise robustness not only fail to note real-world examples, but also how state-of-the-art verification techniques can be applied to real-time systems. With regard to the safety assurance of an autonomous system, pointwise robustness fails to support or provide evidence for system robustness.

We thus recommend:

Creation of relevant safety specifications unique to ML algorithms, with corresponding mathematical frameworks. The noted specifications must contribute to the assurance of an AI system, specifically, the context of an assurance case (i.e., CAE). Some ML algorithms (e.g., vision) may be intrinsically unverifiable against the properties which are of interest to the safety of an autonomous vehicle, however, other properties can in principle be formulated for other types of ML systems (e.g., planning) in autonomous vehicles.

Collaboration between ML and verification researchers resulting in deep learning systems that are more amenable to verification. Novel formal verification techniques are needed which can address the newly defined specifications.

Static Analysis

The ML lifecycle relies heavily on data processed in a complex chain of libraries and tools traditionally implemented, often in Python. It has been demonstrated that implementation in these systems may propagate and affect the accuracy and functionality of the ML algorithm itself. We have demonstrated that static analysis tools can be used to build confidence in supporting systems. However, the verification of existing ML software infrastructure may pose particular challenges.

We thus recommend:

Creation of novel formally-based static analysis techniques addressing Python, and more generally, dynamically typed languages, given that they are not currently available. Formal methods can have a strong role in ensuring provenance of training and data processing.

Organisations should consider rewriting any deployed safety critical software in a verifiable language if the appropriate analysis tools for Python are unavailable.

Organisations must understand the extent to which existing integrity static analysis tools can contribute to the confidence of the development of ML algorithms. The complexities arising from choice of implementation language, e.g., issues from using C or C++, should be well understood.

Simulation

The roles of the different simulation variants should be specified and justified, and confidence in the simulation environment needs to be established. This may include confidence in the modelled behaviour of the tested system, as well as confidence in the software running the simulation.
Attempts should be made to make the tests as repeatable as possible, however, if this is not possible the impact in confidence on the test results must be considered.

Adjustments in system behaviour may be needed to accommodate the simulation environments and these will need to be justified so that test evidence can be used in the overall assurance case.

The use of diversity to improve reliability and safety is a sound principle. In particular it should be used to achieve higher dependability of safety mechanisms. The stakeholders for a mobility service or AVs should undertake a review of defence in depth and define a diversity and defence in depth strategy balancing the advantages of diversity with possible increases in complexity and attack surface.

Diversity should be considered within the construction of a system’s architecture to reduce the trust needed in a single ML component. Independence of failures should not be assumed and failure correlation should be considered based where possible on experimental data. An architectural approach which limits reliance on sub-components of the system that need to be highly trusted (e.g., ML algorithms) should be taken.

Safety monitor architectures should be considered to reduce the trust needed in ML components as they monitor both the state of the environment and the AV. Where feasible, they can be used to gain performance and safety benefits of deploying complex ML components, whilst mitigating the risks of using such technologies.


Diversity and defence in depth

The use of diversity to improve reliability and safety is a sound principle. In particular it should be used to achieve higher dependability of safety mechanisms. The stakeholders for a mobility service or AVs should undertake a review of defence in depth and define a diversity and defence in depth strategy balancing the advantages of diversity with possible increases in complexity and attack surface.

Diversity should be considered within the construction of a system’s architecture to reduce the trust needed in a single ML component. Independence of failures should not be assumed and failure correlation should be considered based where possible on experimental data. An architectural approach which limits reliance on sub-components of the system that need to be highly trusted (e.g., ML algorithms) should be taken.

Safety monitor architectures should be considered to reduce the trust needed in ML components as they monitor both the state of the environment and the AV. Where feasible, they can be used to gain performance and safety benefits of deploying complex ML components, whilst mitigating the risks of using such technologies.

 

Security-informed safety

Security-informed safety (SIS) should be addressed at all stages of the innovation cycle from conceptualisation, experimentation, and prototyping through to production.  A security-informed hazard analysis should be undertaken during development. The hazard analysis should be reviewed periodically during operation or when a safety related component has been updated or if additional threat and vulnerability information has been identified.

The UK PAS 11281 can be used to systematically consider security through addressing: security policy, organization and culture; security-aware development process; maintaining effective defences; incident management; and safe and secure design, all contributing to a safe and secure world. The PAS can be applied progressively during the innovation lifecycle of an AV or RAS, and adapted to provide a project specific implementation.

Security-informed safety cases are still novel, and the experience of developing and integrating security issues into the safety analysis should be captured. In the industry as a whole, more training and expertise for SIS analysis is required, as many decisions rely on expert judgement. Although methodology that has been developed in other sectors can also be applied to AVs, AI and ML based technologies will provide novel security challenges that must additionally be addressed.


Standardisation and guidance

Duplication of standardisation work on similar topics should be reduced to a minimum. Efforts to prevent duplication have been ongoing in the international standardisation community, but we have observed that AV and RAS relevant topics often have duplicate standards, which may not be aligned. An example of possible duplication is in risk management.

An authoritative and introductory guideline covering necessary knowledge for AVs and RASs should be developed for new entrants to this arena. Particularly, guidelines should include surveys on foundational standards of the field. Many IT companies are entering into the market without the experience of the traditional manufacturers. The current lack of such overall guidelines can lead to IT stakeholders to overly concentrate on their strength within a particular area without essential knowledge of AI/ML or safety. The recommended guideline would help ensure that innovative technologies and traditional engineering and assurance practices are aligned.


 

The notes are available in two bundles.
 
  • Part 1 addresses: Assurance-overview and issues, Resilience and Safety Requirements, Open Systems Perspective and Formal Verification and Static Analysis of ML Systems. https://arxiv.org/abs/2003.00789 (TTN part 1)
  • Part 2 addresses: Simulation and Dynamic Testing, Defence in Depth and Diversity, Security-Informed Safety Analysis, Standards and Guidelines. https://arxiv.org/abs/2003.00790 (TTN part 2)

Future

The autonomous systems field is international and has a wide variety of players of differing maturity. Some entrants are unfamiliar with classical safety engineering, yet have expertise related to AI and ML-based systems. Others are mature and familiar with classical assurance approaches but lack a grasp on the challenges autonomy brings about. Given this wide range of maturity and backgrounds, the TIGARS outputs aim to address a range of different audiences. We hope they will be accessible and of interest to both engineers as well as policy makers.

We will try to exploit the results of the project on the evaluation and development of RASs and on the development of assurance methodologies. The results will feed into projects on autonomy in the UK and a possible joint Japanese/UK project on assuring a real Mobility as a Service (MaaS) application.

Ready to start a conversation?
Interested in keeping up to date with the latest news?