Towards a New Configurable and Practical Remote Automotive Security Testing Platform

Table of Links

Abstract and 1 Introduction

2. Current Security Testing Platforms

2.1. Recent progress

3. A New Testing Platform and 3.1. Testing platform roles

3.2. Web-based remote access

3.3. Testbed setup

4. Enabled Testing Methodologies

4.1. Secure Development Lifecycle (SDL) testing and 4.2. Penetration testing

4.3. Research testing

5. Conclusion & Outlook, and References

Abstract—In the automotive security sector, the absence of a testing platform that is configurable, practical, and user-friendly presents considerable challenges. These difficulties are compounded by the intricate design of vehicle systems, the rapid evolution of attack vectors, and the absence of standardized testing methodologies. We propose a next generation testing platform that addresses several challenges in vehicle cybersecurity testing and research domains. In this paper, we detail how the Vehicle Security Engineering Cloud (VSEC) Test platform enables easier access to test beds for efficient vehicle cybersecurity testing and advanced (e.g., penetration, fuzz) testing and how we extend such test beds to benefit automotive security research. We highlight methodology on how to use this platform for a variety of users and use cases with real implemented examples.

1. Introduction

In light of recent security standards [3] and regulations [16], the automotive cybersecurity industry demands more efficient and widespread cybersecurity testing of automotive products. For example, ISO/SAE 21434 requires verification and validation (V&V) testing among the vehicle, its subsystems, and its components. Types of testing include cybersecurity functional testing, vulnerability scanning, penetration testing, and fuzz testing. This cybersecurity testing is further complicated by the current push towards software-defined vehicles, which weaves safety with security and demands faster cycles of development (and thus equally faster cycles of testing).

Unfortunately, executing security tests poses a substantial challenge as a result of limited access to hardware, the lack of configurable networks, the difficulty of connecting and configuring automotive electronic control units (ECUs) for non-specialists, and the lack of qualified personnel. Fully-enabled cybersecurity features often limit the capability of functional test engineers to be able to measure and calibrate features on the spot, which results in these critical features being disabled for the majority of the test cycle. Cybersecurity testing is usually performed after functional and environmental testing, which can make it challenging for security teams to push for patching identified vulnerabilities if significant changes are required [19].

In addition to the lack of thorough security feature testing, penetration testing on complete systems is also an area that needs much improvement. It is already difficult to find security experts who are well-versed in vehicle design and infrastructure, and the physical nature of the vehicle means often these experts must be hands-on with the entire vehicle to find the path of least resistance for a threat. The size and price tag of the targets makes penetration testing of vehicles a costly process. Likewise, the ECUs within a vehicle are complicated to set up outside of a running vehicle, especially without access to proprietary knowledge (e.g., software, wiring diagrams).

There are existing efforts to combat these challenges and make automotive security testing easier. Some solutions focus on cloud-based tests with virtual ECUs [1]; however, these are typically isolated and may not represent a whole system. Another solution is hardware-in-the-loop (HIL) testing [8], [18]. This type of testing approach is often tailored for testing individual ECUs or (at most) a couple of ECUs, and it would be difficult and costly to replicate a whole vehicle network of physical ECUs as it would require several HIL systems. In addition, custom test beds are often manually built to target a specific function, but these test beds are not configurable enough to be used efficiently and cost-effectively for all necessary security testing. Due to this effort, instrumenting these test beds and connecting them to a larger testing infrastructure has not traditionally been a common approach.

There has been some significant progress in automotive security testing. The Eclipse openDuT framework [4] aims to automate testing and validation for automotive software and applications by offering a description language for test cases, management of test cases, and orchestration of distributed ECUs. VitroBench [20] connects ECUs on a vehicle network bus to an FPGA that can launch a variety of network attacks while simulating signals from vehicle sensors. CANlay [7] offers a network abstraction layer to facilitate transmission of controller area network (CAN) data and sensor signals between (remote) ECUs. Additionally, CRATE [13] offers an alli-n-one-box hardware solution that simplifies the connection to multiple ECUs and vehicle networks and offers other convenient capabilities, such as dedicated power switching. While each of these works targets some aspect of automotive security testing, a platform that combines many of these features into a single configurable, user-friendly, and practical remote security testing platform still eludes the industry.

In this paper, we build such a testing platform and explore several testing methodologies that this platform enables for penetration testers, ECU engineers, and security researchers. We share our design and vision for this testing platform, and we posit that such a platform should offer the following key characteristics: (1) managed multiple user access control in a familiar environment, (2) access to real hardware with a realistic, configurable network, (3) hardware support at centralized locations with remote user access, and (4) ability to add custom applications for security testing.