Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Current Security Testing Platforms 2. Current Security Testing Platforms 2.1. Recent progress 2.1. Recent progress 3. A New Testing Platform and 3.1. Testing platform roles 3. A New Testing Platform and 3.1. Testing platform roles 3.2. Web-based remote access 3.2. Web-based remote access 3.3. Testbed setup 3.3. Testbed setup 4. Enabled Testing Methodologies 4.1. Secure Development Lifecycle (SDL) testing and 4.2. Penetration testing 4.1. Secure Development Lifecycle (SDL) testing and 4.2. Penetration testing 4.3. Research testing 4.3. Research testing 5. Conclusion & Outlook, and References 5. Conclusion & Outlook, and References 2. Current Security Testing Platforms Currently, we see two main categories of verification activities for vehicle cybersecurity that testing platforms have aimed to support. As a part of automotive engineering development activities, cybersecurity verification is done as a part of a standard V&V process. The usual procedure for aligning this with ISO/SAE 21434 Clause 10.4.2 is as follows: Verification methods: This type of testing usually follow the ASPICE engineering process [17] of Unit Testing, Integration Testing, and Qualification Testing. Cybersecurity requirements are combined with functional requirements and verified as per engineering process standards. This type of testing includes requirement-based testing, interface testing, resource evaluation, control and data flow verification, and static and dynamic analysis. The techniques used in this type of testing follow general software testing methodology and can be easily performed by a qualified test engineer.


Testing for unidentified weakness or vulnerabilities: This type of testing is unique to the field of cybersecurity and requires the test engineer to have specialized knowledge of or experience with cyber attacks (e.g., the Sam Curry attacks [2]) in order to be effective. These activities include targeted functional testing, vulnerability assessment, fuzz testing, and penetration testing. Verification methods: This type of testing usually follow the ASPICE engineering process [17] of Unit Testing, Integration Testing, and Qualification Testing. Cybersecurity requirements are combined with functional requirements and verified as per engineering process standards. This type of testing includes requirement-based testing, interface testing, resource evaluation, control and data flow verification, and static and dynamic analysis. The techniques used in this type of testing follow general software testing methodology and can be easily performed by a qualified test engineer. Verification methods: This type of testing usually follow the ASPICE engineering process [17] of Unit Testing, Integration Testing, and Qualification Testing. Cybersecurity requirements are combined with functional requirements and verified as per engineering process standards. This type of testing includes requirement-based testing, interface testing, resource evaluation, control and data flow verification, and static and dynamic analysis. The techniques used in this type of testing follow general software testing methodology and can be easily performed by a qualified test engineer. Verification methods: Testing for unidentified weakness or vulnerabilities: This type of testing is unique to the field of cybersecurity and requires the test engineer to have specialized knowledge of or experience with cyber attacks (e.g., the Sam Curry attacks [2]) in order to be effective. These activities include targeted functional testing, vulnerability assessment, fuzz testing, and penetration testing. Testing for unidentified weakness or vulnerabilities: This type of testing is unique to the field of cybersecurity and requires the test engineer to have specialized knowledge of or experience with cyber attacks (e.g., the Sam Curry attacks [2]) in order to be effective. These activities include targeted functional testing, vulnerability assessment, fuzz testing, and penetration testing. Testing for unidentified weakness or vulnerabilities: Out of the numerous cybersecurity testing activities discussed above, the verification testing methods are mostly addressed with a comprehensive engineering process. These methods stand to benefit from broader acceptance of other automotive quality-oriented disciplines, such as ASPICE and functional safety, which should lead to improvements. As the industry adopts better and more effective software quality engineering, in part, thanks to regulatory pressure, we are starting to see many tools being developed to address the activities of vulnerability assessment and fuzz testing, specifically targeting automotive use cases. Challenges. Even with the entrance of more service providers into the field, automotive cybersecurity functional testing and penetration testing, particularly for complex components or systems, still suffer from shortcomings in technical proficiency and cost effectiveness. Below, we list some of the main challenges we see in these types of testing: Challenges Technically qualified automotive cybersecurity testing specialists are hard to find,


Testing engagements are difficult to manage due to the (implicit) need to access hardware remotely, and


Fully remote penetration testing is still not realistic for most automotive use cases and difficult to repeat over time. Technically qualified automotive cybersecurity testing specialists are hard to find, Technically qualified automotive cybersecurity testing specialists are hard to find, Testing engagements are difficult to manage due to the (implicit) need to access hardware remotely, and Testing engagements are difficult to manage due to the (implicit) need to access hardware remotely, and Fully remote penetration testing is still not realistic for most automotive use cases and difficult to repeat over time. Fully remote penetration testing is still not realistic for most automotive use cases and difficult to repeat over time. As evident in other cybersecurity disciplines, such as web security or Internet-of-Things (IoT) security, a test engineer should readily have access to targets. However, getting access to a test target is currently much more difficult for automotive pentesters due to the cost, availability, complexity of modern vehicle architectures, and equipment needed for car hacking. Due to the unique skill set of pentesters and the small talent pool, most automotive OEMs and suppliers do not have the internal capability to perform fuzzing or penetration testing on every product. Penetration testing requires a completely different skill set and mindset from the traditional test engineer or development engineer. This activity is increasingly outsourced to third-party service providers. Compared to internal V&V, this outsourcing is much more difficult to manage in the event of project delays, dependency on third-party expertise and resources, concerns about data security and confidentiality, and limited hardware availability, which can result in timeline penalties and potentially cause mistakes in test target configurations. There has been some effort in addressing these issues with running software simulations of ECUs and networks on the cloud. However, these are often extremely limited and do not represent even a fraction of all attack surfaces of a modern vehicle [12]. More importantly, any hardwarebased weaknesses and vulnerabilities are often completely absent in software-only simulations. Traditionally, test beds for security projects are custom built for penetration testing with a highly customized interface. These test beds have a short lifetime (i.e., the length of a penetration testing contract), are very time consuming to build, and require specialized knowledge to interface with them. Due to the custom nature of these test beds and their fast turnarounds, it has not been worthwhile to build a generalized testing platform. As vehicles evolve to become more software-defined, a high demand is placed on testing far beyond product development and into the production and operation stages of the vehicle life cycle. Without a test bed to rerun penetration tests on, identifying new vulnerabilities during the life of a product can be difficult. This paper is available on arxiv under CC BY 4.0 DEED. This paper is available on arxiv under CC BY 4.0 DEED. available on arxiv Authors:
(1) Sekar Kulandaivel, Robert Bosch LLC — Research and Technology Center;
(2) Wenjuan Lu, Block Harbor Cybersecurity;
(3) Brandon Barry, Block Harbor Cybersecurity;
(4) Jorge Guajardo, Robert Bosch LLC — Research and Technology Center. Authors: Authors: (1) Sekar Kulandaivel, Robert Bosch LLC — Research and Technology Center; (2) Wenjuan Lu, Block Harbor Cybersecurity; (3) Brandon Barry, Block Harbor Cybersecurity; (4) Jorge Guajardo, Robert Bosch LLC — Research and Technology Center.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Abstract

Bosch

Even

Current Security Testing Platforms: Verification Methods and More

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A New Testing Platform and Testing Platform Roles

Enabled Testing Methodologies: Secure Development Lifecycle (SDL) Testing and Penetration Testing

Angular & Spring Boot - Using Serenity BDD for Integration Testing

API Integrations Can Give Your Business A Single Source of Truth

Boosting Productivity: A Guide to Implementing a New QA Role for Faster Releases

Cypress will change the way you develop

A New Testing Platform and Testing Platform Roles

Enabled Testing Methodologies: Secure Development Lifecycle (SDL) Testing and Penetration Testing

Angular & Spring Boot - Using Serenity BDD for Integration Testing

API Integrations Can Give Your Business A Single Source of Truth

Boosting Productivity: A Guide to Implementing a New QA Role for Faster Releases

Cypress will change the way you develop

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps