DevOps: A Holy Grail for Automotive or a Budget Buster?

Co-author of ‘DevOps: A Holy Grail for Automotive or a Budget Buster?’ is Vladimir Lebedev — DevOps Evangelist and Practitioner, CEO of DevOps consultancy — Triangu

What’s it like to be an automotive company today? Outsourcing of every aspect of operation, has an automotive company become a brand management/marketing agency with little exposure to technology? Not exactly.

The automotive industry has a stellar one hundred years of experience in technology, design and, most importantly, manufacturing. Software became an indispensable part of vehicles only few decades ago. In many aspects, it’s similar to mechanical components: both consist of interconnected systems with inputs and outputs. Like any mechanical part, software goes through a standard development cycle: design, development, testing in an isolated environment, and production. But there’s still a significant difference between software and mechanical components.

Software boasts zero cost-of-copy and anticipated infinite time-to-failure, as there are no deteriorating parts. This leads to one common misconception, as many don’t take the software development lifecycle seriously: considering software functionality to be easier to create, deliver, and manage. Let’s see how to avoid this mistake.

Why is software a reason for production failures?

Automotive vendors, both OEM and Tier 1, know like no other companies that it takes years to build a brand’s reputation and only days to lose it. Take the Volkswagen Dieselgate: three years have passed since the scandal, but it’s still not over.

As software extends the car architecture, its quality becomes critical for the entire assembly. Erroneous code can impact vehicle performance and safety, which may be completely devastating for car manufacturers. Fiat Chrysler had to recall 1.25 million pickup trucks worldwide to address a coding error, and General Motors recalled nearly 4.3 million cars because of a software bug capable of preventing airbags from deploying in a crash.

Software-related vehicle recalls

Even though the software industry has no means to reach production quality similar to Six Sigma without making software Mars-rover-expensive, we have a decent alternative — DevOps in automotive. This methodology combines software Development and Operation practices, spans through the entire software delivery pipeline, and delivers what most OEMs can only dream of. Companies that adopted DevOps for automation report significant benefits, including:

• continuous product quality improvement;
• decreased time-to-market;
• boost in customer satisfaction;
• more reliable releases;
• accelerated productivity and efficiency;
• more reliable and qualitative product development with fast experimentation.

DevOps initiatives also change the way companies work, triggering new collaborative mechanisms for IT operations, developers, and quality assurance teams during the software development lifecycle. Getting these groups to work smoothly together is a critical challenge in enterprise DevOps and automation adoption, especially in automotive.

Why hasn’t DevOps paved its way to automotive yet?

DevOps strives to, but several things make it a bit cumbersome.

Collision of approaches

First, the automotive R&D lifecycle uses a specification-based A to Z waterfall approach that’s controlled at each step. Software quality systems like A-Spice attempt to merge this approach with DevOps by setting up DMAIC (Define, Measure, Analyze, Improve, Control) processes. But DevOps for automotive prospers when paired with Agile — the iterative development methodology for project management that eliminates planning in advance.

Agile and DevOps automation approach software as an ever-changing realm and deal with such external factors as user expectations, environments, and security challenges. This combination works for SaaS but is difficult to map to the automotive sector, which is used to rigorous planning inherited from the just-in-time delivery management. Building a car in iterations is a weird idea: you may stick wheels on a frame during the first sprint, but that hardly makes it a car.

Embedded software from a galaxy far, far away

The second thing that gets in the way of adopting DevOps in the automotive sector is deeply embedded in the automotive software stack. Today’s vehicle infrastructure exceeds 100 different electronic control units (ECUs) as well as accompanying software for production maintenance and an advanced user experience, including human-machine interfaces and all kinds of companion apps.

Automotive embedded systems

Embedded software has many limitations:

• low compute and memory footprints
• diverse and disconnected tools
• real-time response requirements
• connectivity issues

Despite the industry’s attempts to minimize and standardize these ECUs, embedded software-based control systems still retain a competitive edge, as they advance functionality with marginal increases to the cost and shorter bill of materials.

Embedded software has to work with a low footprint, leaving no room for orchestrators or extras like hypervisors, loaders, or maintenance-related code. However, with the rapid technology progress, modern hypervisors and orchestrators are getting more lightweight today, and who knows, we might even see them in ECUs one day — this would allow the embedded practice to get all the benefits of DevOps automation.

DevOps toolchain loops

The automation challenges

Real-time embedded software systems have to ensure that response times are predictable and aren’t sensitive to the system condition or collision of events. In most cases, to design and verify such systems, very specific software stacks and tools, usually created with no ideas of process automation in mind, are used. Thus, leaving no way for introduction of any new process improvements (like DevOps automation testing) without completely changing software stack.

How are deployments handled?

Are there any chances for DevOps to overcome these pitfalls?

Let’s start with connectivity. Even the first-tier countries are far from 100% coverage of car networks. The launch of 5G comes in handy to solve this issue.

Software over-the-air connectivity

With low-latency services that target less than a millisecond, 5G will be a disruptor for the automotive industry. Its ultra-speed and extensive coverage will fuel such mission-critical functionality for connected driving as vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) and vehicle-to-everything (V2X) communication with over-the-air updates. Capable of processing massive loads of data quickly, 5G will power autonomous driving and ensure vehicle safety along with enhanced comfort.

In a DevOps context, a 5G network meets the demand for velocity, enabling faster process optimization and flow control across the entire development cycle. Focused on agility and automation, 5G-powered DevOps will significantly increase the efficiency of the software delivery cycle through continuous integration/continuous deployment (CI/CD) practices. And once implemented in cloud-based software development, CI/CD will dramatically reduce the duration of the DMAIC loop.

Continuous delivery enables two versions of development, delivery, and deployment infrastructures, allowing both of them to support different requests. CD also enables advanced A/B implementation scenarios with independent production instances that compete for user’s adoption.

Inherit best practices from open-source

The standardization of APIs and open-source tools are another set of drivers behind DevOps adoption in automotive. For the first time in its history, the automotive industry is facing an industry that doesn’t want to follow automotive standards yet forces automotive companies to follow automation and DevOps practices. The AutoSAR Classic to Adaptive platform transition is a vivid example of the reverse track from a proprietary automotive to an IT-derived solution.

Additionally, many automotive vendors already benefit from open-source software. Take the Linux Foundation’s Automotive Grade Linux (AGL), an open-source platform that could be an answer to today’s fragmented, often confusing automotive operating ecosystem. This all-embracing platform may turn into an industry standard and fuel innovation across autonomous driving.

Dynamic system scaling

Automotive-grade virtualization for onboard ECUs and the infotainment stack creates a lot of opportunities for simulated testing, dynamic system resource allocation, and performance scaling. Combining the DevOps world with automotive, we can automate the ongoing processes of running tests with embedded devices and sensors in a car on a motorway at high speed. DevOps handles software integration and continuous deployment assists automakers in predictive maintenance and recovery from errors.

There are still challenges to address in the development of real-time and hardware systems. However, the rapid development of field-programmable gate arrays, hybrid application-specific integrated circuits, and systems on a chip (SoCs) is promising. This progress allows us to finally integrate real-time and hardware systems into a single simulated environment where they will be seamlessly managed through the same DevOps approach.

Are there any drawbacks?

Even though we stand for the DevOps culture in automotive, it wouldn’t be fair to leave the shortcomings unnoticed.

First, you’ll find neither a standardization body for DevOps nor a hope that one will ever exist. Yes, my dear automotive colleagues — nobody takes care of DevOps definitions, standard processes, or tooling. And this may sound like a nightmare for an industry with over a hundred years of the standardized development process.

Although the DevOps methodology is reaching its productivity plateau, the diversity of approaches, tools, and lack of software standardization makes its adoption for automotive cumbersome. Plus, the divergence of development and operational processes makes it difficult to break apart the Dev and Ops parts. This is the opposite of the long-standing R&D to production lifecycle concept and is often rejected by automotive vendors.

For legacy enterprises, the lack of a referenceable standard, verification, and certification body is considered a great risk. Even A-Spice seems to be driving the whole industry in the opposite direction from Agile, it doesn’t contradict this approach. Still, the industry should have a lot of conceptual flexibility and knowledge of both Agile and A-Spice to merge them successfully in software production. But with the highly adaptive nature of DevOps, standardization won’t burden its concept.

Drive innovation in automotive

Building a DevOps practice for the automotive industry requires strong visionary skills to match values, technologies, and production perspectives. It’s increasingly important to OEMs and Tier 1 vendors to partner with strong software service providers with hands-on experience of DevOps setup. With no DevOps theory or certificates available, only a solid background and expertise can prove a company’s professionalism. But if well-implemented, DevOps can benefit automakers with short time-to-market, lower development costs, minimum service breakage, and constantly improving quality.

Sounds like a Holy Grail promise, doesn’t it?