This is part of a series of blog posts in which various CloudBees technical experts have summarized presentations from the Jenkins User Conferences (JUC). This post is written by Steve Harris, SVP Products, CloudBees, about a JUC session on embedded systems development.
Embedded systems development is an incredibly complex world. Robert (Robby) Martin of BMW spoke at JUC Berlin on the topic of Multi-Stage CI in an Embedded World (slides , video ). Robby spent a lot of his career at Nokia prior to coming to the BMW Car IT team. While many of the embedded systems development and delivery principles are common between phones and cars, the complexity and supply chain issues for modern automobiles are much larger. For example, a modern BMW depends on over 100 million lines of code, much of which originates with external suppliers, each of whom have their own culture and QA processes. Robby used an example scenario throughout his presentation, where a development team consisting of three developers and one QA person produce a software component, which is then integrated with other components locally and rolled up for delivery as part of a global integration which must be installed and run as part of the overall product.
The magnifying effect of an error at an early stage being propagated and discovered at a later stage becomes obvious. Its impact is most clearly felt in the end-to-end "hang time" needed to deliver a one-line change into a production product. Measuring the hang-time automatically and working to speed it up continuously is one of his key recommendations. Fast feedback and turnaround in the event of errors, and minimizing the number of commits within a change-triggered CI stage, is critical. Robby also clarified the difference and importance of using a proper change-triggered approach for CI, as opposed to nightly integration.
Robby described the multi-stage CI approach they're using, which is divided into four stages:
- DEV-CI - Single developer, max five minutes
- TEAM-CI - Single SW component, max 30 minutes
- VERTICAL-CI - Multiple SW components, max 30 minutes (e.g., camera system, nav system)
- SYSTEM-CI - System level, max 30 minutes (e.g., the car)
The first stage is triggered by a developer commit, and each subsequent stage is automatically triggered by the appropriate overall promotion criteria being met within the previous CI stage. Note how the duration, while minimal for developers, is still held to 30 minutes even at the later stages. Thus, feedback loops to the responsible team or developer are kept very short, even up to the product release at the end. This approach also encourages people to write tests, because it's dead obvious to them that better testing gets their changes to production more quickly, both individually and as a team, and lowers their pain.
One problem confronting embedded systems developers is limited access to real hardware (and it is also a problem for mobile development, particularly in the Android world). Robby recommended using a hardware "farm" consisting of real and emulated hardware test setups, managed by multiple Jenkins masters. He also noted how CloudBees Jenkins Operations Center would help make management of this type of setup simpler. In their setup, the DEV-CI stage does not actually test with hardware at all, and depending on availability and specifics, even the TEAM-CI stage may be taken up into VERTICAL-CI without actual hardware-based testing.
Robby's recommendations are worthwhile noting:
- Set up your integration chain by product, not by organizational structure
- Measure the end-to-end "hang time" automatically, and continuously improve it (also key for management to understand the value of CI/CD)
- Block problems at the source, but always as early as possible in the delivery process
- After a developer commits, everything should be completely automated, including reports, metrics, release notes, etc.
- Make sure the hardware prototype requirements for proper CI are committed to by management as part of the overall program
- Treat external suppliers like internal suppliers, as hard as that might be to make happen
Steven Harris is senior vice president of products at CloudBees.
Follow Steve on Twitter.