Late last week our Engineering team cranked out two new public major releases – ElectricCommander 3.9 and CloudBees Accelerator 5.4 . Some very interesting and exciting updates in both products that our customers and users have asked for and that will allow us to expand our footprint even further - I encourage you to click on the previous links for release notes and more details.
We are very fortunate being able to use our own products for managing our internal private development cloud – how would it otherwise be possible for this engineering team with a limited set of resources (like all others, right?) to produce and deliver at the pace and quality that we’ve seen repeating itself over and over again ?
Complexity can be measured in many ways – one that we like to talk about and that we find resonates with many R&D and engineering teams is the “problem of the multiples”: multiple products, multiple projects, multiple versions, multiple sites, multiple platforms, multiple teams, multiple time zones… - the list goes on and on and for every additional dimension the engineering effort and coordination to manage this complexity grows exponentially.
Luckily the current engineering situation within CloudBees allows us to run our private development cloud at a single site at our headquarters here in Sunnyvale, CA, and our engineering teams are all working in Pacific Standard Time – so no messing around with geographical distribution following the sun like many others. Apart from that, we pretty much can check each of the multiples that I listed above – just as an example I encourage you to have a look at the installation guides for both of our products and consider the list of supported platforms: we cover and support pretty much all the commonly used Windows and Linux flavors out there, Solaris in their two flavors and a number of other platforms. Adds up to quite a matrix that we constantly need to build and test our products on!
(By the way, Eric Melski who is our chief architect for CloudBees Accelerator just very recently published an excellent article on how to manage a small subset of this platform-compatibility problem from a developer’s point of view - HOWTO: ship a custom kernel driver for Linux ) .
Another interesting example is the number of shipped versions that we still need to support and verify end-to-end upgrade compatibility with. Our typical EOL is 18 months after release and with the current pace of approximately one release every three months it quickly adds up. Consider the fact that our products support multiple backend databases (running on all the different platforms) and you should get a feeling for the level of complexity in play when just managing upgrade compatibility between versions.
So, with two product releases announced in one day, I figured it would be interesting to pull up some data from our internal private development cloud on the effects of how we actually are eating our own dog food. Since ElectricCommander has such a sophisticated yet simple way of managing metadata, this was quite simple to do and only took a few hours to generate, aggregate and summarize. Here you go:
| EA 5.4 | EC 3.9 | |
| Release cycle in days(1) : | 227 | 71 |
| Number of distinct jobs(2) : | 1314 | 630 |
| Total job-execution-time(3) : | 265d 2h 44m 21s | 71d 0h 54m 54s |
| Average job-execution-time: | 4h 50m 33s | 2h 42m 23s |
| Number of distinct job-steps(4) : | 255,222 | 101,008 |
| Total job-step-execution-time(5) : | 880d 11h 23m 20s | 243d 18h 5m 16s |
| Average job-step-execution-time: | 4m 59s | 3m 29s |
| Number of compiles(6) : | 6,565,530 | 2,108,683 |
| Number of tests(6) : | 98,687,849 | 6,717,957 |
| Average number of steps per job: | 194.2 | 160.3 |
| Average number of compiles per job: | 4996.6 | 3347.1 |
| Average number of tests per job: | 75104.9 | 10663.4 |
| Average parallelization ratio(7) : | 3.3 | 3.4 |
Notes:
1. Measured as days in between two public major releases. For those of you catching up on details, EA 5.3 has been released but for certain internal reasons it was only shipped to specific customers – prior to EA 5.4, 5.2 was the latest public version. In between the dates used in the above measurements there has been a number of minor patch releases, though all data in our private development cloud from those minor versions has been filtered out.
2. In ElectricCommander, a job is a unique invocation of a top-level fully automated process that includes all the various logic and mechanisms necessary to fulfill its purpose.
Think of this as e.g. a full multi-platform Continuous Integration or Preflight build, an end-to-end upgrade compatibility test including all the supported versions, or a complete System Test execution.
In parallel to these jobs there are all sorts of maintenance and administration jobs running in the system that were left out of this data, e.g. reporting and resource/workspace cleanups.
3. The aggregated execution time of all the jobs.
4. In ElectricCommander, a job-step can have either of two characteristics – calling custom/predefined command-line/script operations, or calling another ElectricCommander sub-process (passing in parameters to foster reusage, or to enable and model an arbitrarily complex parallelization/grouping structure). In the above data, only the command-line/script based steps actually executing work has been accounted for (in other words, only the steps actually utilizing a resource by running operations on it).
Job-steps typically invoke dynamic virtual machine provisioning, build-scripts (make, scons, ant, …), test-scripts, SCM source checkouts, resource configuration, file/copy operations, installers, deployment steps, or any other piece of logic necessary to fulfill a specific task.
5. The aggregated execution time of all the job-steps.
6. ElectricCommander will automatically extract, highlight and count the number of occurrences of given or custom matchers in a step output. It handles the most common build and test processes out there, and is also configurable to work with any custom data.
7. Total job-step-execution time over total job-execution time. In this case many of the individual build and test steps are using CloudBees Accelerator for the fine-grained file-level parallelization, so this number only represents the realized coarse-grain process-based parallelization. If we were to take into account for the parallelization also realized by CloudBees Accelerator, the ratio would be much higher.
I think there are some pretty interesting and fascinating numbers in the table above, representing and presenting a lot of value on the effects of how we actually are eating our own dog food. In more generic terms, it highlights to a large degree the transformative effects of using a fully deployed and efficient smart private development cloud, tailored to an organization’s existing development/production process yet being flexible and scalable enough to evolve with it.