Understanding overall infrastructure cost and assessing its impact on value is crucial. As a rule of thumb, the level of performance and resilience you desire directly correlates with the complexity involved. However, it's important to note that achieving High Availability (HA) does come with a cost.
The introduction of High Availability in CloudBees CI has garnered positive feedback, offering several benefits when implemented on critical controllers. These benefits include reducing or eliminating downtime during infrastructure upgrades.
Furthermore, the recent addition of HA supporting hibernation it helps to reduce the cost of enabling HA. For example, if your shop operates on a 5x10 schedule, having controllers spun up (HA or not) during off-work hours when they are not in use (that adds up to 118 hours per week) becomes an unnecessary expense.
Considering the inquiries regarding the costs associated with enabling HA, let's walk through an example of infrastructure costs using CloudBees CI with HA enabled on AWS.
Cost Analysis on AWS
AWS costs can be decomposed into services. Here, we will focus on the cost of services that differ between the different ways to operate CloudBees CI with High Availability Mode (HA) enabled and non High Availability mode.
Several factors are considered when computing costs for non-HA or HA scenarios.
non-HA: Is running on multiple zones required? (EBS vs. EFS)
Either HA/non HA: store artifacts and stashes to S3
HA: What kind of event do you protect against?
Planned outages only? You can run HA with 1 replica, scaling to 2 replicas temporarily during the maintenance window.
Unplanned and planned outages? You need to run HA with at least 2 replicas distributed in different availability zones
Horizontal Scalability (HA Mode enabled): Is your controller on its knees during rush hour? With 2 replicas or more, the load is spread evenly across replicas, improving overall stability.
Please be aware that this is only a simple cost estimation. All cloud providers, like AWS, provide several cost-optimizing features depending on actual usage (reserved instance, S3 tiering, etc.). The provided costs don’t consider these potential optimizations, as they would vastly increase the matrix's complexity.
Controller Specifications
The following sample controller is used as a reference to compute the running costs:
3 vCPUs
7GB memory limit
150 GB storage + 1TB S3
Compute
With these specifications, a host VM with 4 vCPUS + 8 GB could accommodate the sample controller.
AWS provides a number of different options depending on CPU architecture as well as hardware generation.
Looking at the c7 generation, a wide range of processors are available: AMD Epyc 4th gen, AWS Graviton3, and Intel Xeon 4th gen.
CloudBees provides container images for both AMD64 and ARM64 architectures. In general, ARM64 processors provide equivalent performance to AMD64 processors but are around 30% less expensive because of their reduced power consumption.
Reference: EC2 On-Demand Instance Pricing – Amazon Web Services
C7a (AMD Epyc)
On-demand hourly cost: $0.20528
On-demand monthly cost: $147.8016
C7g (Arm-based AWS Graviton 3)
On-demand hourly cost: $0.1445
On-demand monthly cost: $104.04
Adjusting the cost for HA is pretty straightforward. Each replica has the same need, so you just need to multiply the cost for one replica by the number of replicas.
Storage
Hot storage - EBS
EBS is the preferred choice when running non-HA in a single availability zone. A multi-AZ EKS cluster can run with multiple controllers in different AZs. Snapshots can be used to migrate volumes between AZs, but they require a bit of downtime. Workloads can’t be automatically shifted between AZs during an AZ outage.
The estimated monthly cost for a gp3 150GB disk used permanently during the month and with 2x daily snapshots is $27.
Hot Storage -EFS
EFS is the preferred choice when multiple availability zones are required. CloudBees CI must also be run in HA mode.
Serverless File System | Amazon Elastic File System Pricing | AWS
CloudBees recommends running EFS in regional (Multi-AZ) with Elastic Throughput.
With this layout, there is a cost to keep stored data and access it.
Monthly storage cost: $45
Additionally, data access and throughput are charged (Based on our internal cluster example)
2.2 TB read/month ($0.03/GB) => $67.584
480 GB write/month ($0.06/GB) => $28.80
Note that these numbers have been obtained running a HA setup with 2 replicas.
Overall storage cost: $141.40
Each new replica would likely increase read access since most writes of a configuration file would likely trigger n-1 read on other replicas.
Cold storage
When using Artifact Manager on S3 and or CloudBees Workspace Caching, artifacts, stashes, and caches are stored in S3.
S3 storage includes a number of costs, depending on how frequently the data is accessed.
For 1TB of storage using S3 standard, the monthly cost would be around $23.
Note: Setting up S3 intelligent tiering could drive this cost down.
Network cost
Network costs between the different options are the same because all the traffic happens in AWS in a single region. AWS doesn’t charge for this kind of traffic.
Additional Questions
Q1: Any differences in cost between CloudBees CI Traditional vs CloudBees CI Modern?
A1: The main difference would be
EKS control plane cost ($0.10/hour, so $73/month). It is a fixed cost that doesn't depend on the cluster size.
Compute overhead for each EKS worker node since EKS related services are running on every node.
Q2: Does the throughput mode for EFS affect the cost?
A2: Yes, though the one we recommend (Elastic Throughput) is the best value for the dollar.
Conclusion
This example outlines the cost considerations for operating CloudBees CI in High Availability (HA) Mode. It highlights the need to assess various factors, including whether running on multiple zones is required, the approach to storing artifacts and stashes, and the type of outages you protect against. In HA mode, considerations include planning for planned and unplanned outages and ensuring horizontal scalability to manage load during peak times effectively. Key points to keep in mind:
It is highly recommended to store artifacts outside of the JENKINS_HOME. Whether you use a proper artifact repository manager or use the Artifact Manager on S3 plugin, this reduces operational costs for storage as S3 is a much cheaper storage solution than either EBS or EFS.
Running HA in AWS results in multiplying compute costs for controllers by the number of replicas of each controller. This is to be balanced against the total cost of running container agents to run builds which would be higher in cost.
Running EFS is about 5 times more expensive in our example, as data access has a cost. It can be mitigated by leveraging S3 wherever applicable (artifact management), resulting in a smaller JENKINS_HOME.
While this provides a basic cost estimation, actual costs can vary based on features like reserved instances and S3 tiering offered by cloud providers like AWS, which are not considered in this simple analysis.