Are you tired of high costs for running your GitLab CI/CD jobs? If so, you’re not alone. Many developers and organizations struggle with the expenses associated with their continuous integration and delivery pipelines. However, there is a solution that can help you reduce these costs: running your GitLab CI/CD jobs on AWS Fargate.
AWS Fargate is a fully managed compute engine for Amazon ECS (Elastic Container Service) that enables you to run containerized applications without having to worry about the underlying infrastructure. Fargate can help you save money on your GitLab CI/CD jobs by allowing you to pay only for the resources you use, rather than having to maintain a fleet of servers.
In this blog post, we’ll explore how you can take advantage of AWS Fargate to reduce the costs of your GitLab CI/CD jobs. We’ll discuss the benefits of using Fargate and walk through the process of setting up and running your jobs on this platform. By the end of this article, you should have a good understanding of how to save money on GitLab CI/CD with AWS Fargate.
Replace your AWS EC2 Runner by Fargate runners
Let’s say you run your GitLab CI/CD Jobs on a t3a.xlarge runner (4 VCPU + 16 GB RAM). The global idea is to replace this “big” runner by a “micro” runner that is able to run jobs on AWS Fargate :
By using a “micro” runner on AWS Fargate, you can take advantage of the platform’s pay-per-use pricing model, which allows you to only pay for the resources you use. This can help you save money on your GitLab CI/CD jobs by reducing the resources required to run them.
How it will reduce your costs
As said above, the pay-per-use pricing model is an efficient way to reduce your costs. Let’s take a simple example : after every git push, you automatically trigger a GitLab CI/CD job which runs unit tests. This job use 4 VCPU and 8 GB RAM during 10 minutes. Your developers push 10 times a day.
If you only use a “big” EC2 t3a.xlarge runner you will pay 0,1504$ / hour * 24 hours * 30 days = 108$ / month.
If you use Fargate runners you will pay :
- EC2 t3a.micro = 0,0094$ / hour * 24 hours * 30 days = 6,8$ / month
- 10 minutes * 10 jobs with 4 VCPU = 0,0002138883333 $ / CPU / minute * 4 VCPU * 10 minutes * 10 jobs * 30 days = 2,6$ / month
- 10 minutes * 10 jobs with 16 GB RAM = 1,1$ / month
- TOTAL = 10,5$ / month
💸 108$ / month VS 10,5$ / month : you have reduced by 90% your GitLab CI/CD jobs cost 🤑
Autoscaling benefits
The architecture described before can help you scale your runners more efficiently because it implements automatically autoscaling. Autoscaling can help improve the reliability of your GitLab CI/CD pipeline. By being able to start an infinite number of runners to run jobs, you can reduce the risk of delays or failures due to insufficient resources. This can help you deliver your applications more quickly and with fewer issues.
Implementation
GitLab’s documentation does an excellent job of explaining how to set up autoscaling for GitLab CI on AWS Fargate. The process is well-documented and easy to follow, making it simple to set up : https://docs.gitlab.com/runner/configuration/runner_autoscale_aws_fargate/
Limitations
One of the biggest limitations when using AWS Fargate as GitLab CI Runner is that it is not currently possible to use “Docker in Docker” (DinD) with Fargate. DinD can be useful for tasks such as building Docker images. However, since Fargate runs containers directly on the host, it does not support the use of DinD probably for security reasons.
This can be a significant limitation for some users, as it means that they cannot use certain tools or techniques that rely on DinD. For example, if your pipeline includes building Docker images as part of the CI process, you will need to find an alternative way to achieve this when using Fargate. Some possible solutions include using an external build service such as Docker Hub or building images on a separate EC2 instance.
To go further
If your CI jobs are fault tolerant, you can use AWS Fargate Spot containers to save up to 70% of the costs of a Fargate non-Spot container. AWS Fargate Spot is a cost-effective alternative to the standard Fargate service but your workload needs to be interruption tolerant. Using AWS Fargate Spot comes with a huge constraint : at any time, AWS decides that our container must stop within 120 seconds.
Conclusion
In conclusion, AWS Fargate can be a cost-effective solution for running GitLab CI/CD pipelines. By leveraging the scalability and cost-saving features of Fargate, you can maximize the efficiency and cost-effectiveness of your GitLab CI/CD pipeline, enabling you to focus on delivering high-quality software quickly and efficiently.