ci/cd
For people who have recently entered the field of programming, it is difficult to imagine that some time ago their forerunners didn't have specific tools to make their lives easier and save time, let alone their equivalents. Perhaps the best illustration of this is the development of CI/CD practices and tools. It is possible (but not advisable) to imagine commercial software development without them in 2023, but it would be highly counterproductive. The number of developed products, paid and free tools, as well as examples, is overwhelming, and the capabilities of cloud services allow for the setup of change delivery paths in the shortest possible time.
At the same time, the increasing automation of this part of the development cycle leads to the adoption of many approaches from software development to working with environments and infrastructure. In other words, the more automated our delivery becomes, the more algorithms are involved, and the more it resembles code. Consequently, the challenges we face increasingly resemble those of writing clean code.
In this article, we would like to share our experience in building CI/CD within the scope of development and compare it with how it fits into existing theory, as well as how it relates to the overall organization of development.
At the same time, the increasing automation of this part of the development cycle leads to the adoption of many approaches from software development to working with environments and infrastructure. In other words, the more automated our delivery becomes, the more algorithms are involved, and the more it resembles code. Consequently, the challenges we face increasingly resemble those of writing clean code.
In this article, we would like to share our experience in building CI/CD within the scope of development and compare it with how it fits into existing theory, as well as how it relates to the overall organization of development.
Ernest Aitiev
Python Developer
First of all, let's clarify the terms. Today, the terms CI and CD are almost always mentioned together and, like all abbreviations, lose their meaning in discussions. However:
CI (Continuous Integration)
the practice of automating the integration of code changes from multiple sources into a single software product. If we were to describe CI in two words today, it would be "working with Git." Git, source code management tools, and UI for reviewing pull requests are all that currently address CI concerns.
CD (Continuous Deployment)
the automated movement of code throughout the software development cycle. CD encompasses both what happens in CI and what happens after merging all changes together. When we talk about CD, we imply that CI is part of the process (but not vice versa).
CD (Continuous Deployment)
the automated movement of code throughout the software development cycle. CD encompasses both what happens in CI and what happens after merging all changes together. When we talk about CD, we imply that CI is part of the process (but not vice versa).
In any case, even if you don't know what is what, you are already using them. But what do you do when you're not satisfied with the current delivery process and/or need to set everything up from scratch? What should such a system be like?
After reviewing several articles and books on development organization, my answer may sound obvious, but: a good CI mechanism is code review, and a good CD system is automation (or simply put, CI - more people - better, CD - no people - excellent). To understand how accurate this formulation is, we can approach it from the opposite perspective. How can you tell if you have a bad continuous delivery system?
After reviewing several articles and books on development organization, my answer may sound obvious, but: a good CI mechanism is code review, and a good CD system is automation (or simply put, CI - more people - better, CD - no people - excellent). To understand how accurate this formulation is, we can approach it from the opposite perspective. How can you tell if you have a bad continuous delivery system?
1
Every release day or deployment of changes is a "big day" for developers. Your team spends a lot of time, effort, and energy just to deploy the latest changes to a new environment. If the backend team needs to explain the structure of requests and responses of new endpoints to frontend and mobile developers during the deployment, you have problems
2
In a development team, only few people know how to do the delivery correctly.
3
Delivery becomes a collection of knowledge and artifacts, both written and verbal, without which no one knows how to do it properly. There is a repository where a bunch of instructions for all possible scenarios is stored, but it looks chaotic. The project repository also contains numerous undocumented scripts responsible for significant operations upon closer inspection
4
Manual testing during integration of changes. A bad scenario: you have no tests, or your tests are not indicative. When you make changes and want to validate them, you have to deploy the changes to the environment and manually test what you've done
5
Infrastructure diversity across environments (development, staging, production). All your environments should have the same configuration of access parameters, folder structure, and software (such as the OS). What does this mean? If we can SSH into the development server using a password, we should be able to do the same in any other environment. If the project files are located in a specific path on one environment, they should be in the same path on all environments. The same operating system should be installed on all environments. Otherwise, you create nuances of logic that no one thinks about, but which become difficult to automate. I recall the exclamation of one of the new DevOps specialists: "We have that too?!" In the end, even if you have a well-established continuous delivery system, pipelines may occasionally fail because some components behave differently in different environments due to the described differences
6
Delivery time. Sometimes it's helpful to simply answer the question: where are we spending all the time integrating changes and delivering them? On what? Counting time is perhaps the simplest criterion, as noted in many resources on building CI/CD. If deploying to production takes more than a few minutes without any objective reasons, then perhaps your processes need improvement (objective reasons could include, for example, the time it takes to run unit tests, which cannot be sped up due to computational limitations)
Now let's talk about what an ideal delivery and continuous integration system should look like
If we consider CI/CD as an essential tool for developers, then why not remember and rephrase: programmers can have friendly tools too. Moreover, they can provide them for themselves. Besides, hardly anyone enjoys struggling with infrastructure, which means that the ideal CI/CD system is a big red "Launch" button.
To build such a system for integrating and delivering changes, you need to abstract yourself as a coder and imagine that your CI/CD is one big application the goal of which the application to the specific point in the shortest possible time. Like any application, CI/CD should have its dedicated stages (layers of business logic, validation, and result delivery), and the clearer these stages are delineated, the easier it is to operate and modify them. Thus, if your application has a precise working algorithm, defined operations, and input and output data stages, it becomes easier to build, modify, and launch everything.
The book "Continuous Delivery: Reliable Software" combines these concepts with the questions and nuances that exist at the higher level of development process organization and introduces the term "deployment pipeline." The term can be deciphered as "automated realization of your application delivery process from version control system to the hands of users."
To build such a system for integrating and delivering changes, you need to abstract yourself as a coder and imagine that your CI/CD is one big application the goal of which the application to the specific point in the shortest possible time. Like any application, CI/CD should have its dedicated stages (layers of business logic, validation, and result delivery), and the clearer these stages are delineated, the easier it is to operate and modify them. Thus, if your application has a precise working algorithm, defined operations, and input and output data stages, it becomes easier to build, modify, and launch everything.
The book "Continuous Delivery: Reliable Software" combines these concepts with the questions and nuances that exist at the higher level of development process organization and introduces the term "deployment pipeline." The term can be deciphered as "automated realization of your application delivery process from version control system to the hands of users."
The principle of stage separation in this flowchart closely resembles the structure of a classical CD pipeline, not only for us but also for others, with some adjustments:
Commit stage
This stage involves committing changes to the shared repository, conducting unit tests, running certain API tests, integration tests, and code analysis, as well as performing a single build for the entire iteration (Unit Testing + Build)
Acceptance stage
This stage includes necessary acceptance tests. Here, the new build undergoes testing with basic acceptance tests (UI Tests)
UAT (User Acceptance Testing) stage
This stage involves thorough acceptance testing
Capacity stage
This stage focuses on testing the application's capabilities
Production stage
This stage marks the release
In addition to the automation process itself, the author emphasizes that a good CI/CD system implies:
1
Automating everything possible: Often during development, teams leave some details to their own discretion, such as verifying the correct transmission of all environment parameters for service deployment. Trust me, this never goes unnoticed
2
Short release cycle: The intervals between each batch of changes in the development environment and the final product should be as short as possible. This helps ease the tracking of the number of bugs encountered after deployments. With an already automated CI, this becomes relatively easy
3
Short feedback cycle provided at all levels:
- From developers to PM or product owner during development.
- From reviewers to developers when creating a pull request.
- From developers to the team once the pull request is accepted and the changes are deployed.
- From customers to the development team when changes are deployed to staging.
- From everyone to everyone when changes are released.
Applying this knowledges to our processes, several points to attention were outlined
- You can never have too much feedback. This has been proven through trial and error, but it also aligns with the requirements for building a proper CI/CD system.
- Do not underestimate smoke tests: Every incident, every initial launch, every DevOps engineer has asked the question, "I've finished it. How can we verify that it works?" Even though they may be simple and primitive, smoke tests participate in all logical operations, yet they are often forgotten.
- Multiple approaches to acceptance testing: The book distinguishes between acceptance tests conducted at the Acceptance stage and user acceptance testing (UAT). Most development and testing teams experience UAT but often struggle to reach a definitive conclusion. The Acceptance stage focuses on basic scenarios, where the fine nuances of interface operations are not critical, while UAT involves comprehensive testing of the entire system.
Thus, building a good CI/CD system entails
1
Building once
2
Automating everything possible
3
Making all deliveries across environments as identical as possible
4
Including smoke tests
5
Having an environment similar to production (not development)
6
Making stages atomic and idempotent (each stage should perform a specific operation and produce the same result when triggered again)
With the abundance of existing tools available today, we work with GitLab CI/CD because:
- It allows configuring the pipeline as code in YAML format.
- It provides a monitoring system with an API that can be used to retrieve information about deployments.
- It offers the ability to configure and structure deliveries for multiple environments.
- It enables scheduling jobs or tasks. For example, if backups need to be taken every month, GitLab provides this capability.
- It allows storing configurations, including environment variables required for each project and build, without which the build won't be deployed.
- GitLab Runner serves as the pipeline executor, doing the work for us. It requires minimal setup and can be connected to a project in just 5 minutes. Isn't that convenient?
© 2023 Attractor Software
