Trust no1
I love work in startups and greenfield projects, and one of the reasons is a free choice of tools and technology. It’s like a cradle for technology and innovation. A team defines comfortable and promising technology and lines of code starting to grow exponentially. Big companies a bit different story, teams need to follow guidelines and the WildWildWest approach won’t sustain. Systems and platforms must be supported by operations or DevOps teams and they can not support every new stack introduced by developer teams. If a new technology proposed to use, the team needs to train and prepare SOP (Standard Operating Procedures) to let other teams know how to support and troubleshoot the stack. There are many articles about People, Processes, Technology, and how they could help to deliver better applications, and one part of them is GitOps.
When you have one k8s cluster it’s fun and team (or teams) easily can work together on development, deployment, and operations. But when you have 100 projects, x10 clusters, and dev teams jump from one project to another, you might need to set some policies. If you contribute to OpenSource projects you might notice that before your change reaches a Merge Request phase, it goes through dozens of CI checks before the core team approve/reject it. Why? To avoid something that doesn’t follow the team’s guidelines or the way how core team operates. So CI is mature in software development, but let’s talk about CI for CD in Kubernetes.
Use Case: Unified CI for OpenShift/Kubernetes deployments
Hard to believe that organisations do not adopt clouds or Kubernetes to accelerate their business, improve operations and deliver new features. Kubernetes is great, but it’s not easy to operate at scale. You may find hundreds of stories about pitfalls and antipatterns, do not ignore them, they are like road rules, written from experience and P1 incidents by survivors.
Definitely, it will take some time before Kubernetes will be a “must-have” tool on the belt of software developers, but before that DevOps and DevSecOps need to help dev teams to setup environments. As we mentioned before, one project is fine, but what if we have 100 teams and a few Kubernetes clusters? If we will attach one DevOps to each team it will be expensive and very soon it will become a bottleneck. With help of DevSecOps, we can introduce the first frontier to check deployment files follow Kubernetes Standards.
Project overview
We have ArgoCD, which gets code from GitLab and deploys the project on OpenShift Cluster. This is sort of industry-standard nowadays, but how to ensure that code in Git is compliant and follows organisation standards? This is where we introduce GitLab CI for compliance and policy checks.
Based on the diagram above every commit to a new or existed branch triggers CI pipeline. If some manifest files not compliant with our standards, the commit will be rejected, so developers need to make changes until all checks pass. If the commit was successful, developers may trigger a merge request what will trigger another pipeline for Master(Production Branch). In other words, this concept moves the burden of manifest quality from DevOps to Developers. If you are OpenShift user, you can add an extra layer of compliance checks with ACM (Advanced Cluster Manager) Probably it’s a topic for another article.
git Repository structure with CI
As you noticed we need to use Git Version Control with CI capabilities. In this example, I will use public GitLab, but there are plenty of others on the market. If you would like to learn more about GitLab CI capabilities, have a look here
It’s very important to protect .gitlab-ci.yml and avoid changes by devs. To do that, create another project and add CI configuration (a reference to another project) in GitLab CI settings:
Setting above ignore CI instructions .gitlab-ci.yml and always use instructions from the referenced repository.
CI Overview
Action
Let’s meet developers Eric, Suren, and Meshva. They are great developers full-stack developers, but new to Kubernetes. DevOps created them repository for Kubernetes deployment with enabled CI and said whatever they push into repository be deployed (Keep in mind ArgoCD) on OpenShift cluster. Kubernetes cluster can support only Kustomize and HELM packaging, so deployment from simple files not possible/allow. In case deployment files have issues, the person committing changes receives an error report.
First attempt. The team pushed deployment as a bunch of *.yaml files
What happened? CI tried to detect that the project contains kustomization.yaml or HELM chart. This check failed. So whoever triggered merge request or create a new branch receives an email notification (or slack message) with pipeline result.
Second attempt. The team added kustomization.yaml file, and commit another change
What happened? The previous error had been was resolved, but now checks fail on KubeLinter and YAML linting
The team needs to resolve these issues to proceed.
I stop here as the further discussion is something I leave you to try with your teams, but I hope you got an idea why we need to enable CI for GitOps.
Feel free to have a look at the source code of
Master CI Pipeline: https://gitlab.com/skozlov1/gitops-cicd-policy
Kustomize project example: https://gitlab.com/skozlov1/acm-gitops-kustomize
HELM project example: https://gitlab.com/skozlov1/acm-gitops
Conclusion
GitOps is a very powerful approach and can help to operate at scale with code development, operations, and DevSecOps teams. But to make it successful in your organisation it might require some effort and a deep understanding of the process. If you big organisation and maintain on-prem Kubernetes cluster, please invest in DevOps teams, let them do their works and talk to tenants and consider OpenShift/Kubernetes cluster as a product to plan lifecycle and features for your tenants.
[1] https://gitlab.com/skozlov1/gitops-cicd-policy/-/blob/master/.gitlab-ci.yml
