Zero Trust GitOps

Trust no1

I love work in startups and greenfield projects, and one of the reasons is a free choice of tools and technology. It’s like a cradle for technology and innovation. A team defines comfortable and promising technology and lines of code starting to grow exponentially. Big companies a bit different story, teams need to follow guidelines and the WildWildWest approach won’t sustain. Systems and platforms must be supported by operations or DevOps teams and they can not support every new stack introduced by developer teams. If a new technology proposed to use, the team needs to train and prepare SOP (Standard Operating Procedures) to let other teams know how to support and troubleshoot the stack. There are many articles about People, Processes, Technology, and how they could help to deliver better applications, and one part of them is GitOps.

When you have one k8s cluster it’s fun and team (or teams) easily can work together on development, deployment, and operations. But when you have 100 projects, x10 clusters, and dev teams jump from one project to another, you might need to set some policies. If you contribute to OpenSource projects you might notice that before your change reaches a Merge Request phase, it goes through dozens of CI checks before the core team approve/reject it. Why? To avoid something that doesn’t follow the team’s guidelines or the way how core team operates. So CI is mature in software development, but let’s talk about CI for CD in Kubernetes.

Use Case: Unified CI for OpenShift/Kubernetes deployments

Hard to believe that organisations do not adopt clouds or Kubernetes to accelerate their business, improve operations and deliver new features. Kubernetes is great, but it’s not easy to operate at scale. You may find hundreds of stories about pitfalls and antipatterns, do not ignore them, they are like road rules, written from experience and P1 incidents by survivors.

Definitely, it will take some time before Kubernetes will be a “must-have” tool on the belt of software developers, but before that DevOps and DevSecOps need to help dev teams to setup environments. As we mentioned before, one project is fine, but what if we have 100 teams and a few Kubernetes clusters? If we will attach one DevOps to each team it will be expensive and very soon it will become a bottleneck. With help of DevSecOps, we can introduce the first frontier to check deployment files follow Kubernetes Standards.

Project overview

OpenShift Project Deployment

We have ArgoCD, which gets code from GitLab and deploys the project on OpenShift Cluster. This is sort of industry-standard nowadays, but how to ensure that code in Git is compliant and follows organisation standards? This is where we introduce GitLab CI for compliance and policy checks.

Schematic Approval Process

Based on the diagram above every commit to a new or existed branch triggers CI pipeline. If some manifest files not compliant with our standards, the commit will be rejected, so developers need to make changes until all checks pass. If the commit was successful, developers may trigger a merge request what will trigger another pipeline for Master(Production Branch). In other words, this concept moves the burden of manifest quality from DevOps to Developers. If you are OpenShift user, you can add an extra layer of compliance checks with ACM (Advanced Cluster Manager) Probably it’s a topic for another article.

git Repository structure with CI

As you noticed we need to use Git Version Control with CI capabilities. In this example, I will use public GitLab, but there are plenty of others on the market. If you would like to learn more about GitLab CI capabilities, have a look here

.gitlab-ci.yml Repositories Dependencies

It’s very important to protect .gitlab-ci.yml and avoid changes by devs. To do that, create another project and add CI configuration (a reference to another project) in GitLab CI settings:

Master CI configuration

Setting above ignore CI instructions .gitlab-ci.yml and always use instructions from the referenced repository.

CI Overview

Action

Let’s meet developers Eric, Suren, and Meshva. They are great developers full-stack developers, but new to Kubernetes. DevOps created them repository for Kubernetes deployment with enabled CI and said whatever they push into repository be deployed (Keep in mind ArgoCD) on OpenShift cluster. Kubernetes cluster can support only Kustomize and HELM packaging, so deployment from simple files not possible/allow. In case deployment files have issues, the person committing changes receives an error report.

First attempt. The team pushed deployment as a bunch of *.yaml files

Error message as HELM nor KUSTOMIZE detected

What happened? CI tried to detect that the project contains kustomization.yaml or HELM chart. This check failed. So whoever triggered merge request or create a new branch receives an email notification (or slack message) with pipeline result.

Error, no Kustomize or Helm in the repo

Second attempt. The team added kustomization.yaml file, and commit another change

Error as KubeLinter and Yaml Linting failed

What happened? The previous error had been was resolved, but now checks fail on KubeLinter and YAML linting

KubeLinter and YAML linter errors

The team needs to resolve these issues to proceed.

I stop here as the further discussion is something I leave you to try with your teams, but I hope you got an idea why we need to enable CI for GitOps.

Feel free to have a look at the source code of
Master CI Pipeline: https://gitlab.com/skozlov1/gitops-cicd-policy
Kustomize project example: https://gitlab.com/skozlov1/acm-gitops-kustomize
HELM project example: https://gitlab.com/skozlov1/acm-gitops

Conclusion

GitOps is a very powerful approach and can help to operate at scale with code development, operations, and DevSecOps teams. But to make it successful in your organisation it might require some effort and a deep understanding of the process. If you big organisation and maintain on-prem Kubernetes cluster, please invest in DevOps teams, let them do their works and talk to tenants and consider OpenShift/Kubernetes cluster as a product to plan lifecycle and features for your tenants.

[1] https://gitlab.com/skozlov1/gitops-cicd-policy/-/blob/master/.gitlab-ci.yml

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store