Incoming call from… CISO
If you are a project lead and you see an incoming call from CISO or his deputy it’s not a good sign. And also I hope you never paid BTC to bad boys who stolen private information of your customers because one of your devs forgot to disable admin/admin on public-facing application or used an old and vulnerable JS library. This is my one of my favorite stories about how easy people can get access to sensitive data, just because devs use libraries without source code review. Devs are brave and creative, they want to write good and efficient software and try new things. If you forbid them to do that, you are slowing an innovation or can lose good people. But new tech might be good and dangerous at the same time. If you are a team lead and work with Kubernetes I assuming your teams use only clean UBI images and well know libraries. But how to breed creativity/innovation and keep your job and reputation at the same time? There are many ways to do that, but today we will speak about DevSecOps.
DevOps + Sec = Dev Sec Ops
DevOps is not a buzzword anymore, you or your company do have them and they bring enormous benefits for the business. These warriors help to automate product delivery with CI/CD pipelines and let devs just push a code and get reports in Stash/Rocketchat/Telegram about build/deployment status. But how is about to add Security bit to ensure that automation delivers secure products? That’s how DevSecOps appeared. I’m not very good in describing methodologies, so here is a good explanation from Red Hat what is DevSecOps.
Let’s assume our team λ started to work on a simple application that has front end and backend deployed on OpenShift cluster. Front-end is internet-facing with an exposed HTTPS Route and a simple backend. Also let’s imagine that our company is adopting GitOps, so developers have limited access to the cluster (at least on Staging and Prod) namespaces. Therefore they can not deploy anything via
oc commands. And to unify processes DevOps defined a structure of repositories that each team has to follow:
│ ├── Dockerfile
│ ├── app.py
│ └── requirements.txt
│ └── be
│ ├── deployment-config.yaml
│ ├── image-builder.yaml
│ └── service.yaml
Where app folder used for application source code and Dockerfile, and kubernetes for project manifest files compatible with Kustomize. We have 2 sub-teams for FrontEnd and Backend and they can make commits whenever they want without asking permission. In this case, minimum delivery pipeline (which is Tekton) might look like this
NB: This is a simple case, so we omit all integration tests in CI/CD, for sure in real life you will have more steps like unit tests and integration testing. Also in real life, we won’t have 2 parallel tasks in one pipeline
If you want to have a look at the pipeline to follow our further discussion, please feel free to apply it to your cluster
# Deploy simple Tekton pipeline
oc apply -k "https://github.com/mancubus77/devsecops-demo-tekton.git/?ref=simple"# Run task
tkn pipeline start build-and-deploy \
I hope you had a look at repository content before :-)
So, we have delivery pipeline and each time devs make a commit git triggers CI/CD pipeline, and project will be built and deployed on the cluster. So far we solved a few challenges:
* We defined unified repository structure
* We are compatible with GitOps
* We can separate teams and let them work individually
OK, what is about Security?
Good question. Nothing stops our Devs from testing new technologies and libraries, but how do we know that they are secure and harmless? The answer is simple we do not, as we do not check repo nor code quality. So any mistake or vulnerability can jeopardize or compromise a project. To avoid this, let add a few steps for our delivery pipeline:
- Code scanning of backend code via Python Bandit
- Code scanning of Node code with njsscan
- Image scanning with Trivy
- Manifest scanning with kube-linter
Therefore, we not only automating CI/CD we also ensure that security vulnerabilities or dodgy code not being deployed in a staging/production environment. Let’s have a look at how the pipeline looks like with DevSecOps’ish approach:
If you wish to have a look at new pipeline deploy updated pipeline on your cluster:
Let’s stop on each step and explain what we can mitigate while we are running it
tkn pipeline start build-and-deploy \
Python-Scan — scan Python for hardcoded passwords or dodgy libraries. For example, my source code had a typical Flask pattern
And Bandit didn’t allow me to proceed, because binding a port to all interfaces is not a good idea.
Kube-Linter — a great product from StackRox, it checks manifest files and reveals configuration issues. Unfortunately, it doesn’t support OpenShift APIs, and as I use “DeploymentConfig” which is not recognized by Kube-Linter, so I bypassing the error. But please consider this tool if you do use other Kubernetes distro.
Kube-Linter could catch orphan services or absence of CPU/Memory limits.
build-and-scan— This task has a few steps to build and scan containers on vulnerabilities. To make scanning in a true Cloud-Native way, instead of mounting the image to Trivy we use Docker side-car container with Trivy container to scan freshly built image.
The steps are:
- Build an image with buildah and push the image to Docker side-car container
- Scan the new image with Trivy
- Push image to the local OpenShift registry
And our base python image from Docker Hub has a lot of surprises:
Pretty interesting, right? This image has 19 critical vulnerabilities that will be working hand in hand with your code. Please be noted that the severity can be tuned and probably some warnings can be skipped, but it depends on your faith in the code or how convincing you might be in front of InfoSec
How is our Frontend container?
Not better actually, 32 critical vulnerabilities.
deploy-k8 — Is a service task that takes our manifest file and deploys them in our namespace. You do not need to create a new serviceaccount as it will be using one from Tekton.
The best way to avoid all issues above is to use certified and clean images, for example from Red Hat. Let’s change Dockerfile to use Python from Red Hat UBI
and trigger pipeline again to see how it change the situation.
As you can see we still have some warnings but there are no critical issues, so we can review them and probably relax severity.
If you do not want to be at google news feed with the name of your company and the words
[“Breach”, “Leak”, “Security”] in the same title or be blackmailed by hacker groups, you might want to consider DevSecOps practices in your organization. DevSecOps does not slow down or stop innovations but allows you to raise red flags to developers if something needs attention.
PS: There are many security scanners, I used Trivy, Bandit, and njsscan because we used them in our projects before. Feel free to make your own research.
 Tekton pipelne (master and simple branches) https://github.com/mancubus77/devsecops-demo-tekton
 App Frontend (NodeJS), https://github.com/mancubus77/devsecops-demo-fe
 App Backend (Python), https://github.com/mancubus77/devsecops-demo-be