GitOps Best Practices

A previous post in this series introduced GitOps and its benefits. This post covers the main points on how to adopt GitOps in your organization.

First of all, it’s not as arduous as the adoption of other systems and practices, primarily because engineers are already using Git in their day-to-day activities. This greatly reduces the learning curve and the level of change or potential disruption engineers will encounter.

When introducing anything new, always start small and grow rather than attempt a big-bang approach. Once everyone is comfortable with the new approach, slowly expand across different environments.

Core GitOps

Git should be the single source of truth for all infrastructure as code, for Kubernetes, this is all the YAML files that define deployments, services, configuration maps, etc. This guarantees consistency across all releases in every environment; there can be no exception. If Git is not the single source of truth, GitOps is not going to work. For some organizations, an amount of initial manual reconciliation may be required to ensure the definitions in Git are 100% accurate.

All changes should be made via Git, no sneaky ad-hoc kubectl tweaks. All user access keys should be revoked, they are no longer required. The GitOps agent will pull the configuration from Git; the single source of truth.

Separate repositories for source code and configuration. Although it is initially more work to set up separate repositories, it makes operations easier in the long run. Trying to manage deployments across multiple environments and multiple different build stages for application components all in one repository soon becomes a big headache. One configuration repository for each logical set of application components is recommended.

There should be a consistent Git branch strategy across all configuration repositories for all environments. This will make it much easier for engineers to work across different parts of the application. For example have separate branches for each environment e.g. dev, stage and prod. A typical workflow would then be:

• Create a new feature branch git checkout -b new-feature dev
• Work on the new branch then create a pull request to merge back into dev.
• Review and approve code changes.
• After merging changes into dev they are automatically reconciled by GitOps into the dev cluster.
• The change can be promoted up through the environments by repeating the pull request, review/approve, and merge steps through stage and prod.

Tighter Integration

GitOps provides advanced automation for the management of application deployments to Kubernetes. Where possible manual steps should be eliminated, reducing the burden of engineers and limiting the possibility of errors.

Tight integration with CI/CD pipelines provides complete end to end automation; the best way to achieve this is via the image registry. The CI/CD system is responsible for taking the source code, performing code quality checks, building the docker container, performing tests and finally pushing the container to a registry. GitOps can be configured to automatically detect a new image, update the deployment manifest, push the change to Git on a new branch and create a pull request. On approval and merge of the pull request, automatic reconciliation takes place and the new image is deployed.

When using this approach to tight integration between CI/CD and GitOps, it is recommended to use a strict SEMVER policy for staging and production. For the dev environment, a more lax latest policy is generally acceptable. For example:

• Dev - some/image:<git commit>
  Automatic merge and deploy
• Stage - some/image:1.2.3-rc1
  New branch and pull request
• Prod - some/image:1.2.3
  New branch and pull request

People

A new way of working is more than just some new tooling, it must include the people who will be adopting the new process. Initial training followed by regular reviews is essential for success. There will be mistakes and setbacks along the way, these will be overcome by involving the entire team in working through these problems.

Implementation

All the parts of a GitOps implementation are available via various open-source projects. It is possible to integrate the separate tools and build an enterprise GitOps platform. However, there is considerable effort required and your organization may not have the skills required. Additionally, it would pull resources away from other projects and may lengthen the feature backlog. Devtron provides a fully integrated GitOps platform with many additional features including access controls, a Kubernetes dashboard, and policy guardrails. Accelerate your software delivery pipeline today with Devtron.