The Problems with GitOps — And How to Fix Them

Codefresh sponsored this post.

Viktor Farcic

Viktor Farcic is a Principal DevOps Architect at Codefresh, a member of the Google Developer Experts and Docker Captains groups, and a published author.

GitOps is a way to manage the actual state of systems, through definitions of the desired state stored in Git. You probably already know what GitOps is and all its benefits. If you don’t, please watch What Is GitOps And Why Do We Want It?. For this post, we’ll ignore the good and focus on the bad and the ugly. In a follow-up post, I will argue that GitOps as an idea is great, but that we are not even close to having that idea be useful in a practical sense. I will even go as far as to say that there is a lot of work to be done and that we need to hit the reset button. For now, let me focus on the issues we are facing with GitOps. Let me give you a summary of the issues:

• GitOps is misunderstood.
• GitOps is NOT only about Kubernetes.
• GitOps tools are NOT promoting GitOps principles.
• We are often not even able to apply GitOps principles in GitOps tools.
• We do NOT have the tools that reflect changes happening inside clusters in Git.
• Observability is immature.
• There are no well-established patterns.
• The connection between Continuous Delivery and GitOps is not yet well established.
• Running GitOps at scale is challenging.
• Managing secrets is a big issue.

Let’s now go down the rabbit hole and see the pain we are facing when adopting GitOps.

GitOps Is Misunderstood

There is a misunderstanding of what GitOps is. Like many other popular waves, software companies are riding this one and putting the “GitOps” sticker on their products. Just as every tool is a DevOps tool today, no matter what that tool does and even what DevOps is, many are being (re)branded as GitOps tools today. Companies are trying to sell us the idea that whatever their tool is doing is what GitOps is. As a result, we are debating what it is and what it isn’t, while in reality it is a simple concept based on a few easy to explain principles. Everything is defined as code; code is stored in git; git contains the desired state; machines are converging the actual into the desired state. That’s it. That’s all there is to it. It’s hard to get there, but it’s straightforward to explain what it is; and if that explanation is accepted, it’s even easier to distinguish those tools that are helping us achieve GitOps and those that aren’t.

GitOps Is NOT Only About Kubernetes

I’m more disappointed with some of the originators of the ideas behind GitOps. For example, the definition that says that “GitOps is a way to do Kubernetes cluster management and application delivery” is misleading. GitOps is not only a way to do “Kubernetes cluster management.” The fact that Kubernetes adoption is enormous does not mean that it is the only thing everyone cares about. GitOps is about something else, and that something is much broader than Kubernetes. Even those who fully embraced Kubernetes do not claim that everything is Kubernetes. There are nodes to be managed. There is networking. There are third-party services. Serverless is becoming more widely used. And so on and so forth. The fact that someone has a solution focused on Kubernetes does not mean that GitOps is only about the state inside a Kubernetes cluster. It is the way of working. It is the approach to defining what should be the state of something, whatever that something is. Even if you do rely a hundred percent on Kubernetes, it will not appear out of thin air and it will not be managed by itself. GitOps should help with that and not be something limited to a very narrow scope. The previous two issues could be classified as bad, and now it is getting ugly.

GitOps Tools Are NOT Promoting GitOps Principles

GitOps tools, even those that represent the shining beacons in the industry, are not truly promoting GitOps. Let me repeat that. GitOps tools are not promoting GitOps principles. Even when they do, many somehow think that it applies to others and that they are the exception. To demonstrate that, we can take a look at the “default” instructions for installing two of the most popular GitOps tools: Flux and Argo CD. Before we continue, let me stress that I think that both tools are excellent. This is not an attempt to “thrash” them, but rather to show that we are still in the very early stages of GitOps adoption and that it is not yet engraved in our minds. First, Flux did not initially try to provide any declarative definition for the installation, at least not from the initial installation page. There was no attempt to guide us towards having something defined as code and stored in git before being applied to the cluster. All that changed recently when Flux v2 was introduced. The tool changed the installation process and now it creates a git repo, pushes Flux manifests, and only then installs Flux. From there on, any change to Flux can be done by making changes to the associated repository. Well done Flux! With Argo CD, we can take the install.yaml manifest, store it in git, and then apply it to the actual state. We could easily hook it into any CD tool and create a webhook that notifies it whenever that definition changes. On the first look, it seems that Argo CD is, at least, giving a semblance of following the principles of GitOps by giving us a YAML file that we can store in a git repository before applying it. But, that seems to be more likely a lucky accident that it chose to be defined in a declarative YAML format that can be stored in git. The truth is that, as soon as we scroll down that same page, we find examples like the following. If we scroll further down, we can find things like the following command to add a cluster to its registry. I’ll skip, for now, the discussion of whether using those tools is truly GitOps. In this context, Argo CD is not adhering to “the entire system described declaratively” principles from the get-go. It is not trying to “push” the users towards the idea that the desired state should be stored in git. The first contact with Argo CD and, until recently, with Flux, tools tells us, “we are the exception; those rules apply to others, but not to us.” Truth be told, the same can be said for most of the tools on the market, including those marketing themselves as GitOps. I picked the above two mostly because they are excellent and popular, not because they are worse than others. In this article we have explored the first three GitOps issues in my shortlist. I’ll discuss the rest in two follow-up articles.

Feature image via Pixabay.