The Monitoring Stack Is the One Thing ArgoCD Does Not Manage¶

Ten microservices on this cluster are deployed by ArgoCD. Prometheus, Grafana, AlertManager, Elasticsearch, Filebeat and Kibana are not. They are installed by Helm, by hand, from a bastion host.

That looks like the part I did not get around to automating. It is deliberate, and the reason is a question worth asking of any GitOps setup: what do you debug ArgoCD with?

The Circular Dependency¶

If ArgoCD manages the monitoring stack, then the monitoring stack depends on ArgoCD being healthy.

Most of the time that is fine, because most of the time ArgoCD is healthy. The case that matters is the one where it is not: a bad sync, a crash loop, an application stuck Progressing, a repo server that cannot reach Git.

At exactly that moment you want Grafana and you want the logs. If both are ArgoCD-managed, you are debugging a broken deployment tool using tools that the broken deployment tool deploys. They will usually still be running, since a crashed ArgoCD does not delete what it already applied, but "usually" is doing real work in that sentence and it is not a property I want to rely on during an incident.

Installing them outside removes the question entirely. The monitoring stack has no dependency on ArgoCD in any direction.

The Ownership Argument, Which Matters More¶

The dependency reasoning is the one people find persuasive. The ownership reasoning is the one that actually reflects how this is done at scale.

Real organisations split these:

• Platform or SRE owns monitoring, logging, ingress controllers, service mesh, storage classes. Managed with Helm, Terraform, or a separate ArgoCD project with its own access model.
• Application teams own their workloads, managed through ArgoCD applications they control.

Those two have different change cadences, different reviewers, and different blast radii. A Prometheus upgrade and a frontend image bump are not the same category of change and should not flow through the same pipeline with the same approvals.

This project mirrors that split. The boutique application is ArgoCD-managed. The platform stack is Helm-managed. The line is drawn where an organisation would draw it, rather than where convenience would.

The Alternative, Stated Fairly¶

The other pattern is app-of-apps: one ArgoCD Application that manages other Applications, including the platform components and often ArgoCD itself.

It is a legitimate design with real benefits. Everything is declarative, everything is in Git, drift is detected on the platform layer too, and there is one mechanism to learn rather than two.

Its costs are the mirror image of the benefits above: the circular dependency is real, bootstrapping is more delicate (something has to install the thing that installs everything), and the platform and application layers share a failure domain.

I chose the split because the project's point was to show separation of concerns clearly. On a larger team already running app-of-apps successfully, I would not argue for changing it.

What This Costs Me¶

Being honest about the price, because it is not zero.

No drift detection on the platform layer. If someone edits the Grafana deployment by hand, nothing notices. ArgoCD would have flagged it.

The install is a set of commands, not a manifest. They are recorded in a runbook, which is not the same as being enforced. Rebuilding this cluster means running them in order rather than pointing ArgoCD at a repository.

Two mechanisms. Anyone joining has to learn that some things come from Git and some come from Helm on a bastion, and that the boundary is meaningful rather than accidental.

The third is the one I would fix first, by moving the platform layer into a second ArgoCD instance or a separate project with its own repo. That keeps the split and removes the "two mechanisms" objection, at the cost of running two ArgoCDs.

Where the Line Actually Falls¶

"Platform" and "application" sound clean and get blurry fast, so it is worth writing down what ended up on each side and why.

Helm-managed, outside ArgoCD:

ArgoCD-managed:

The test that separates them: does this change when application code changes? Ten services do, on every push. The storage class does not, and if it ever does, that is a platform change with a very different review path.

There is one row in the platform table that could sit on either side. The Gateway serves the application, so an argument exists for the application owning it. It also serves ArgoCD, Grafana, Prometheus and Kibana, which settles it: a resource four other things depend on cannot belong to one of them.

The Bootstrap Problem That Does Not Go Away¶

Whichever model you pick, something has to be installed by hand first. ArgoCD cannot deploy ArgoCD onto an empty cluster.

App-of-apps hides this rather than removing it: you install ArgoCD manually, apply one root Application, and everything else follows. That is a genuinely nice property and it is why the pattern is popular.

What it costs is that the bootstrap step is now load-bearing and rarely exercised. It runs once per cluster, which for most teams is once a year, on the day everything is already going wrong.

The split model has more manual steps and each one is smaller, ordinary, and documented as a runbook you follow rather than a mechanism you trust. For a cluster I rebuild often, in a lab where rebuilding is the point, that trade favours the split. For a long-lived production cluster rebuilt almost never, the argument reverses and app-of-apps looks better.

Where the Secret Lives¶

One implementation detail from the monitoring stack is worth extracting, because it generalises.

AlertManager routes critical alerts to Slack and everything else to an email default:

route:
  receiver: 'email-default'
  routes:
    - receiver: 'slack-notification'
      matchers:
        - severity = "critical"

The Slack webhook URL is a credential. It does not go in the values file:

kubectl create secret generic alertmanager-slack-webhook \
  --from-literal=slack-webhook-url="<REDACTED>" \
  -n monitoring

And AlertManager reads it from a mounted path rather than from an environment variable or an inline value:

alertmanager:
  alertmanagerSpec:
    secrets:
      - alertmanager-slack-webhook
  config:
    receivers:
      - name: 'slack-notification'
        slack_configs:
          - api_url_file: /etc/alertmanager/secrets/alertmanager-slack-webhook/slack-webhook-url

api_url_file rather than api_url is the whole point. The values file, which is in Git, contains a path. The secret is created separately and never appears in a repository, a Helm release, or helm get values output.

That distinction matters more in the Helm-managed model than it would under ArgoCD, because there is no sealed-secrets or external-secrets operator in the path here. The discipline has to come from the chart's own affordance, and any chart worth using offers one.

The Routing Detail Worth Copying¶

The default receiver being email rather than Slack is deliberate.

A route that only matches severity = critical and has no default means every non-critical alert is silently discarded. That is a common misconfiguration and it looks like a quiet, well-tuned system right up until something important arrives at a severity you did not anticipate.

Setting the default to something low-traffic and matching the noisy channel narrowly means unclassified alerts still land somewhere. Anything that arrives at the default is either a real alert you had not categorised or a signal that the matcher needs adjusting, and both are worth knowing.

The Test¶

If you are deciding this for your own cluster, one question resolves most of it:

When ArgoCD is broken, what are you looking at to find out why?

If the answer is a dashboard that ArgoCD deploys, that is worth a second thought. Not because it will definitely fail, but because the failure mode is correlated with the exact moment you need it, and correlated failures are the ones that turn an incident into a longer incident.

Source¶

• Observability runbook, both stacks with full install commands
• The AlertManager values patch
• What ArgoCD does manage, and how the handoff works