Deploy Prometheus and Grafana on Kubernetes

Edka installs the kube-prometheus-stack chart from the prometheus-community repo into the monitoring namespace on your own cluster. That single chart brings Prometheus for metrics, Grafana for dashboards, Alertmanager for routing, and the Prometheus Operator to manage all three as Kubernetes resources.

Assembling this by hand means reconciling four sets of manifests, wiring service discovery, and writing the recording rules yourself. The chart is the reason nobody does that anymore.

What’s Included

Prometheus Server with Kubernetes service discovery, 30s scrape interval, and 15-day retention
Grafana with the chart’s default dashboards and the Prometheus datasource pre-wired
Alertmanager for alert routing and silencing, with 120h retention
Prometheus Operator to manage Prometheus, Alertmanager, and ServiceMonitor resources
kube-state-metrics for Kubernetes object metrics
node-exporter, the Edka add-on, scraped for host and OS metrics

Sizing Prometheus Storage

Storage is the setting people revisit first, so it is worth understanding before you deploy rather than after a disk fills.

Prometheus disk use follows retention, ingestion rate, and sample size. The upstream formula is retention_time × samples_per_second × bytes_per_sample, and compressed samples land in the range of one to two bytes each. What moves the number in practice is series count: every distinct label combination is its own series, and a chatty application with high cardinality labels can outweigh the rest of the cluster.

Edka’s defaults:

Setting	Default
Prometheus storage	20Gi, selectable up to 500Gi
Grafana storage	10Gi
Retention	15 days
Scrape interval	30s

20Gi at 15 days is comfortable for a small cluster running the default scrape targets. If you add applications exposing many series, watch the volume before extending retention. Both volumes are backed by hcloud-volumes.

Required Add-ons

This app depends on four Edka add-ons:

metrics-server for resource metrics
node-exporter for host and OS metrics
cert-manager for TLS certificates
letsencrypt-issuer for the Let’s Encrypt cluster issuer

Configuration Options

Monitoring

Kubernetes component monitoring: API server, kubelet, CoreDNS
Node metrics from the node-exporter add-on, with node dashboards and alerting rules
Workload metrics for pods, deployments, and applications via kube-state-metrics
Automatic service discovery for services and pods carrying Prometheus annotations

Alerting

Default alerting and recording rules from the chart, covering Kubernetes apps, resources, storage, and the system
Routing by severity, with critical and warning routes and inhibition rules
Slack notifications via webhook, sent to a configurable channel

Grafana

Authentication required by default, with a generated admin password
Optional read-only anonymous access
Persistent storage for dashboards and configuration across restarts

Default Rules

The chart ships default alerting and recording rules. Active rule groups include:

Kubernetes apps, resources, storage, and system
API server availability, burn rate, histogram, and SLOs
kubelet and kube-state-metrics
Node and node-exporter rules, when node metrics are enabled
Prometheus and Prometheus Operator
Alertmanager and general rules

etcd, kube-scheduler, kube-proxy, and controller-manager rules are disabled by default.

Service Endpoints

After deployment, Edka surfaces these endpoints:

Grafana Dashboard: https://<grafana-hostname> when exposed externally
Prometheus Web UI: https://<prometheus-hostname> when exposed externally
Alertmanager UI: https://<alertmanager-hostname> when exposed externally
Internal cluster services for Grafana (:80), Prometheus (:9090), and Alertmanager (:9093) in the monitoring namespace

External exposure runs through your selected ingress class, with TLS issued by the cert-manager cluster issuer.

Deploy Prometheus Stack