Kube-Prometheus Setup Guide

Kube-prometheus is a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules that provides easy-to-operate end-to-end Kubernetes cluster monitoring. It bundles Prometheus Operator, Prometheus, Alertmanager, and Grafana with pre-configured dashboards and alerting rules.

Prerequisites

• A running Kubernetes cluster (version 1.16 or later recommended)
• kubectl configured to communicate with the cluster
• Helm (optional, for alternative installation methods)
• Sufficient resources (at least 2 CPUs and 4GB RAM available)

Installation Methods

Method 1: Using kube-prometheus-stack Helm Chart

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

Method 2: Manual Installation from Source

git clone https://github.com/prometheus-operator/kube-prometheus.git
cd kube-prometheus
kubectl create -f manifests/setup
until kubectl get servicemonitors --all-namespaces; do sleep 1; done
kubectl create -f manifests/

Configuration Options

Customize the installation by modifying values.yaml for Helm or editing the manifests directly:

prometheus:
  prometheusSpec:
    resources:
      requests:
        memory: 2Gi
        cpu: 1
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: "standard"
          resources:
            requests:
              storage: 50Gi

Verifying the Installation

Check pod status:

kubectl get pods -n monitoring

Access Prometheus UI:

kubectl port-forward -n monitoring svc/prometheus-operated 9090

Access Grafana (default credentials admin/prom-operator):

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

Post-Installation Tasks

• Configure persistent storage for Prometheus and Grafana
• Set up ingress or load balancer for external access
• Customize alert rules in PrometheusRule objects
• Add additional scrape configurations through ServiceMonitors

Monitoring Components

The setup includes monitoring for:

• Kubernetes API server
• Nodes
• Pods
• Kubelet
• etcd (if accessible)
• Kube-controller-manager
• Kube-scheduler

Alert Management

Pre-configured alerts include:

• Pod crash loop
• Node memory pressure
• Job failures
• API server errors
• Persistent volume claims filling up

Maintenance Considerations

• Regularly update the stack to get new features and security fixes
• Monitor Prometheus storage usage
• Review and tune alert thresholds
• Back up Grafana dashboards
• Consider federation for large clusters

Troubleshooting Common Issues

For pod crashes, check logs:

kubectl logs -n monitoring <pod-name>

For configuration issues, verify CRDs:

kubectl get prometheuses.monitoring.coreos.com -n monitoring
kubectl get servicemonitors.monitoring.coreos.com -n monitoring

For resource constraints, check events:

kubectl get events -n monitoring --sort-by='.metadata.creationTimestamp'