Step by step instructions: How to deploy Prometheus Operator inside Kubernetes Cluster (EKS)
Prerequirements
Install Helm
brew install helmbrew install helmfile brew install kubectl
Add Plugin Helm
helm plugin install https://github.com/databus23/helm-diffhelm plugin install https://github.com/hypnoglow/helm-s3.git
Add Helm Repository S3 Bucket
### LAB ###
helm s3 init s3://devopscorner-helm-chart/lab
AWS_REGION=ap-southeast-1 helm repo add devopscorner-lab s3://devopscorner-helm-chart/lab
### STAGING ###
helm s3 init s3://devopscorner-helm-chart/staging
AWS_REGION=ap-southeast-1 helm repo add devopscorner-staging s3://devopscorner-helm-chart/staging
### PRODUCTION ###
helm s3 init s3://devopscorner-helm-chart/prod
AWS_REGION=ap-southeast-1 helm repo add devopscorner s3://devopscorner-helm-chart/prod
helm repo update
Update Repository
helm repo add stable https://charts.helm.sh/stable
helm repo add grafana https://grafana.github.io/helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm repo list
NAME URL
prometheus-community https://prometheus-community.github.io/helm-charts
grafana https://grafana.github.io/helm-charts
stable https://charts.helm.sh/stable
Create Namespace
kubectl create namespace observability
Install Prometheus Operator
helm install prometheus-operator prometheus-community/kube-prometheus-stack --create-namespace -n observability
NAME: prometheus-operatorLAST DEPLOYED: Sun Nov 5 02:03:06 2023NAMESPACE: observabilitySTATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace observability get pods -l "release=prometheus-operator"
kubectl get po -n observability
NAME READY STATUS RESTARTS AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0 2/2 Running 0 32m
prometheus-grafana-55fb596bf5-5257r 3/3 Running 0 32m
prometheus-kube-prometheus-operator-757f8788d4-v6tk5 1/1 Running 0 32m
prometheus-kube-state-metrics-898dd9b88-98qlj 1/1 Running 0 32m
prometheus-prometheus-kube-prometheus-prometheus-0 2/2 Running 0 32m
prometheus-prometheus-node-exporter-llfn2 1/1 Running 0 32m
prometheus-prometheus-node-exporter-nrpkq 1/1 Running 0 32m
kubectl --namespace observability get pods -l "release=prometheus-operator"
NAME READY STATUS RESTARTS AGEprometheus-operator-kube-p-operator-7cc49d6ffb-ktjnv 1/1 Running 0 2m16sprometheus-operator-kube-state-metrics-797d9866bd-s4xhb 1/1 Running 0 2m16sprometheus-operator-prometheus-node-exporter-vg42d 1/1 Running 0 2m16s
kc get po -n observability
kubectl get svc -n observability
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 11m
prometheus-operated ClusterIP None <none> 9090/TCP 11m
prometheus-operator-grafana ClusterIP 172.20.33.136 <none> 80/TCP 11m
prometheus-operator-kube-p-alertmanager ClusterIP 172.20.136.150 <none> 9093/TCP,8080/TCP 11m
prometheus-operator-kube-p-operator ClusterIP 172.20.219.78 <none> 443/TCP 11m
prometheus-operator-kube-p-prometheus ClusterIP 172.20.195.49 <none> 9090/TCP,8080/TCP 11m
prometheus-operator-kube-state-metrics ClusterIP 172.20.214.227 <none> 8080/TCP 11m
prometheus-operator-prometheus-node-exporter ClusterIP 172.20.230.46 <none> 9100/TCP 11m
kc get svc -n observability
Edit Prometheus Service
Change to type LoadBalancer from ClusterIP
kubectl edit svc prometheus-kube-prometheus-prometheus -n observability
Edit Grafana Service
Change to type LoadBalancer from ClusterIP
kubectl edit svc prometheus-grafana -n observability
Access Load Balancer
Grafana Access
http://a7754204d8c2e41969dfa8134d4a3d78-2128039805.ap-southeast-1.elb.amazonaws.com
UserName: admin
Password: prom-operator
Change Credentials Grafana
kubectl get secret -n observability
NAME TYPE DATA AGE
alertmanager-prometheus-operator-kube-p-alertmanager Opaque 1 40h
alertmanager-prometheus-operator-kube-p-alertmanager-generated Opaque 1 40h
alertmanager-prometheus-operator-kube-p-alertmanager-tls-assets-0 Opaque 0 40h
alertmanager-prometheus-operator-kube-p-alertmanager-web-config Opaque 1 40h
prometheus-kube-prometheus-admission Opaque 3 4d12h
prometheus-operator-grafana Opaque 3 40h
prometheus-operator-kube-p-admission Opaque 3 40h
prometheus-operator-kube-p-prometheus Opaque 0 40h
prometheus-prometheus-operator-kube-p-prometheus Opaque 1 40h
prometheus-prometheus-operator-kube-p-prometheus-tls-assets-0 Opaque 1 40h
prometheus-prometheus-operator-kube-p-prometheus-web-config Opaque 1 40h
Change Base64 Credentials
- Change User Encode (admin-user)
echo “devopscorner-admin” | base64
4oCcZGV2b3BzY29ybmVyLWFkbWlu4oCdCg==
- Change Password Encode (admin-password)
echo “devopscorner-secret” | base64
4oCcZGV2b3BzY29ybmVyLXNlY3JldOKAnQo=
- Edit admin-password and admin-user from encoded base64 string
kubectl edit secret prometheus-operator-grafana -n observability
Prometheus Access
http://af58dac1154de4b57be3d0d63d60936b-925795512.ap-southeast-1.elb.amazonaws.com:9090
Using Jumpods
References:
https://github.com/devopscorner/devopscorner-helm/tree/master/helmfile/jumppod
Test with Curl (Inside Jumpods)
curl prometheus-kube-prometheus-prometheus.observability.svc.cluster.local:9090
curl prometheus-grafana.observability.svc.cluster.local
Using Network LoadBalancer (NLB) via NGINX Controller
Get Deployment Manifest
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/aws/deploy.yaml
Change Manifest
sed -i 's/externalTrafficPolicy: Local/externalTrafficPolicy: Cluster/g' deploy.yaml
Deploy Manifest
kubectl create -f deploy.yaml
Expose LB for Grafana Dashboard
kubectl expose deployment prometheus-operator-grafana --name=grafana-lb --type=LoadBalancer --port=80 --target-port=3000 --protocol=TCP --namespace=observability
Helm Release Version
helm list --all-namespaces
helm list --namespace=observability
devopscorner.id 