Production Observability on AWS EKS: Implementing a Cost-Efficient, High-Availability Metrics and Logging Stack (Prometheus, Grafana, Alertmanager, Loki, Promtail, S3, & IRSA)

Exposing dashboards securely, managing massive amounts of state, and receiving clean, actionable alerts requires careful planning. If you simply install default charts, you’ll quickly run into astronomical cloud bills from multiple AWS Application Load Balancers, complex credential leaks, and silent alert delivery failures.

In this masterclass guide, we will walk through how to deploy a production-grade, highly-available monitoring and logging setup on an Amazon EKS cluster. We will combine metrics, alerts, and log aggregation into a unified pipeline using:

1. kube-prometheus-stack for metrics collection, Grafana visualization, and Alertmanager routing.
2. Grafana Loki (Simple Scalable mode) for centralizing cluster logs using AWS S3 as backend storage.
3. Grafana Promtail to collect and ship container log streams.
4. AWS IAM Roles for Service Accounts (IRSA) for secure, credential-free S3 authentication.
5. AWS ALB Ingress Grouping to route Grafana, Prometheus, and Alertmanager behind a single Application Load Balancer (saving you over $50/month in idle load balancer fees).
6. Custom HTML Mail Alerts styled with actionable metadata routed securely through Gmail SMTP.

---

🏗️ Architecture Overview

The following diagram illustrates how all these components fit together inside your Amazon EKS cluster:

graph TD
    subgraph Clients & Users
        DevOps[On-Call Engineer]
    end

    subgraph AWS ALB [Single AWS ALB Instance]
        GrafanaDNS["grafana.yourdomain.com"]
        PromDNS["prometheus.yourdomain.com"]
        AlertDNS["alertmanager.yourdomain.com"]
    end

    subgraph EKS Cluster [monitoring namespace]
        Ingress[AWS Ingress Controller] -->|Routes| Grafana[Grafana Service]
        Ingress -->|Routes| PromSvc[Prometheus Service]
        Ingress -->|Routes| AMSvc[Alertmanager Service]

        KubeState[Kube-State-Metrics] -->|Scrape| Prom[Prometheus Server]
        NodeExp[Node Exporter DaemonSet] -->|Scrape| Prom

        Prom -->|Triggers Alerts| AM[Alertmanager Pods]

        Promtail[Promtail DaemonSet] -->|Tails Container Logs| Gateway[Loki Gateway Nginx]
        Gateway -->|Write Path| LokiWrite[Loki Ingester/Write]
        Gateway -->|Read Path| LokiRead[Loki Querier/Read]
    end

    subgraph External Services
        AM -->|Gmail SMTP| SMTP[smtp.gmail.com:587]
        SMTP -->|Email Alert| DevOps
        Grafana -->|Query Logs| Gateway
    end

    subgraph AWS Storage
        LokiWrite -->|Chunks & Indexes| S3[(AWS S3: loki-inspira-logs)]
        LokiRead -->|Reads Logs| S3
        SA[Service Account: eks-loki-access-s3-role-sa] -.->|IRSA Role Assume| S3
    end

    GrafanaDNS --> Ingress
    PromDNS --> Ingress
    AlertDNS --> Ingress

---

🛠️ Step 1: Base Configuration (values.yaml)

We start by configuring the core metric collection stack. In our values.yaml, we enable RBAC, configure resource quotas to prevent memory leak crashes, specify persistent EBS storage using the AWS gp3 storage class, and leverage AWS ALB Ingress Grouping to route all three dashboards through a single load balancer.

Save the following content as values.yaml in your repository:

# values.yaml
global:
  rbac:
    create: true

defaultRules:
  create: true

# Prometheus Operator Controller
prometheusOperator:
  enabled: true
  resources:
    limits:
      cpu: 200m
      memory: 256Mi
    requests:
      cpu: 100m
      memory: 128Mi

kubeStateMetrics:
  enabled: true

nodeExporter:
  enabled: true

# Grafana Configuration
grafana:
  enabled: true
  adminUser: admin
  assertNoLeakedSecrets: false # Required to pass SMTP password directly in secrets file

  # Auto-provision Loki as a datasource inside Grafana
  additionalDataSources:
    - name: Loki
      type: loki
      access: proxy
      url: http://loki-gateway.monitoring.svc.cluster.local:80
      jsonData:
        maxLines: 1000

  service:
    type: ClusterIP
    port: 80

  persistence:
    enabled: true
    storageClassName: gp3
    size: 20Gi
    accessModes:
      - ReadWriteOnce

  ingress:
    enabled: true
    ingressClassName: alb
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ca-central-1:410398365640:certificate/e6d48371-73f1-48c1-bbdd-fcbaa5bde49a
      alb.ingress.kubernetes.io/group.name: app-inspira-ca-lb
      alb.ingress.kubernetes.io/group.order: '-3'
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/ssl-redirect: '443'
      alb.ingress.kubernetes.io/target-type: ip
    hosts:
      - grafana-ca.inspiratest.com
    path: /
    pathType: Prefix

  resources:
    requests:
      cpu: 100m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi

# Prometheus Server Configuration
prometheus:
  enabled: true
  service:
    type: ClusterIP
    port: 9090

  ingress:
    enabled: true
    ingressClassName: alb
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ca-central-1:410398365640:certificate/e6d48371-73f1-48c1-bbdd-fcbaa5bde49a
      alb.ingress.kubernetes.io/group.name: app-inspira-ca-lb
      alb.ingress.kubernetes.io/group.order: '-3'
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/ssl-redirect: '443'
      alb.ingress.kubernetes.io/target-type: ip
    hosts:
      - prometheus-ca.inspiratest.com
    paths:
      - /
    pathType: Prefix

  prometheusSpec:
    retention: 30d
    scrapeInterval: 30s
    evaluationInterval: 30s
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: "2"
        memory: 4Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 100Gi

# Alertmanager Configuration
alertmanager:
  enabled: true
  service:
    type: ClusterIP
    port: 9093

  ingress:
    enabled: true
    ingressClassName: alb
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ca-central-1:410398365640:certificate/e6d48371-73f1-48c1-bbdd-fcbaa5bde49a
      alb.ingress.kubernetes.io/group.name: app-inspira-ca-lb
      alb.ingress.kubernetes.io/group.order: '-3'
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/ssl-redirect: '443'
      alb.ingress.kubernetes.io/target-type: ip
    hosts:
      - alertmanager-ca.inspiratest.com
    paths:
      - /
    pathType: Prefix

  alertmanagerSpec:
    replicas: 2
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: gp3
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi

  config:
    global:
      resolve_timeout: 1m
    route:
      group_wait: 10s
      group_interval: 2m
      repeat_interval: 2m
      receiver: 'gmail-notifications'
      routes: [] # CRITICAL: Clean default routing overrides
    receivers:
      - name: 'gmail-notifications'

---

🔒 Step 2: Configuring SMTP Credentials (values-secrets.yaml)

To maintain clean Git hygiene, keep your passwords and SMTP configs out of values.yaml. Instead, declare them in an isolated values-secrets.yaml file:

# values-secrets.yaml
grafana:
  # Grafana Admin Password
  adminPassword: "YourSecureAdminPassword"
  grafana.ini:
    smtp:
      enabled: true
      host: "smtp.gmail.com:587"
      user: "your-devops-email@gmail.com"
      password: "gmail-app-password" # 16-character Google App Password
      from_address: "your-devops-email@gmail.com"
      from_name: "Grafana EKS Alerts"
      skip_verify: false

alertmanager:
  config:
    global:
      smtp_smarthost: 'smtp.gmail.com:587'
      smtp_from: 'your-devops-email@gmail.com'
      smtp_auth_username: 'your-devops-email@gmail.com'
      smtp_auth_password: 'gmail-app-password'
      smtp_require_tls: true

    receivers:
      - name: 'gmail-notifications'
        email_configs:
          - to: 'recipient-team@yourdomain.com'
            send_resolved: true
            auth_username: 'your-devops-email@gmail.com'
            auth_password: 'gmail-app-password'
            from: 'your-devops-email@gmail.com'
            smarthost: 'smtp.gmail.com:587'
            headers:
              Subject: '🇨🇦 [{{ .Status | toUpper }}] EKS Alert: {{ if .CommonLabels.alertname }}{{ .CommonLabels.alertname }}{{ else }}{{ .Alerts | len }} alerts{{ end }}'
            html: |
              <!-- HTML template is embedded here (See Step 3) -->

---

🚨 Troubleshooting Alertmanager SMTP: Silent Pitfalls

Setting up SMTP alerts often leads to debugging cycles. Here are three critical issues and how to fix them:

Pitfall 1: Helm Array Overwrite Bug (undefined receiver “null”)

By default, the kube-prometheus-stack chart defines internal sub-routing rules that forward warnings/watchdogs to a receiver named null. When you customize your receivers, Helm does not append to the array—it completely overwrites it.

If you define a custom receivers array without defining a null receiver, Alertmanager fails to reconcile and throws:
failed to initialize from secret: undefined receiver "null" used in route

• The Fix: Explicitly declare routes: [] under route: in your values.yaml to clear the default sub-routing rules that reference the missing null receiver.

Pitfall 2: The auth_identity Handshake Rejection

When using Go-based PlainAuth (which Alertmanager utilizes under the hood) with Gmail, specifying your email in the auth_identity field causes the SMTP TLS handshake to fail.

• The Fix: Remove the auth_identity field entirely or set it as an empty string. Go’s auth handler automatically assumes the login username when auth_identity is blank.

Pitfall 3: Microsoft 365 Group Senders Block

If you route alerts to an Office 365 Group email (e.g., alerts@yourdomain.com) and Alertmanager logs success, but you receive no emails:

• The Fix: Open your M365 Outlook Group settings, edit details, and ensure the checkbox “Let people outside the organization email the groups” is checked. Otherwise, Exchange blocks incoming mail from your external Gmail address.

---

🎨 Step 3: Designing a Custom HTML Email Template

To convert unformatted alert text containing hundreds of raw labels into an actionable email, embed this clean, responsive HTML structure inside the html: parameter of values-secrets.yaml:

<html>
<head>
  <style>
    body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif; color: #333333; line-height: 1.6; margin: 0; padding: 20px; background-color: #f9f9f9; }
    .container { max-width: 600px; margin: 0 auto; background: #ffffff; padding: 25px; border-radius: 8px; border: 1px solid #e1e4e8; box-shadow: 0 4px 12px rgba(0,0,0,0.05); }
    .header { margin-bottom: 20px; border-bottom: 2px solid #eaecef; padding-bottom: 10px; }
    .header h2 { margin: 0; color: #24292e; font-size: 20px; }
    .alert-item { margin-bottom: 20px; padding: 15px; border-radius: 6px; border-left: 5px solid #d9534f; background-color: #fff9f9; }
    .alert-item.resolved { border-left-color: #28a745; background-color: #f6ffed; }
    .alert-item.warning { border-left-color: #ffc107; background-color: #fffdf5; }
    .alert-title { font-size: 16px; font-weight: bold; margin-bottom: 10px; color: #24292e; }
    .meta-table { width: 100%; border-collapse: collapse; margin-top: 10px; }
    .meta-table td { padding: 4px 0; vertical-align: top; font-size: 13px; }
    .meta-label { width: 100px; font-weight: bold; color: #586069; }
    .meta-val { color: #24292e; }
    .description-box { margin-top: 10px; padding: 10px; background: #f6f8fa; border-radius: 4px; font-size: 13px; border: 1px solid #e1e4e8; }
    .footer { margin-top: 30px; font-size: 12px; color: #6a737d; text-align: center; border-top: 1px solid #eaecef; padding-top: 15px; }
    .btn { display: inline-block; padding: 6px 12px; font-size: 13px; font-weight: bold; color: #ffffff !important; background-color: #0366d6; text-decoration: none; border-radius: 4px; }
    .btn-runbook { background-color: #28a745; margin-left: 10px; }
  </style>
</head>
<body>
  <div class="container">
    <div class="header">
      <h2>🇨🇦 Canada EKS Production Alerts</h2>
    </div>
    {{ range .Alerts }}
      <div class="alert-item {{ if eq .Status "resolved" }}resolved{{ else }}{{ .Labels.severity }}{{ end }}">
        <div class="title alert-title">
          [{{ .Status | toUpper }}] {{ .Labels.alertname }}
        </div>
        <table class="meta-table">
          <tr>
            <td class="meta-label">Severity:</td>
            <td class="meta-val"><span style="text-transform: capitalize; font-weight: bold; color: {{ if eq .Labels.severity "critical" }}#d9534f{{ else if eq .Labels.severity "warning" }}#ffc107{{ else }}#28a745{{ end }}">{{ .Labels.severity }}</span></td>
          </tr>
          {{ if .Labels.namespace }}
          <tr>
            <td class="meta-label">Namespace:</td>
            <td class="meta-val"><code>{{ .Labels.namespace }}</code></td>
          </tr>
          {{ end }}
          {{ if .Labels.pod }}
          <tr>
            <td class="meta-label">Pod:</td>
            <td class="meta-val"><code>{{ .Labels.pod }}</code></td>
          </tr>
          {{ end }}
          {{ if .Labels.container }}
          <tr>
            <td class="meta-label">Container:</td>
            <td class="meta-val"><code>{{ .Labels.container }}</code></td>
          </tr>
          {{ end }}
          {{ if .Annotations.summary }}
          <tr>
            <td class="meta-label">Summary:</td>
            <td class="meta-val">{{ .Annotations.summary }}</td>
          </tr>
          {{ end }}
        </table>
        {{ if .Annotations.description }}
        <div class="description-box">
          <strong>Description:</strong><br/>
          {{ .Annotations.description }}
        </div>
        {{ end }}
        <div style="margin-top: 15px;">
          <a class="btn" href="{{ .GeneratorURL }}" target="_blank">View in Prometheus</a>
          {{ if .Annotations.runbook_url }}
            <a class="btn btn-runbook" href="{{ .Annotations.runbook_url }}" target="_blank">Runbook</a>
          {{ end }}
        </div>
      </div>
    {{ end }}
    <div class="footer">
      <p>Alertmanager: <a href="{{ .ExternalURL }}">View Active Alerts Console</a></p>
      <p style="font-size: 10px;">Sent automatically by Canada EKS Monitoring System.</p>
    </div>
  </div>
</body>
</html>

---

🪵 Step 4: Implementing Centralized Logging (Loki & Promtail)

Rather than paying for expensive managed logging platforms or writing logs to transient container disks that disappear when pods restart, we deploy Grafana Loki in Simple Scalable Mode using AWS S3 as our durable log store.

1. Passwordless Storage Access via AWS IRSA

Using static IAM keys (accessKeyId and secretAccessKey) inside cluster configuration is a major security vulnerability. Instead, we configure EKS IAM Roles for Service Accounts (IRSA):

1. We create an AWS IAM policy allowing read/write actions on the S3 bucket loki-inspira-logs.
2. We associate that role with the Kubernetes Service Account eks-loki-access-s3-role-sa via our cluster’s OIDC provider.
3. Loki pods are assigned this service account, allowing them to assume the role and perform secure, token-based S3 requests.

Save the following file as loki-values.yaml:

# loki-values.yaml
serviceAccount:
  create: true
  name: eks-loki-access-s3-role-sa
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::410398365640:role/LokiServiceAccountRole

loki:
  auth_enabled: false
  commonConfig:
    path_prefix: /var/loki
    replication_factor: 3

  storage:
    type: s3
    bucketNames:
      chunks: loki-inspira-logs
      ruler: loki-inspira-logs
      admin: loki-inspira-logs
    s3:
      region: ca-central-1
      s3ForcePathStyle: false
      insecure: false
      # Left null to use EKS IRSA token authentication
      accessKeyId: null
      secretAccessKey: null

2. Scraping Node Logs using Promtail

To automatically parse container and system logs on every EKS node, we configure Promtail daemonset to forward logs directly to our Loki Gateway:

Save the following configuration as promtail-values.yaml:

# promtail-values.yaml
config:
  clients:
    - url: http://loki-gateway.monitoring.svc.cluster.local/loki/api/v1/push

---

🚀 Step 5: Single-Click Automation (deploy.ps1)

To prevent manual error when upgrading individual Helm charts, we orchestrate the complete deployment pipeline using a single PowerShell script. This handles Helm repositories, namespace checks, and upgrades the stack sequentially:

Save the following file as deploy.ps1:

# deploy.ps1
$namespace = "monitoring"
$releaseName = "kube-prometheus-stack"
$helmRepoName = "prometheus-community"
$helmRepoUrl = "https://prometheus-community.github.io/helm-charts"

Write-Host "=== Starting EKS Monitoring & Logging Deployment ===" -ForegroundColor Green

# 1. Verify Prerequisites
if (!(Get-Command helm -ErrorAction SilentlyContinue)) {
    Write-Error "Helm is not installed."
    exit 1
}
if (!(Get-Command kubectl -ErrorAction SilentlyContinue)) {
    Write-Error "kubectl is not installed."
    exit 1
}

# 2. Check cluster connection
$clusterInfo = kubectl cluster-info
if ($LASTEXITCODE -ne 0) {
    Write-Error "Unable to connect to the Kubernetes cluster."
    exit 1
}

# 3. Create Namespace
kubectl create namespace $namespace --dry-run=client -o yaml | kubectl apply -f -

# 4. Manage Helm Repositories
Write-Host "[3/4] Adding and updating Helm repositories..." -ForegroundColor Cyan
helm repo add $helmRepoName $helmRepoUrl
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# 5. Execute Deployment
Write-Host "[4/4] Deploying / upgrading EKS Monitoring stack with Helm..." -ForegroundColor Cyan

# A. Deploy kube-prometheus-stack
Write-Host "Deploying kube-prometheus-stack..." -ForegroundColor Yellow
helm upgrade --install $releaseName "$helmRepoName/kube-prometheus-stack" `
    --namespace $namespace `
    --values values.yaml `
    --values values-secrets.yaml
if ($LASTEXITCODE -ne 0) { Write-Error "Failed to deploy kube-prometheus-stack"; exit 1 }

# B. Deploy Loki
Write-Host "Deploying Loki (simple-scalable)..." -ForegroundColor Yellow
helm upgrade --install loki grafana/loki-simple-scalable `
    --namespace $namespace `
    --values loki-values.yaml
if ($LASTEXITCODE -ne 0) { Write-Error "Failed to deploy Loki"; exit 1 }

# C. Deploy Promtail
Write-Host "Deploying Promtail..." -ForegroundColor Yellow
helm upgrade --install promtail grafana/promtail `
    --namespace $namespace `
    --values promtail-values.yaml

if ($LASTEXITCODE -eq 0) {
    Write-Host "`n=== Deployment completed successfully! ===" -ForegroundColor Green
    Write-Host "To monitor deployment status, run:" -ForegroundColor Yellow
    Write-Host "  kubectl get pods -n $namespace -w" -ForegroundColor White
} else {
    Write-Error "`n=== Deployment failed! ==="
    exit 1
}

To run the deployment pipeline:

.\deploy.ps1

---

💡 Troubleshooting Gotchas

The False-Positive Health Check Error

When setting up Loki inside Grafana’s settings tab, clicking Save & test may trigger a red alert:
Unable to connect with Loki. Please check the server logs for more details.

If you inspect your Grafana container logs, you will see:
Loki health check failed error="error from loki: parse error at line 1, col 1: syntax error: unexpected IDENTIFIER"

• Why it happens: Grafana v10+ runs an internal health check using a Prometheus-style metric query (vector(1)+vector(1)). Loki versions < 2.8.x (like 2.6.1) use LogQL v2, which does not support the vector() keyword, causing the query validation to fail with a syntax error.
• The Fix: This is a false positive. Simply ignore this validation warning. Head over to the Explore tab in Grafana, select the Loki data source, and query actual LogQL:
  text {namespace="monitoring"}
  You will see that the live logs are streaming into your Grafana instance perfectly!

---

📈 Conclusion

By grouping our Ingress routes behind a single AWS Application Load Balancer, we drastically cut down AWS Load Balancer costs. Setting up structured secrets keeps production parameters safe, and configuring a beautiful HTML template ensures that your on-call engineers get alerts that are readable, color-coded, and highly actionable. Furthermore, by integrating Grafana Loki with S3 and EKS OIDC, we achieve durable, secure logging at close to zero additional cost.

Happy Monitoring!