Exporting Kubernetes Events to AWS Elastic Search


Kubernetes Events are not persisted by default. Using a log aggregator like AWS Elastic Search will allow us to persist these events and use it to do forensics later. Later on you can also use event-exporter to alert on these events using prometheus.

    Kubernetes Event Exporter

    Events are a type of logs in cluster that helps us debug or troubleshoot. Events are available when we run kubectl describe pods or kubectl get events

    The problem is that by default they only last for 1 hour in order to conserve etcd. In EKS they are only available for 5 minutes by default.

    In theory you can get all kubernetes events by running kubectl get events --watch and pumping the results of that into a sink like elasticsearch. However we want to be able to filter only the events we need.

    To export these `events` we will be using OpsGenie's kubernetes-events-exportertool.

    Optional Prerequisites

    • Running Kubernetes Cluster
    • Running Elastic Search Cluster

    For the purposes of demo we will be using AWS Full managed Services but you can use any flavor of Kubernetes or Elastic Search:

    • Kubernetes: AWS EKS
    • Elastic Search: AWS Elastic Search

    (Optional) Setting up AWS ElasticSearch

    Please skip this part if you have a running Elastic Search Cluster already. If you follow this part please be aware that this is for testing purposes only in production ensure that your Elastic Search Cluster is production ready.

    For testing purposes we will manually create an elastic search cluster with username password. First make sure that your AWS CLI is on the latest version

    pip3 instlal awscli --ugprade

    get the id of aws managed KMS key called aws/es

    ES_KMS_ID=$(aws kms list-aliases | jq -r '.Aliases[] | select(.AliasName=="alias/aws/es").TargetKeyId')

    export environment variables needed to setup AWS ElasticSearch


    Create the AWS Elastic Search via AWS CLI

    aws es create-elasticsearch-domain \
      --region eu-west-1 \
      --domain-name $ES_DOMAIN \
      --domain-endpoint-options EnforceHTTPS=true,TLSSecurityPolicy=Policy-Min-TLS-1-2-2019-07 \
      --elasticsearch-version 7.4 \
      --elasticsearch-cluster-config InstanceType=r4.large.elasticsearch,InstanceCount=1 \
      --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=10 \
      --node-to-node-encryption-options Enabled=true \
      --encryption-at-rest Enabled=true,KmsKeyId=$ES_KMS_ID \
      --advanced-security-options "{\"Enabled\":true,\"InternalUserDatabaseEnabled\":true,\"MasterUserOptions\": {\"MasterUserName\":\"$USERNAME\",\"MasterUserPassword\":\"$PASSWORD\"}}" \
      --access-policies "{\"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Principal\": {\"AWS\": \"*\" }, \"Action\":\"es:*\", \"Resource\": \"arn:aws:es:us-west-1:$AWS_ACCOUNT:domain/$ES_DOMAIN/*\" } ] }"

    Testing AWS Elastic Search Setup

    AUTH=$(echo -ne fakeuser:Fakepassword1! | base64)
    echo $AUTH
    curl -XGET "$ES_ENDPOINT" -H "Authorization: Basic $AUTH"
    curl -XPUT "$ES_ENDPOINT/movies/_doc/1?pretty" -H "Authorization: Basic $AUTH" -H 'Content-Type: application/json' -d '{ "title": "John Carter" }'
    curl -XGET "$ES_ENDPOINT/movies/_doc/1?pretty" -H "Authorization: Basic $AUTH"

    For more ES Queries:

    • https://sysadmins.co.za/getting-started-with-aws-elasticsearch-service/

    (Optional) Setting up Kubernetes using AWS EKS

    brew tap weaveworks/tap
    brew install weaveworks/tap/eksctl
    git clone [email protected]:kenichi-shibata/cluster-test
    cd eksctl/
    eksctl create cluster -f dev-cluster-1.yaml
    # wait for the cluster to be created

    Setting up events-exporter

    Make sure to change the elasticsearch.hosts[] in the configmap.yaml file to the actual Elastic Search Endpoint. Or if you have a different authentication make sure you specify that instead.

    Create three yaml files in order

    • roles.yaml
    • configmap.yaml
    • deployment.yaml
    apiVersion: v1
    kind: Namespace
      name: monitoring
    apiVersion: v1
    kind: ServiceAccount
      namespace: monitoring
      name: event-exporter
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
      name: event-exporter
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: view
      - kind: ServiceAccount
        namespace: monitoring
        name: event-exporter
    apiVersion: v1
    kind: ConfigMap
      name: event-exporter-cfg
      namespace: monitoring
      config.yaml: |
          # Main route
            # This route allows dumping all events because it has no fields to match and no drop rules.
            - match:
              - receiver: elasticsearch-dump
          - name: "elasticsearch-dump"
              - https://search-replace-this-endpoint-.eu-west-1.es.amazonaws.com
              index: kube-events
              indexFormat: "kube-events-{2006-01-02}"
              username: fakeuser
              password: Fakepassword1!
    apiVersion: apps/v1
    kind: Deployment
      name: event-exporter
      namespace: monitoring
      replicas: 1
            app: event-exporter
            version: v1
          serviceAccountName: event-exporter
            - name: event-exporter
              image: opsgenie/kubernetes-event-exporter:0.7
              imagePullPolicy: IfNotPresent
                - -conf=/data/config.yaml
                - mountPath: /data
                  name: cfg
            - name: cfg
                name: event-exporter-cfg
          app: event-exporter
          version: v1

    Check logs if authenticaiton worked

    kubectl logs deployment/event-exporter -n monitoring

    Create a failed event

    kubectl run -it --image=fakeimage/fakeimage123:123 thisshouldfail -n test

    If you want to make the kubernetes fields searchable in elasticsearch you might need to index them like kube-events-*

    If you want search try to reindex them management -> index pattern -> new index pattern --> kube-events-* --> refresh icon

    If all went well you should see something like this entries


    Elastic Search Indexing

    Get similar stories in your inbox weekly, for free

    Share this story with your friends

    Latest stories

    DevOps and Downed Systems: How to Prepare

    Downed systems can cost thousands of dollars in immediate losses and more in reputation damage …

    Cloud: AWS Improves the Trigger Functions for Amazon SQS

    The improved AWS feature allows users to trigger Lambda functions from an SQS queue.

    Google Takes Security up a Notch for CI/CD With ClusterFuzzLite

    Google makes fuzzing easier and faster with ClusterFuzzLite

    HashiCorp Announces Vault 1.9

    Vault 1.9 released into general availability with new features

    Azure Container Apps: This Is What You Need to Know

    HTTP-based autoscaling and scale to zero capability on a serverless platform