Exporting Kubernetes Events to AWS Elastic Search

in DevOps , AIOps , Kubernetes , Logging


Kubernetes Events are not persisted by default. Using a log aggregator like AWS Elastic Search will allow us to persist these events and use it to do forensics later. Later on you can also use event-exporter to alert on these events using prometheus.

Kubernetes Event Exporter

Events are a type of logs in cluster that helps us debug or troubleshoot. Events are available when we run kubectl describe pods or kubectl get events

The problem is that by default they only last for 1 hour in order to conserve etcd. In EKS they are only available for 5 minutes by default.

In theory you can get all kubernetes events by running kubectl get events --watch and pumping the results of that into a sink like elasticsearch. However we want to be able to filter only the events we need.

To export these `events` we will be using OpsGenie's kubernetes-events-exportertool.

The results of this exercise should look like something below.

Screenshot 2021-01-11 at 17.07.18.png
Screenshot 2021-01-11 at 17.07.51.png

Optional Prerequisites

  • Running Kubernetes Cluster
  • Running Elastic Search Cluster

For the purposes of demo we will be using AWS Full managed Services but you can use any flavor of Kubernetes or Elastic Search:

  • Kubernetes: AWS EKS
  • Elastic Search: AWS Elastic Search

(Optional) Setting up AWS ElasticSearch

Please skip this part if you have a running Elastic Search Cluster already. If you follow this part please be aware that this is for testing purposes only in production ensure that your Elastic Search Cluster is production ready.

For testing purposes we will manually create an elastic search cluster with username password. First make sure that your AWS CLI is on the latest version

pip3 install awscli --ugprade

get the id of aws managed KMS key called aws/es

ES_KMS_ID=$(aws kms list-aliases | jq -r '.Aliases[] | select(.AliasName=="alias/aws/es").TargetKeyId')

export environment variables needed to setup AWS ElasticSearch


Create the AWS Elastic Search via AWS CLI

aws es create-elasticsearch-domain \
  --region eu-west-1 \
  --domain-name $ES_DOMAIN \
  --domain-endpoint-options EnforceHTTPS=true,TLSSecurityPolicy=Policy-Min-TLS-1-2-2019-07 \
  --elasticsearch-version 7.4 \
  --elasticsearch-cluster-config InstanceType=r4.large.elasticsearch,InstanceCount=1 \
  --ebs-options EBSEnabled=true,VolumeType=standard,VolumeSize=10 \
  --node-to-node-encryption-options Enabled=true \
  --encryption-at-rest Enabled=true,KmsKeyId=$ES_KMS_ID \
  --advanced-security-options "{\"Enabled\":true,\"InternalUserDatabaseEnabled\":true,\"MasterUserOptions\": {\"MasterUserName\":\"$USERNAME\",\"MasterUserPassword\":\"$PASSWORD\"}}" \
  --access-policies "{\"Version\": \"2012-10-17\", \"Statement\": [ { \"Effect\": \"Allow\", \"Principal\": {\"AWS\": \"*\" }, \"Action\":\"es:*\", \"Resource\": \"arn:aws:es:us-west-1:$AWS_ACCOUNT:domain/$ES_DOMAIN/*\" } ] }"

Testing AWS Elastic Search Setup

AUTH=$(echo -ne fakeuser:Fakepassword1! | base64)
echo $AUTH

curl -XGET "$ES_ENDPOINT" -H "Authorization: Basic $AUTH"

curl -XPUT "$ES_ENDPOINT/movies/_doc/1?pretty" -H "Authorization: Basic $AUTH" -H 'Content-Type: application/json' -d '{ "title": "John Carter" }'

curl -XGET "$ES_ENDPOINT/movies/_doc/1?pretty" -H "Authorization: Basic $AUTH"

For more ES Queries:

  • https://sysadmins.co.za/getting-started-with-aws-elasticsearch-service/

(Optional) Setting up Kubernetes using AWS EKS

brew tap weaveworks/tap
brew install weaveworks/tap/eksctl

git clone git@github.com:kenichi-shibata/cluster-test
cd eksctl/

eksctl create cluster -f dev-cluster-1.yaml
# wait for the cluster to be created

Setting up events-exporter

Make sure to change the elasticsearch.hosts[] in the configmap.yaml file to the actual Elastic Search Endpoint. Or if you have a different authentication make sure you specify that instead.

Create three yaml files in order

  • roles.yaml
  • configmap.yaml
  • deployment.yaml
apiVersion: v1
kind: Namespace
  name: monitoring
apiVersion: v1
kind: ServiceAccount
  namespace: monitoring
  name: event-exporter
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
  name: event-exporter
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
  - kind: ServiceAccount
    namespace: monitoring
    name: event-exporter
apiVersion: v1
kind: ConfigMap
  name: event-exporter-cfg
  namespace: monitoring
  config.yaml: |
      # Main route
        # This route allows dumping all events because it has no fields to match and no drop rules.
        - match:
          - receiver: elasticsearch-dump
      - name: "elasticsearch-dump"
          - https://search-replace-this-endpoint-.eu-west-1.es.amazonaws.com
          index: kube-events
          indexFormat: "kube-events-{2006-01-02}"
          username: fakeuser
          password: Fakepassword1!
apiVersion: apps/v1
kind: Deployment
  name: event-exporter
  namespace: monitoring
  replicas: 1
        app: event-exporter
        version: v1
      serviceAccountName: event-exporter
        - name: event-exporter
          image: opsgenie/kubernetes-event-exporter:0.7
          imagePullPolicy: IfNotPresent
            - -conf=/data/config.yaml
            - mountPath: /data
              name: cfg
        - name: cfg
            name: event-exporter-cfg
      app: event-exporter
      version: v1

Once the yaml files are created apply them

kubectl apply -f roles.yaml

kubectl apply -f configmap.yaml

kubectl apply -f deployments.yaml

Check logs if authentication worked

kubectl logs deployment/event-exporter -n monitoring

Create a failed event

kubectl run -it --image=fakeimage/fakeimage123:123 thisshouldfail -n test

If you want to make the kubernetes fields searchable in elasticsearch you might need to index them like kube-events-*

If you want search try to reindex them management -> index pattern -> new index pattern --> kube-events-* --> refresh icon

Also to better integrate with Kubernetes Objects make sure to use metadata.creationTimestamp the index dialog is screenshotted below

Screenshot 2021-01-11 at 17.06.07.png
Screenshot 2021-01-11 at 17.06.21.png
Screenshot 2021-01-11 at 17.06.38.png

If all went well you should see something like this entries in kube-events-*

Screenshot 2021-01-11 at 17.19.28.png
Screenshot 2021-01-11 at 17.19.44.png

Share this story with your friends