EMK - Kubernetes Audit Events Logging

Estimated time to read: 5 minutes

Kubernetes audit events help you figure out what happened during security incidents, prove you're meeting compliance requirements, and debug issues. But to actually store them long-term, feed them into other tools, or set up alerts, you need a solid way to get those logs out of the cluster.

The auditing extension sets up a log forwarder inside your cluster's control plane. This component picks up audit logs from the API server, adds specific context, and forwards the events along to whatever backend systems you've configured.

In this tutorial we will create a demo setup to accept and process forwarded audit events over a mTLS secured connection.

Prerequisites:

• EMK cluster
• EMK Service Account, see service account access
• VM or instance with a public IPv4 and exposed port 9999/tcp

Audit Processing

To process audit events we will use Telegraf, a lightweight, open source tool that gathers, processes, and forwards events, metrics, logs, and other data to a variety of backends, including InfluxDB, Prometheus, Elasticsearch, Kafka, and HTTP endpoints. It is designed to be backend-agnostic, allowing easy configuration changes to switch destinations.

Mutual TLS Secured Connection

As audit events may containg sensitive information, transferring these will require a secure tunnel using a proven and battle-tested technology called Mutual TLS.

For simplicity we will be using the mkcert tool to generate mTLS certificates. These certificates will support RSA public key algorithm and have a 2048 bit public key. For production use you will want to use other tools like openssl, step-ca and cert-manager.

Setup

1. Install the following tools: telegraf, mkcert, curl and kubectl

2. Stop the Telegraf service when active:

   % systemctl stop telegraf
   

3. Create a directory to hold the mTLS files:

   % mkdir -p /etc/telegraf/certs/
   % cd /etc/telegraf/certs/
   

4. Generate the mTLS files:

   # Create a new Certificate Authority
   % export CAROOT=$PWD
   % mkcert -install
   
   # Add the VMs public IPv4 as a SAN to the server certificate
   % IP=$(curl -s -4 ifconfig.me)
   
   # Generate server certificate with SANs
   % mkcert -key-file server.key -cert-file server.crt auditing-server localhost $IP
   
   # Generate client certificate
   % mkcert -client -key-file client.key -cert-file client.crt auditing-client
   

   Files generated:

   % ls -1
   ca.crt
   client.crt
   client.key
   rootCA-key.pem
   rootCA.pem
   server.crt
   server.key
   

5. Make sure the telegraf user can access generated files:

   % chown telegraf:telegraf -R .
   

6. Replace the default Telegraf configuration at /etc/telegraf/telegraf.conf, update accordingly:

   [global_tags]
   [agent]
     interval = "10s"
     round_interval = true
     metric_batch_size = 1000
     metric_buffer_limit = 10000
     collection_jitter = "0s"
     flush_interval = "10s"
     flush_jitter = "0s"
     precision = "0s"
   
   [[inputs.http_listener_v2]]
     service_address = ":9999"
     paths = ["/events"]
   
     tls_cert = "/etc/telegraf/certs/server.crt"
     tls_key = "/etc/telegraf/certs/server.key"
     tls_allowed_cacerts = ["/etc/telegraf/certs/rootCA.pem"]
     tls_min_version = "TLS13"
   
     methods         = ["POST"]
     data_source     = "body"
     data_format = "json_v2"
   
     [[inputs.http_listener_v2.json_v2]]
       measurement_name = "k8s_audit"
   
       [[inputs.http_listener_v2.json_v2.object]]
         path              = "items"
         timestamp_key     = "stageTimestamp"
         timestamp_format  = "2006-01-02T15:04:05.999999999Z"
         tags = ["auditID", "requestURI", "userAgent", "verb"]
   
   [[outputs.file]]
     files = ["/var/log/telegraf/events.json"]
     rotation_max_size = "100MB"
     rotation_max_archives = 30
     data_format = "json"
     json_timestamp_units = "1s"
   

7. Start the Telegraf service:

   systemctl start telegraf
   

8. Validate the /events endpoint is accessible from the VMs public IPv4 address and TLS handshake is succesful:

   % curl -i -X POST -s https://$IP:9999/events \
     --cacert /etc/telegraf/certs/rootCA.pem \
     --key /etc/telegraf/certs/client.key \
     --cert /etc/telegraf/certs/client.crt
   

   Any response but a timeout or certificate mismatch should indicate the service is ready to accept audit events.

9. Create a Kubernetes Secret in your EMK project containing the mTLS client certificate, use the EMK service account:

   kubectl create secret generic audit-mtls-client \
     --from-file=ca.crt=./rootCA.pem \
     --from-file=client.crt=./client.crt \
     --from-file=client.key=./client.key
   

   The audit log forwarder inside your cluster's control plane uses the client certificate to establish a secure connection to the Telegraf service.

10. Create an audit policy for kube-apiserver in your EMK project containing rules to match events you want to forward, use the EMK service account:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: audit-policy
    data:
      policy: |-
        apiVersion: audit.k8s.io/v1  
        kind: Policy  
        rules:  
        # Log configmap and secret changes in all other namespaces at the Metadata level.
        - level: Metadata
          resources:
          - group: "" # core API group
            resources: ["secrets", "configmaps"]
    

    See the Kubernetes documentation on audit policies for more information on configuring audit policy rules.

    Apply the audit policy:

    % kubectl apply -f audit-policy.yaml
    

Now continue with the next step to enable the auditing extension in your EMK cluster.

Enable auditing extension

Update your EMK cluster configuration in order of appearance:

spec.kubernetesspec.resourcesspec.extensions

Enable the audit policy:

kind: Shoot
spec:
  kubernetes:
    kubeAPIServer:
      auditConfig:
        auditPolicy:
          configMapRef:
            name: audit-policy

Add the audit-mtls-client secret as a resources:

kind: Shoot
spec:
  resources:
  - name: audit-mtls-client
    resourceRef:
      apiVersion: v1
      kind: Secret
      name: audit-mtls-client

Enable and configure the audit extension, replace $IP with the VM's public IPv4 address:

kind: Shoot
spec:
  extensions:
  - type: auditing
    providerConfig:
      apiVersion: auditing.extensions.gardener.cloud/v1alpha1
      kind: AuditConfiguration
      backends:
      - http:
          url: https://$IP:9999/events
          tls:
            secretReferenceName: audit-mtls-client

After saving the updated YAML configuration, the cluster will reconcile and apply the settings automatically.

The following options are available to configure the auditing extension:

• The providerConfig.backends field is a list of backends that will receive audit logs.
• The providerConfig.backends[*].http.url field is
• The providerConfig.backends[*].http.tls.secretReferenceName field is a name reference that leads to a Secret containing the TLS configuration
• The providerConfig.backends[*].deliveryMode optional field to specify how messages are delivered to this backend (Guaranteed or BestEffort)

Collect audit events

When your EMK cluster is finished reconciling, the log forwarder inside your cluster's control plane is ready and events matching the rules in your audit policy should now be collected in the events log setup in Telegraf,

Watch for collected audit events:

% tail -f /var/log/telegraf/events.json

Disabling the extension

Remove the auditing entry from spec.extensions. The extension will clean up deployed resources. The audit policy and mTLS secret will remain but can be deleted if it is no longer needed.

Closing Remarks

This setup has shown how to log audit events to file. For a production environment you will want to save audit events to a datastore for further processing and review. Fortunately Telegraf supports multiple backends making integrations easier.