From 8a8b6acc9de0715225116095acef364fbf80c0fb Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 12 Feb 2019 18:47:22 -0800 Subject: [PATCH] Add k8s audit instructions for kops Also reorder the steps so falco is fully deployed before modifying the apiserver configuration. --- examples/k8s_audit_config/README.md | 52 ++++++++++++------- .../apiserver-config.patch.sh | 25 ++++++--- examples/k8s_audit_config/enable-k8s-audit.sh | 36 +++++++++++++ 3 files changed, 87 insertions(+), 26 deletions(-) create mode 100644 examples/k8s_audit_config/enable-k8s-audit.sh diff --git a/examples/k8s_audit_config/README.md b/examples/k8s_audit_config/README.md index 50012e0bfec..e024e66f761 100644 --- a/examples/k8s_audit_config/README.md +++ b/examples/k8s_audit_config/README.md @@ -1,38 +1,54 @@ # Introduction -The files in this directory can be used to configure k8s audit logging. The relevant files are: +This page describes how to get K8s Audit Logging working with Falco. For now, we'll describe how to enable audit logging in k8s 1.11, where the audit configuration needs to be directly provided to the api server. In 1.13 there is a different mechanism that allows audit confguration to be managed like other k8s objects, but these instructions are for 1.11. + +The main steps are: + +1. Deploy Falco to your K8s cluster +1. Define your audit policy and webhook configuration +1. Restart the API Server to enable Audit Logging +1. Observe K8s audit events at falco -* [audit-policy.yaml](./audit-policy.yaml): The k8s audit log configuration we used to create the rules in [k8s_audit_rules.yaml](../../rules/k8s_audit_rules.yaml). You may find it useful as a reference when creating your own K8s Audit Log configuration. -* [webhook-config.yaml.in](./webhook-config.yaml.in): A (templated) webhook configuration that sends audit events to an ip associated with the falco service, port 8765. It is templated in that the *actual* ip is defined in an environment variable `FALCO_SERVICE_CLUSTERIP`, which can be plugged in using a program like `envsubst`. You may find it useful as a starting point when deciding how to route audit events to the embedded webserver within falco. +## Deploy Falco to your K8s cluster -These files are only needed when using Minikube, which doesn't currently -have the ability to provide an audit config/webhook config directly -from the minikube commandline. See [this issue](https://github.com/kubernetes/minikube/issues/2741) for more details. +Follow the [K8s Using Daemonset](../../integrations/k8s-using-daemonset/README.md) instructions to create a falco service account, service, configmap, and daemonset. -* [apiserver-config.patch.sh](./apiserver-config.patch.sh): A script that changes the configuration file `/etc/kubernetes/manifests/kube-apiserver.yaml` to add necessary config options and mounts for the kube-apiserver container that runs within the minikube vm. +## Define your audit policy and webhook configuration -A way to use these files with minikube to run falco and enable audit logging would be the following: +The files in this directory can be used to configure k8s audit logging. The relevant files are: -#### Start Minikube with Audit Logging Enabled +* [audit-policy.yaml](./audit-policy.yaml): The k8s audit log configuration we used to create the rules in [k8s_audit_rules.yaml](../../rules/k8s_audit_rules.yaml). +* [webhook-config.yaml.in](./webhook-config.yaml.in): A (templated) webhook configuration that sends audit events to an ip associated with the falco service, port 8765. It is templated in that the *actual* ip is defined in an environment variable `FALCO_SERVICE_CLUSTERIP`, which can be plugged in using a program like `envsubst`. -Run the following to start minikube with Audit Logging Enabled: +Run the following to fill in the template file with the ClusterIP ip address you created with the `falco-service` service above. Although services like `falco-service.default.svc.cluster.local` can not be resolved from the kube-apiserver container within the minikube vm (they're run as pods but not *really* a part of the cluster), the ClusterIPs associated with those services are routable. ``` -minikube start --kubernetes-version v1.11.0 --mount --mount-string $PWD:/tmp/k8s_audit_config --feature-gates AdvancedAuditing=true +FALCO_SERVICE_CLUSTERIP=$(kubectl get service falco-service -o=jsonpath={.spec.clusterIP}) envsubst < webhook-config.yaml.in > webhook-config.yaml ``` -#### Create a Falco DaemonSet and Supporting Accounts/Services +## Restart the API Server to enable Audit Logging -Follow the [K8s Using Daemonset](../../integrations/k8s-using-daemonset/README.md) instructions to create a falco service account, service, configmap, and daemonset. +A script [enable-k8s-audit.sh](./enable-k8s-audit.sh) performs the necessary steps of enabling audit log support for the apiserver, including copying the audit policy/webhook files to the apiserver machine, modifying the apiserver command line to add `--audit-log-path`, `--audit-policy-file`, etc. arguments, etc. (For minikube, ideally you'd be able to pass all these options directly on the `minikube start` command line, but manual patching is necessary. See [this issue](https://github.com/kubernetes/minikube/issues/2741) for more details.) + +It is run as `bash ./enable-k8s-audit.sh `. `` can be one of the following: + +* "minikube" +* "kops" -#### Configure Audit Logging with a Policy and Webhook +When running with variant="kops", you must either modify the script to specify the kops apiserver hostname or set it via the environment: `APISERVER_HOST=api.my-kops-cluster.com bash ./enable-k8s-audit.sh kops` -Run the following commands to fill in the template file with the ClusterIP ip address you created with the `falco-service` service above, and configure audit logging to use a policy and webhook that directs the right events to the falco daemonset. Although services like `falco-service.default.svc.cluster.local` can not be resolved from the kube-apiserver container within the minikube vm (they're run as pods but not *really* a part of the cluster), the ClusterIPs associated with those services are routable. +Its output looks like this: ``` -FALCO_SERVICE_CLUSTERIP=$(kubectl get service falco-service -o=jsonpath={.spec.clusterIP}) envsubst < webhook-config.yaml.in > webhook-config.yaml -minikube ssh sudo bash /tmp/k8s_audit_config/apiserver-config.patch.sh +$ bash enable-k8s-audit.sh minikube +***Copying audit policy/webhook files to apiserver... +audit-policy.yaml 100% 2519 1.2MB/s 00:00 +webhook-config.yaml 100% 248 362.0KB/s 00:00 +apiserver-config.patch.sh 100% 1190 1.2MB/s 00:00 +***Modifying k8s apiserver config (will result in apiserver restarting)... +***Done! +$ ``` +## Observe K8s audit events at falco K8s audit events will then be routed to the falco daemonset within the cluster, which you can observe via `kubectl logs -f $(kubectl get pods -l app=falco-example -o jsonpath={.items[0].metadata.name})`. - diff --git a/examples/k8s_audit_config/apiserver-config.patch.sh b/examples/k8s_audit_config/apiserver-config.patch.sh index 95df528a29c..1011e4a5485 100644 --- a/examples/k8s_audit_config/apiserver-config.patch.sh +++ b/examples/k8s_audit_config/apiserver-config.patch.sh @@ -2,7 +2,8 @@ IFS='' -FILENAME="/etc/kubernetes/manifests/kube-apiserver.yaml" +FILENAME=${1:-/etc/kubernetes/manifests/kube-apiserver.yaml} +VARIANT=${2:-minikube} if grep audit-webhook-config-file $FILENAME ; then echo audit-webhook patch already applied @@ -12,23 +13,31 @@ fi TMPFILE="/tmp/kube-apiserver.yaml.patched" rm -f "$TMPFILE" +APISERVER_PREFIX=" -" +APISERVER_LINE="- kube-apiserver" + +if [ $VARIANT == "kops" ]; then + APISERVER_PREFIX=" " + APISERVER_LINE="/usr/local/bin/kube-apiserver" +fi + while read LINE do echo "$LINE" >> "$TMPFILE" case "$LINE" in - *"- kube-apiserver"*) - echo " - --audit-log-path=/tmp/k8s_audit_config/audit.log" >> "$TMPFILE" - echo " - --audit-policy-file=/tmp/k8s_audit_config/audit-policy.yaml" >> "$TMPFILE" - echo " - --audit-webhook-config-file=/tmp/k8s_audit_config/webhook-config.yaml" >> "$TMPFILE" - echo " - --audit-webhook-batch-max-wait=5s" >> "$TMPFILE" + *$APISERVER_LINE*) + echo "$APISERVER_PREFIX --audit-log-path=/var/lib/k8s_audit/audit.log" >> "$TMPFILE" + echo "$APISERVER_PREFIX --audit-policy-file=/var/lib/k8s_audit/audit-policy.yaml" >> "$TMPFILE" + echo "$APISERVER_PREFIX --audit-webhook-config-file=/var/lib/k8s_audit/webhook-config.yaml" >> "$TMPFILE" + echo "$APISERVER_PREFIX --audit-webhook-batch-max-wait=5s" >> "$TMPFILE" ;; *"volumeMounts:"*) - echo " - mountPath: /tmp/k8s_audit_config/" >> "$TMPFILE" + echo " - mountPath: /var/lib/k8s_audit/" >> "$TMPFILE" echo " name: data" >> "$TMPFILE" ;; *"volumes:"*) echo " - hostPath:" >> "$TMPFILE" - echo " path: /tmp/k8s_audit_config" >> "$TMPFILE" + echo " path: /var/lib/k8s_audit" >> "$TMPFILE" echo " name: data" >> "$TMPFILE" ;; diff --git a/examples/k8s_audit_config/enable-k8s-audit.sh b/examples/k8s_audit_config/enable-k8s-audit.sh new file mode 100644 index 00000000000..6e210767ba4 --- /dev/null +++ b/examples/k8s_audit_config/enable-k8s-audit.sh @@ -0,0 +1,36 @@ +#!/bin/bash + +set -euo pipefail + +VARIANT=${1:-minikube} + +if [ $VARIANT == "minikube" ]; then + APISERVER_HOST=$(minikube ip) + SSH_KEY=$(minikube ssh-key) + SSH_USER=docker + MANIFEST="/etc/kubernetes/manifests/kube-apiserver.yaml" +fi + +if [ $VARIANT == "kops" ]; then +# APISERVER_HOST=api.your-kops-cluster-name.com + SSH_KEY=~/.ssh/id_rsa + SSH_USER=admin + MANIFEST=/etc/kubernetes/manifests/kube-apiserver.manifest + + if [ -z "${APISERVER_HOST+xxx}" ]; then + echo "***You must specify APISERVER_HOST with the name of your kops api server" + exit 1 + fi +fi + +echo "***Copying audit policy/webhook files to apiserver..." +ssh -i $SSH_KEY $SSH_USER@$APISERVER_HOST "sudo mkdir -p /var/lib/k8s_audit && sudo chown $SSH_USER /var/lib/k8s_audit" +scp -i $SSH_KEY audit-policy.yaml $SSH_USER@$APISERVER_HOST:/var/lib/k8s_audit +scp -i $SSH_KEY webhook-config.yaml $SSH_USER@$APISERVER_HOST:/var/lib/k8s_audit +scp -i $SSH_KEY apiserver-config.patch.sh $SSH_USER@$APISERVER_HOST:/var/lib/k8s_audit + +echo "***Modifying k8s apiserver config (will result in apiserver restarting)..." + +ssh -i $SSH_KEY $SSH_USER@$APISERVER_HOST "sudo bash /var/lib/k8s_audit/apiserver-config.patch.sh $MANIFEST $VARIANT" + +echo "***Done!"