Note
This deployment is an opinionated collection of offensive and defensive open-source security and observability tools. It contains destructive elements, and it is NOT to be used in production under any circumstances
Currently, we are rewriting the stack a second time with pixie becoming the user-facing component with adapters for data-src and data-sinks. Most of the wiki/Readme should still work
What this project is: essentially an open-source SOC for k8s built on the most amazing projects out there, so that you can visualize ongoing attacks, post-exploitation.
Welcome to the K8sStormCenter main repository. Here you will find everything you need to set up your own HoneyCluster: a Kubernetes cluster that is instrumented with bait and tripwires to collect data on the attacks carried out against it. With our complimentary lightening-rod , the attack paths can be visualized and are made shareable using STIX. You can also use the lightening-feature to attack yourself to create a threat-model-starting point or to generally scan your setup. The storm feature is the collection of CTI from many HoneyClusters into a open shared threat-intel collection.
We expressely thank the upstream maintainers of our main components: KubeHound, KubeScape, Vector, Tetragon, Falco, Oasis/STIX, Kubesploit and Pixie
If you like it -> consider leaving a star ⭐
- K8sStormCenter HoneyCluster
You start with your "normal" cluster, where you wish to
- verify/quantify theoretical threat modelling assumptions, or to
- simply observe how your cluster will be attacked by interpreting the anomalous signals
However, you don't want to expose your cluster to real threats. That's where the HoneyCluster comes in. A HoneyCluster is a cluster that looks like a normal cluster, but is instrumented with tripwires and bait to collect data on the attacks carried out against it. Because the HoneyCluster looks like a real cluster, the collected data is representative of the attacks that would be carried out against a real cluster and therefore can give us insights into the behaviour of attackers and the ways in which they target our clusters. You decide, how much of your production scope you ll clone into this "HoneyCluster". We recommend, using the same IaC you already have and put it into a throw-away tenant(Azure)/project(GCP). I d recommend additionally, not using any IAM/IdP or other identiies or key material that has scopes beyond this tenant. Especially if you want to see critical kind of attacks... Please note, that you are solely responsible for the risk of lateral-movements/pivots, if you deploy the HoneyCluster in shared- configuration.
WIP: I ll add GCP terraform templates including all IAM to work standalone, as a reference.
To set up a HoneyCluster, you start by creating a copy of your "normal" cluster. This copy has the same services as the real cluster, but without its sensitive data and with some additional instrumentation to collect data on the attacks carried out against it. This data is then used to create a baseline of normal behaviour, which is used to filter out benign signals. The remaining signals are then used to detect and understand the attacks carried out against the cluster.
The HoneyCluster is a tool to gather data about attack patterns, but on its own, it does not provide us with any actionable insights. To attain knowledge and insights about attacks, the following steps are necessary:
- Threat Model -> Attack Model -> Critical Attack Path
- Instrument a honeycluster with eBPF tripwires and some bait
- Trace and stream events, remove baseline
- Disseminate the Threat Intelligence
A relatively generic Threat Model could for example look like this:
flowchart TD
A[Access sensitive data] --> B{Establish \npersistence}
A --> C{Command \nand Control}
B --> BB[Pod with \nwriteable hostPath]
A --> BB
A --> H[Pod uses PVC \nwhich references a \nhostPath PV]
B --> BC[crontab \non node]
B --> BD[OR static pod ]
B --> C
C --> D{Weapon \npod}
H --> D
C --> DE[reverse shell]
D --> E[App Vulnerability\nallows RCE]
D --> EE[ServiceAccount \n creates Pod]
WIP: replace with more generic
One of the critical aspects of modelling threats is having a good threat model to start with. This is why we looked at the cloud-native OSS space and are collecting the best tools and combining them to generate a good starting point. We thank the Kubehound maintainers for a lot of inspiration!
Based on the Threat Model, you now create a concrete AttackModel (or many).
You can use this for multiple purposes:
- to understand if a ThreatModel can be exploited IRL . You can attack yourself or hire an offensive expert, create a bug bounty program etc.
- to calibrate all event-producing instrumentation in your deployment: can you see events from executing your attack-model, if not, you might need to add more TracingPolicies or change some filters.
- to verify the pattern-matching between your events and your STIX observables: the
cti-stix-visualizerUI can help calibrating - to simulate a breach: once you have at least one attack model implemented (e.g. via bash-script), you can test diverse detective/responsive processes in your deployment, e.g. if your pager starts blinking.
You want to deploy your own HoneyCluster? We'll guide you through the setup of a local Kubernetes (kind) cluster, the installation of the necessary instrumentation and the collection of the baseline behaviour. Once you have set up your HoneyCluster, you can start experimenting with a standardized attack-suite, writing patterns, analysing logs, calibrating a baseline and observing the final signals.
Note
If you have already set up a Kubernetes cluster you want to utilize as a HoneyCluster, you can skip this part and head to step 2. Make sure to have cert-manager installed on it.
In order to set up your own HoneyCluster, you first need a Kubernetes cluster on which you can install the necessary instrumentation. For local explorative scenarios we provide a Kubernetes setup based on Kind, which you can create using the following command:
make cluster-upImportant
Make sure to have the correct Kubernetes config context set. From this point on we will make changes to the cluster specified as the current context, and we wouldn't want them to end up on the wrong cluster.
Once you have a running Kubernetes cluster, you can go ahead and install the instrumentation to put the honey in your cluster with the following command:
make honey-upThe honey-up target installs all the necessary components to collect eBPF traces, application logs and soon audit logs. It is important that the cluster is not yet exposed to an active threat at this point because after the installation we collect the baseline behaviour of the cluster to filter out benign signals.
While we provide you with a set of default traces and log forwarding configurations, you can adjust them to your needs in the traces and honeystack/vector/values.yaml files respectively. Find out more about it in the Tailoring the instrumentation to your needs section.
While in Calibration Mode , you ll likely want to work with a small batch of logs to get the coarse filters in place:
kubectl port-forward service/redis-headless -n storm 6379:6379
kubectl port-forward service/stix-visualizer -n storm 80:3000
kubectl port-forward service/lighteningrod -n storm 8000:8000After port-forwarding, you can access the UI http://localhost:30000, edit your Patterns and activate Patterns to be used by the lightening-rod (a microservice that matches patterns and converts logs to a standard STIX 2.1 format)
WIP: For the Kubehound-inspired Calibration, there is a set of sample patterns lightening-rod/testpost.sh. You should add/modify: e.g. if you click on Add Pattern, it'll give you the minimal example of one to fill out:
At this point, you ll likey want to mark all your current logs as benign, this will reduce the noise considerably.
If you want to be sure that your HoneyCluster has reached equilibrium, check the rate at which logs are appended to the redisDB "table"=tetra . On kind, the rate will go to zero after the all services are booted up.
You achieve this, first put on all traces, then load all your current logs into a temporary table:
make --makefile=Makefile_calibrate_kubehound calibration-traces
http://localhost:3000/reload-tetra
kubectl port-forward -n storm lightening-rod 8000:8000
chmod +x lightening-rod/testpost.sh
./lightening-rod/testpost.sh
and once all the preps are done: press the Mark all logs as BENIGN button in the UI.
At this point (if you inspect redis), you have 4 tables:
tetra = list of all logs (vector-sink)
raw_logs = temporary list to stage logs for processing
benign_logs = hashtable for md5-hashes of bengin logs
tetra_pattern = hashtable for json STIX 2.1 attack-patterns
Important
Not all attacks are implemented, not all traces make sense ATM, this is a great contribution opportunity
Attack yourself with the calibration attacks in the namespace lightening to trigger the tripwires (else running the analysis over the Active Patterns is not going to turn up any matched STIX-bundles). You can choose between calibration or lightening depending on
make --makefile=Makefile_calibrate_kubehound calibrate
or
make lightening
Give it a minute or two until those attacks that will succeed have succeeded (pull their images or booted up etc), then you should remove the lightening again (else you ll have duplicate logs that contain no new insights)
make --makefile=Makefile_calibrate_kubehound wipe
or
make lightening-off
Now, in your redis DB, the table tetra will have accumulated a few thousand logs.
Now, back in the UI, press Select all logs for processing
With the HoneyCluster set up and the baseline behaviour filtered out, and the calibration being sucessful (i.e. your traces, patterns, filters and other yaml files are edited to your satisfaction), we can now attack the cluster with ever more real scenarios.
It is recommended to test the quality of your threat-observability by running some simple attacks that - unlike the calibration suite- are not 100% pre-configured.
This repo provides some examples/ideas YMMV: In the attack makefile, we provide you with a simple simple bait to deploy and attack execute on the cluster. The bait consists of an ssh server with weak credentials, giving an attacker root access on the pod from which more damage can be done because of misconfigured RBAC. To deploy the bait:
make --makefile=Makefile_attack baitAfterwards, you can simply try connecting to the ssh server first:
make --makefile=Makefile_attack ssh-connectWhen prompted, the password is root. You should see the newly established ssh connection being picked up by the eBPF traces.
Now that you have gotten a feel of how unusual traffic is picked up by the HoneyCluster, you can try executing an actual attack. For that, we prepared an attack path made possible if an attacker can create /var/log hostPath Persistent Volumes on a cluster, inspired by this blog post, depicted in the following attack tree.
flowchart TD
A[Access sensitive \ninfo on node] --> B{kubectl logs BAD_POD}
A --> H[Pod uses PVC \nwhich references a \nhostPath PV]
B --> I[Pod writes symlink \nto 0.log file]
B --> C[Container can \nrun as root]
B --> D[Pod with \nwriteable hostPath \nto /var/log]
D --> E{Ability to create \nK8s resources}
H --> E
E --> F[Misconfigured RBAC]
E --> G[Initial access \nto Pod]
Close the SSH connection, and run the full attack which will again make an SSH connection to our vulnerable server, run a malicious script which will create a HostPath type PersistentVolume, allowing a pod to access /var/log on the host (inspired by this blog post), using the Python Kubernetes client library. Note that you could modify the hostPath in the Python script to go directly for the data on the host that you want to compromise, however, in order to increase the number of attack steps in our scenario (and hence the number of indicators that we can look for), let's imagine that we are not able to create arbitrary hostPaths. In this scenario, perhaps a hostPath type PersistentVolume is allowed for /var/log so that a Pod can monitor other Pod's logs.
WIP: TODO: check if below commands are still supported
make --makefile=calibrate_kubehound calibrate
make --makefile=Makefile_attack attackWhen prompted, the password is again root.
If the service account compromised by our attacker could inspect the logs of the containers it can create, running kubectl logs bad-pv-pod --tail=-1 (or making an API call from within the bad pod) will enable an attacker to view arbitrary files (line by line) on the host. In this example, we have a single node cluster, so we can access control plane data.
TOOO: insert SSH detction here.
After you have finished experimenting with your HoneyCluster, you can remove the bait:
make --makefile=Makefile_attack bait-delete
make --makefile=calibrate_kubehound wipeAnd finally, you can wipe the HoneyCluster instrumentation from your Kubernetes cluster:
make wipeThis repository gives you a framework to run experiments, simulations and to make threat-modelling concrete and actionable.
The "Stormcenter" contains a blue (honey and lightening-rod) and a red (lightening and storm) half: blue is all about bait, observability, detection and response while red is about having a clear entry-point for anyone to use their red-tooling and measure the damage or attack-efficiency.
In the end, I m interested in understanding how humans behave in cyber-systems - on either side.
This paragraph is about choosing Tetragon tracing policies that work for you. Tetragon uses eBPF technology to trace kernel and system events, providing detailed insights into system behavior.
Here's an example tracing policy:
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
name: "monitor-network-activity-outside-cluster-cidr-range"
spec:
kprobes:
- call: "tcp_connect"
syscall: false
args:
- index: 0
type: "sock"
selectors:
- matchArgs:
- index: 0
operator: "NotDAddr"
values:
- 127.0.0.1
- 172.16.0.0/28
- 192.168.64.0/24In this example, the policy monitors tcp_connect events, filtering out connections to specific IP ranges and capturing only those to other addresses. This helps ensure that only relevant and interesting tcp information is gathered. See subfolder /traces for more examples.
This paragraph is about application (incl audit) and networking logs. WIP
Coming soon: examples and how to test it locally
The collected logs are transformed into STIX observables, which are then matched against STIX indicators. Observables, which match the provided indicators represent potentially malicious behavior and are persisted into a document store. More detailed information on the setup of the indicators and how the matching works is provided in the README of the threatintel repository.
We show a simple and unspecific detection of Leaky Vessel via Supply Chain (cf. KubeCon Europe 2024) and an elaborate breach using Leaky Vessel for priviledge escalation (cf. KCD Munich 2024). No additional cluster instrumentation was needed, no specific assumptions were made, etc.
You can watch a recording of the KCD Munich 2024 talk here:
The KubeCon Europe 2024 talk can be viewed here:
The video above shows the poisoning of a registry with an image exploiting CVE-2024-21626 "Leaky-Vessel" by tagging and pushing the poisoned image with identical name/tag as the original image. (This is a type of Supply Chain Attack).
Two different RKE2 clusters (intentionally running a vulnerable runc) are observed by streaming the smb topic in the Redpanda UI. When the poisoned image is pulled and started up, the traces appear on the topic. As well as we see the sensitive-file-access to the private key on the host-node, as well as the newly created file LEAKYLEAKY on the host node.
We don't share details about the bait used in our experiments here since we don't want potential attackers to know what to look for. However, if you are interested feel free to reach out to us on our Slack channel!
Given this is an insecure and experimental setup of a honeypot-infrastructure, there are several additional measures taken that are not covered in the talk or this repo. This repo is for demonstration purposes only.
Contributions are always welcome!
(For example in the form of testing, feedback, code, PRs, eBPF tripwires, realistic threatmodels, mappings onto the critical attack path or anything you think could be useful for others)
TODO: write contributor guidelines