Skip to content

SCONE on Kubernetes

This tutorial shows how to deploy confidential applications with the help of SCONE in a Kubernetes cluster.

The tutorial is organized as follows:

  1. Before you begin
  2. Hello World!
  3. Hello World! with remote attestation
  4. Hello World! with TLS certificates auto-generated by CAS
  5. Encrypt your source code

Before you begin

A Kubernetes cluster and a separate namespace

This tutorial assumes that you have access to a Kubernetes cluster (through kubectl), and that you are deploying everything to a separate namespace, called hello-scone. Namespaces are a good way to separate resources, not to mention how it makes the clean-up process a lot easier at the end:

kubectl create namespace hello-scone

Don't forget to specify it by adding -n hello-scone to your kubectl commands.

Install SGX device plugin

SGX applications need access to the Intel SGX driver, a char device located at the host machine. The device name might differ depending on the driver implementation, but the most common ones are /dev/sgx/enclave (DCAP) and /dev/isgx (IAS/out-of-tree). Starting on 5.11, the Linux kernel includes the SGX devices at /dev/sgx_enclave and /dev/sgx_provision. If you are in a cluster context, you need the right device mounted into the container. NOTE: For IAS drivers, it can be quite difficult to get SGX to run: Please use DCAP-capable servers instead.

We provide an SGX device plugin for Kubernetes, which lets you request the SGX driver through the resource.limits field (just like CPU or RAM). The device plugin handles different driver names transparently, and also allows your applications to run unprivileged (which is always good for security).

The SGX device plugin can be easily installed with the help of Helm:

helm install sgxdevplugin sconeapps/sgxdevplugin

Please refer to Kubernetes SGX plugin documentation for more details and Helm setup instructions.

This tutorial assumes that you have the SGX device plugin running on your cluster.

NOTE: If you don't have Intel SGX in your cluster, you can run SCONE in simulated mode by setting the environment variable SCONE_MODE=sim. Intel SGX security guarantees do not apply in this case.

Set yourself a workspace

A lot of files are created through this tutorial, and they have to be somewhere. Choose whatever directory you want to run the commands below, but it might be a good idea to create a temporary one:

cd $(mktemp -d)

We use the following base image:

export BASE_IMAGE=registry.scontain.com/sconecuratedimages/kubernetes:python-3.7.3-alpine3.10-scone5.6.0

Have fun!

Set a Configuration and Attestation Service (CAS)

When remote attestation enters the scene, you will need a CAS to provide attestation, configuration and secrets. Export now the address of your CAS. If you don't have your own CAS, use our public one at scone-cas.cf:

export SCONE_CAS_ADDR=scone-cas.cf

Now, generate client certificates (needed to submit a new policy (a.k.a. session) to CAS):

mkdir -p conf
if [[ ! -f conf/client.crt || ! -f conf/client-key.key  ]] ; then
    openssl req -x509 -newkey rsa:4096 -out conf/client.crt -keyout conf/client-key.key  -days 31 -nodes -sha256 -subj "/C=US/ST=Dresden/L=Saxony/O=Scontain/OU=Org/CN=www.scontain.com" -reqexts SAN -extensions SAN -config <(cat /etc/ssl/openssl.cnf \
<(printf '[SAN]\nsubjectAltName=DNS:www.scontain.com'))
fi

Create a CAS namespace for your sessions.

export NS=HelloWorld-$RANDOM-$RANDOM
cat > namespace.yaml <<EOF
name: $NS
version: "0.3"

access_policy:
  read:
   - CREATOR
  update:
   - CREATOR
  create_sessions:
   - CREATOR

EOF
curl -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @namespace.yaml -X POST https://$SCONE_CAS_ADDR:8081/v1/sessions

Hello World!

Let's start by writing a simple Python webserver: it only returns the message Hello World! and the value of the environment variable GREETING.

mkdir -p app
cat > app/server.py << EOF
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import os


class HTTPHelloWorldHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        """say "Hello World!" and the value of \`GREETING\` env. variable."""
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello World!\n\$GREETING is: %s\n' % (os.getenv('GREETING', 'no greeting :(').encode()))


httpd = HTTPServer(('0.0.0.0', 8080), HTTPHelloWorldHandler)

print("Starting server")

httpd.serve_forever()
EOF

To build this application with SCONE, simply use our Python 3.7 curated image as base. The Python interpreter will run inside of an enclave!

cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 8080
COPY app /app
CMD [ "python3", "/app/server.py" ]
EOF

Set the name of the image. You must have push permissions to the choosen repo! If you already have an image, skip the building process.

export IMAGE=registry.scontain.com/sconecuratedimages/kubernetes:hello-k8s-scone0.1
docker build --pull . -t $IMAGE && docker push $IMAGE

Deploying the application with Kubernetes is also simple: we just need a deployment and a service exposing it.

Write the Kubernetes manifests:

cat > app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  selector:
    matchLabels:
      run: hello-world
  replicas: 1
  template:
    metadata:
      labels:
        run: hello-world
    spec:
      containers:
      - name: hello-world
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: GREETING
          value: howdy!
        resources:
          limits:
            sgx.intel.com/enclave: 1
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
  labels:
    run: hello-world
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    run: hello-world
EOF

Now, submit the manifests to the Kubernetes API, using kubectl command:

kubectl create -f app.yaml -n hello-scone

Now that everything is deployed, you can access your Python app running on your cluster. Forward your local 8080 port to the service port:

kubectl port-forward svc/hello-world 8080:8080 -n hello-scone

The application will be available at your http://localhost:8080:

$ curl localhost:8080
Hello World!
$GREETING is: howdy!

Run with remote attestation

SCONE provides a remote attestation feature, so you make sure your application is running unmodified. It's also possible to have secrets and configuration delivered directly to the attested enclave!

The remote attestation is provided by two components: LAS (local attestation service, runs on the cluster) and CAS (a trusted service that runs elsewhere. We provide a public one in scone-cas.cf).

You can deploy LAS to your cluster through our LAS Helm chart:

helm install las sconeapps/las

To setup remote attestation, you will need a session file and the MRENCLAVE, which is a unique signature of your application. Extract MRENCLAVE of your application by running its container with the environment variable SCONE_HASH set to 1:

MRENCLAVE=$(docker run -i --rm -e "SCONE_HASH=1" $IMAGE)

Then create a session file. Please note that we are also defining a secret GREETING. Only the MRENCLAVES registered in this session file will be allowed to see such secret.

cat > session.yaml << EOF
name: $NS/hello-k8s-scone
version: "0.3"

security:
  attestation:
    tolerate: [debug-mode, hyperthreading, outdated-tcb, software-hardening-needed]
    ignore_advisories: "*"

services:
   - name: application
     mrenclaves: [$MRENCLAVE]
     command: python3 /app/server.py
     pwd: /
     environment:
        GREETING: hello from SCONE!!!
EOF

Now, post the session file to your CAS of choice:

curl -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @session.yaml -X POST https://$SCONE_CAS_ADDR:8081/v1/sessions

Once you submit the session file, you just need to inject the CAS address into your Deployment manifest. To showcase that, we're creating a new Deployment:

cat > attested-app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attested-hello-world
spec:
  selector:
    matchLabels:
      run: attested-hello-world
  replicas: 1
  template:
    metadata:
      labels:
        run: attested-hello-world
    spec:
      containers:
      - name: attested-hello-world
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: SCONE_CAS_ADDR
          value: $SCONE_CAS_ADDR
        - name: SCONE_CONFIG_ID
          value: $NS/hello-k8s-scone/application
        - name: SCONE_LAS_ADDR
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        resources:
          limits:
            sgx.intel.com/enclave: 1
---
apiVersion: v1
kind: Service
metadata:
  name: attested-hello-world
  labels:
    run: attested-hello-world
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    run: attested-hello-world
EOF

NOTE: The environment defined in the Kubernetes manifest won't be considered once your app is attested. The environment will be retrieved from your session file.

Deploy your attested app:

kubectl create -f attested-app.yaml -n hello-scone

Once again, forward your local port 8082 to the service port:

kubectl port-forward svc/attested-hello-world 8082:8080 -n hello-scone

The attested application will be available at your http://localhost:8082:

$ curl localhost:8082
Hello World!
$GREETING is: hello from SCONE!!!

TLS with certificates auto-generated by CAS

The CAS service can also generate secrets and certificates automatically. Combined with access policies, it means that such auto-generated secrets and certificates will be seen only by certain applications. No human (e.g. a system operator) will ever see them: they only exist inside of SGX enclaves.

To showcase such feature, let's use the same application as last example. But now, we serve the traffic over TLS, and we let CAS generate the server certificates.

Start by rewriting our application to serve with TLS:

cat > app/server-tls.py << EOF
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import os
import socket
import ssl


class HTTPHelloWorldHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        """say "Hello World!" and the value of \`GREETING\` env. variable."""
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello World!\n\$GREETING is: %s\n' % (os.getenv('GREETING', 'no greeting :(').encode()))


httpd = HTTPServer(('0.0.0.0', 4443), HTTPHelloWorldHandler)


httpd.socket = ssl.wrap_socket(httpd.socket,
                               keyfile="/app/key.pem",
                               certfile="/app/cert.pem",
                               server_side=True)

print("Starting server")

httpd.serve_forever()
EOF

Our updated Dockerfile looks like this:

cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 4443
COPY app /app
CMD [ "python3" ]
EOF

Build the updated image:

export IMAGE=registry.scontain.com/sconecuratedimages/kubernetes:hello-k8s-scone0.2
docker build --pull . -t $IMAGE && docker push $IMAGE

The magic is done in the session file. First, extract the MRENCLAVE again:

MRENCLAVE=$(docker run -i --rm -e "SCONE_HASH=1" $IMAGE)

Now, create a Session file to be submitted to CAS. The certificates are defined in the secrets field, and are injected into the filesystem through images.injection_files:

cat > session-tls-certs.yaml << EOF
name: $NS/hello-k8s-scone-tls-certs
version: "0.3"

security:
  attestation:
    tolerate: [debug-mode, hyperthreading, outdated-tcb, software-hardening-needed]
    ignore_advisories: "*"

services:
   - name: application
     image_name: application_image
     mrenclaves: [$MRENCLAVE]
     command: python3 /app/server-tls.py
     pwd: /
     environment:
        GREETING: hello from SCONE with TLS and auto-generated certs!!!

images:
   - name: application_image
     injection_files:
       - path:  /app/cert.pem
         content: \$\$SCONE::SERVER_CERT:crt\$\$
       - path: /app/key.pem
         content: \$\$SCONE::SERVER_CERT:privatekey\$\$

secrets:
   - name: SERVER_KEY
     kind: private-key
   - name: SERVER_CERT
     kind: x509
     private_key: SERVER_KEY
EOF

Post the session file to your CAS of choice:

curl -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @session-tls-certs.yaml -X POST https://$SCONE_CAS_ADDR:8081/session

The steps to run your app are similar to before: let's create a Kubernetes manifest and submit it.

cat > attested-app-tls-certs.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attested-hello-world-tls-certs
spec:
  selector:
    matchLabels:
      run: attested-hello-world-tls-certs
  replicas: 1
  template:
    metadata:
      labels:
        run: attested-hello-world-tls-certs
    spec:
      containers:
      - name: attested-hello-world-tls-certs
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 4443
        env:
        - name: SCONE_CAS_ADDR
          value: $SCONE_CAS_ADDR
        - name: SCONE_CONFIG_ID
          value: $NS/hello-k8s-scone-tls-certs/application
        - name: SCONE_LAS_ADDR
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        resources:
          limits:
            sgx.intel.com/enclave: 1
---
apiVersion: v1
kind: Service
metadata:
  name: attested-hello-world-tls-certs
  labels:
    run: attested-hello-world-tls-certs
spec:
  ports:
  - port: 4443
    protocol: TCP
  selector:
    run: attested-hello-world-tls-certs
EOF
kubectl create -f attested-app-tls-certs.yaml -n hello-scone

Now it's time to access your app. Again, forward the traffic to its service:

kubectl port-forward svc/attested-hello-world-tls-certs 8083:4443 -n hello-scone

And send a request:

$ curl -k https://localhost:8083
Hello World!
$GREETING is: hello from SCONE with TLS and auto-generated certs!!!

Encrypt your source code

Moving further, you can run the exact same application, but now the server source code will be encrypted using SCONE's file protection feature, and only the Python interpreter that you register will be able to read them (after being attested by CAS).

Encrypt the source code and filesystem

Let's encrypt the source code using SCONE's Fileshield. Run a SCONE CLI container with access to the files:

docker run -it --rm -e SCONE_MODE=sim -v $PWD:/tutorial $BASE_IMAGE sh

Inside the container, create an encrypted region /app and add all the files in /tutorial/app (the plain files) to it. Lastly, encrypt the key itself.

cd /tutorial
rm -rf app_image && mkdir -p app_image/app
cd app_image
scone fspf create fspf.pb
scone fspf addr fspf.pb / --not-protected --kernel /
scone fspf addr fspf.pb /app --encrypted --kernel /app
scone fspf addf fspf.pb /app /tutorial/app /tutorial/app_image/app
scone fspf encrypt fspf.pb > /tutorial/app/keytag

The contents of /tutorial/app_image/app are now encrypted. Try to cat them from inside the container:

$ cd /tutorial/app_image/app && ls
server-tls.py
$ cat server-tls.py
$�ʇ(E@��
�����Id�Z���g3uc��mѪ�Z.$�__�!)ҹS��٠���(�� �X�Wϐ�{$���|�זH��Ǔ��!;2����>@z4-�h��3iݾs�t�7�<>H4:����(9=�a)�j?��p����q�ߧ3�}��Hظa�|���w-�q��96���o�9̃�Ev�v��*$����TU/���Ӕ��v�G���T�1<ڹ��C#p|i��AƢ˅_!6���F���w�@
                   �C��J���+81�8�
Build the new image

You can now exit the container. Let's build the server image with our files, now encrypted:

cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 4443
COPY app_image /
CMD [ "python3" ]
EOF

Choose the new image name and build it:

export IMAGE=registry.scontain.com/sconecuratedimages/kubernetes:hello-k8s-scone0.3
docker build --pull . -t $IMAGE && docker push $IMAGE
Run in Kubernetes

Time to deploy our server to Kubernetes. Let's setup the attestation first, so we make sure that only our application (identified by its MRENCLAVE) will have access to the secrets to read the encrypted filesystem, as well as our secret greeting.

Extract the MRENCLAVE:

MRENCLAVE=$(docker run -i --rm -e "SCONE_HASH=1" $IMAGE)

Extract SCONE_FSPF_KEY and SCONE_FSPF_TAG:

export SCONE_FSPF_KEY=$(cat app/keytag | awk '{print $11}')
export SCONE_FSPF_TAG=$(cat app/keytag | awk '{print $9}')
export SCONE_FSPF=/fspf.pb

Create a session file:

cat > session-tls.yaml << EOF
name: $NS/hello-k8s-scone-tls
version: "0.3"

security:
  attestation:
    tolerate: [debug-mode, hyperthreading, outdated-tcb, software-hardening-needed]
    ignore_advisories: "*"

services:
   - name: application
     image_name: application_image
     mrenclaves: [$MRENCLAVE]
     command: python3 /app/server-tls.py
     pwd: /
     environment:
        GREETING: hello from SCONE with encrypted source code and auto-generated certs!!!
     fspf_path: $SCONE_FSPF
     fspf_key: $SCONE_FSPF_KEY
     fspf_tag: $SCONE_FSPF_TAG

images:
   - name: application_image
     injection_files:
       - path:  /app/cert.pem
         content: \$\$SCONE::SERVER_CERT:crt\$\$
       - path: /app/key.pem
         content: \$\$SCONE::SERVER_CERT:privatekey\$\$

secrets:
   - name: SERVER_KEY
     kind: private-key
   - name: SERVER_CERT
     kind: x509
     private_key: SERVER_KEY
EOF

Post the session file to your CAS of choice:

curl -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @session-tls.yaml -X POST https://$SCONE_CAS_ADDR:8081/session

Create the manifest and submit it to your Kubernetes cluster:

cat > attested-app-tls.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attested-hello-world-tls
spec:
  selector:
    matchLabels:
      run: attested-hello-world-tls
  replicas: 1
  template:
    metadata:
      labels:
        run: attested-hello-world-tls
    spec:
      containers:
      - name: attested-hello-world-tls
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 4443
        env:
        - name: SCONE_CAS_ADDR
          value: $SCONE_CAS_ADDR
        - name: SCONE_CONFIG_ID
          value: $NS/hello-k8s-scone-tls/application
        - name: SCONE_LAS_ADDR
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        resources:
          limits:
            sgx.intel.com/enclave: 1
---
apiVersion: v1
kind: Service
metadata:
  name: attested-hello-world-tls
  labels:
    run: attested-hello-world-tls
spec:
  ports:
  - port: 4443
    protocol: TCP
  selector:
    run: attested-hello-world-tls
EOF
kubectl create -f attested-app-tls.yaml -n hello-scone

Time to access your app. Forward the traffic to its service:

kubectl port-forward svc/attested-hello-world-tls 8084:4443 -n hello-scone

And send a request:

$ curl -k https://localhost:8084
Hello World!
$GREETING is: hello from SCONE with encrypted source code and auto-generated certs!!!

Clean up

To clean up all the resources created in this tutorial, simply delete the namespace:

kubectl delete namespace hello-scone

© scontain.com, 2020-2022. Questions or Suggestions?