Skip to content

SCONE on Kubernetes

This tutorial shows how to deploy confidential applications with the help of SCONE in a Kubernetes cluster.

The tutorial is organized as follows:

  1. Before you begin
  2. Hello World!
  3. Hello World! with remote attestation
  4. Hello World! with TLS certificates auto-generated by CAS
  5. Encrypt your source code

Before you begin

A Kubernetes cluster and a separate namespace

This tutorial assumes that you have access to a Kubernetes cluster (through kubectl), and that you are deploying everything to a separate namespace, called hello-scone. Namespaces are a good way to separate resources, not to mention how it makes the clean-up process a lot easier at the end:

kubectl create namespace hello-scone

Don't forget to specify it by adding -n hello-scone to your kubectl commands.

Set yourself a workspace

This tutorial generates a number of files, which need to be stored in a specific location. You may run the following commands in any directory of your choice; however, it is recommended to create a temporary directory to keep things organized.

cd $(mktemp -d)
Install SCONE Operator

SGX applications need access to the Intel SGX driver. We provide an SGX device plugin for Kubernetes, which lets you request the SGX driver through the resource.limits field (just like CPU or RAM). The device plugin handles different driver versions transparently, and also allows your applications to run unprivileged (which is always good for security).

NOTE: For IAS drivers, it can be quite difficult to get SGX to run: Please use DCAP-capable servers instead.

To install the SGX device plugin on your Kubernetes cluster, you can use the SCONE Operator. The SCONE Kubernetes Operator automates the management of SCONE-related services. Once deployed, it will install both the SGX device plugin and the Local Attestation Service (LAS), futher required for remote attestation.

The SCONE Operator can be easily installed with:

export SCONE_REGISTRY_USERNAME=<YOUR-SCONTAIN-REGISTRY-USERNAME>
export SCONE_REGISTRY_EMAIL=<YOUR-EMAIL-THAT-YOU-REGISTERED-AT-SCONTAIN-REGISTRY>
export SCONE_REGISTRY_ACCESS_TOKEN=<YOUR-SCONTAIN-REGISTRY-TOKEN>
export DCAP_KEY=<YOUR-DCAP-KEY>
export SCONE_VERSION=5.9.0-rc.9

curl -fsSL https://raw.githubusercontent.com/scontain/SH/master/$SCONE_VERSION/operator_controller | bash -s - \
  --plugin --reconcile --update --secret-operator --verbose \
  --username $SCONE_REGISTRY_USERNAME \
  --access-token $SCONE_REGISTRY_ACCESS_TOKEN \
  --email $SCONE_REGISTRY_EMAIL \
  --dcap-api $DCAP_KEY \
  --set-version $SCONE_VERSION

Please refer to SCONE Operator user manual for more details.

NOTE: If you don't have Intel SGX in your cluster, you can run SCONE in simulated mode by setting the environment variable SCONE_MODE=sim. Intel SGX security guarantees do not apply in this case.

Set a Configuration and Attestation Service (CAS)

When remote attestation enters the scene, you will also need a CAS to provide attestation, configuration and secrets. To set a CAS in the cluster you should run:

kubectl provision cas cas -n hello-scone

This tutorial assumes that at the end of this step you should have a cluster with a dedicated hello-scone namespace, the SCONE Operator installed (including the SGX device plugin and LAS), and a CAS running.

$ kubectl get cas -n hello-scone 

NAME   STATUS    PHASE     PROVISIONED   ATTESTABLE   SNAPSHOTS   MIGRATION   VERSION      AGE
cas    HEALTHY   HEALTHY   Yes           Yes          Persisted   3/3         5.9.0-rc.9   7m27s

Hello World!

Let's start by writing a simple Python webserver: it only returns the message Hello World! and the value of the environment variable GREETING.

mkdir -p app
cat > app/server.py << EOF
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import os


class HTTPHelloWorldHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        """say "Hello World!" and the value of \`GREETING\` env. variable."""
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello World!\n\$GREETING is: %s\n' % (os.getenv('GREETING', 'no greeting :(').encode()))


httpd = HTTPServer(('0.0.0.0', 8080), HTTPHelloWorldHandler)

print("Starting server")

httpd.serve_forever()
EOF

To build this application with SCONE, simply use our Python curated image as base. The Python interpreter will run inside of an enclave!

export BASE_IMAGE="registry.scontain.com/sconecuratedimages/python:3.10.6-alpine3.16-scone5.9.0"

cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 8080
COPY app /app
CMD [ "python3", "/app/server.py" ]
EOF

Set the name of the image. You must have push permissions to the chosen repo! If you already have an image, skip the building process.

export IMAGE=<YOUR REGISTRY>/<YOUR REPO>:hello-k8s-scone0.1
docker build --pull . -t $IMAGE && docker push $IMAGE

Deploying the application with Kubernetes is also simple: we just need a deployment and a service exposing it.

NOTE: If your image is stored in a private registry, don't forget to configure the secrets to pull it in your Kubernetes manifest. Take the command below in the terminal. 

kubectl create secret docker-registry k8s-scone-demo-pull-secrets \
  --docker-server="<YOUR REGISTRY>" \
  --docker-username="<YOUR REGISTRY USERNAME>" \
  --docker-password="<YOUR REGISTRY ACCESS TOKEN>" \
  --docker-email="<YOUR REGISTRY EMAIL>" \
  --namespace hello-scone
And modify all the application Kubernetes manifest below adding the section imagePullSecrets. 
spec:
  containers:
  - name: hello-world
    ...
  imagePullSecrets:
  - name: k8s-scone-demo-pull-secrets

Write the Kubernetes manifests:

cat > app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  selector:
    matchLabels:
      run: hello-world
  replicas: 1
  template:
    metadata:
      labels:
        run: hello-world
    spec:
      containers:
      - name: hello-world
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: SCONE_ALLOW_DLOPEN
          value: "2"
        - name: GREETING
          value: howdy!
        resources:
          limits:
            sgx.intel.com/enclave: 1
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
  labels:
    run: hello-world
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    run: hello-world
EOF

NOTE: SCONE_ALLOW_DLOPEN="2" should only be used for debugging. It allows loading of unauthenticated, unencrypted libraries after startup (e.g., Python modules). For production, use SCONE_ALLOW_DLOPEN="1" or leave it undefined. Never use "2" in production.

Now, submit the manifests to the Kubernetes API, using kubectl command:

kubectl create -f app.yaml -n hello-scone

Now that everything is deployed, you can access your Python app running on your cluster. Forward your local 8080 port to the service port:

kubectl port-forward svc/hello-world 8080:8080 -n hello-scone &

The application will be available at your http://localhost:8080:

$ curl localhost:8080
Hello World!
$GREETING is: howdy!

Run with remote attestation

SCONE provides a remote attestation feature, so you make sure your application is running unmodified. It's also possible to have secrets and configuration delivered directly to the attested enclave!

The remote attestation is provided by two components: LAS (local attestation service, runs on the cluster) and CAS (a trusted service that runs elsewhere. We provide a public one in scone-cas.cf).

To setup remote attestation, you will need a session file and the MRENCLAVE, which is a unique signature of your application. Extract MRENCLAVE of your application by running its container with the environment variable SCONE_HASH set to 1:

MRENCLAVE=$(docker run -i --rm -e "SCONE_HASH=1" -e "SCONE_ALLOW_DLOPEN=2" $IMAGE)

Then create a session file. Please note that we are also defining a secret GREETING. Only the MRENCLAVES registered in this session file will be allowed to see such secret.

cat > session.yaml << EOF
name: hello-k8s-scone
version: "0.3.11"

security:
  attestation:
    tolerate: [debug-mode, hyperthreading, outdated-tcb, software-hardening-needed, insecure-configuration]
    ignore_advisories: "*"

services:
   - name: application
     mrenclaves: [$MRENCLAVE]
     command: python3 /app/server.py
     pwd: /
     environment:
        GREETING: hello from SCONE!!!
EOF

Now, post the session file to your CAS:

kubectl port-forward sts/cas 8081:8081 -n hello-scone &
scone cas attest 127.0.0.1:8081 --accept-group-out-of-date --accept-sw-hardening-needed --accept-configuration-needed --allow-cas-owner-secret-access --isvprodid 41316 --retries 10 --isvsvn 5 --mrsigner 195e5a6df987d6a515dd083750c1ea352283f8364d3ec9142b0d593988c6ed2d
scone session create --cas 127.0.0.1:8081 session.yaml

Once you submit the session file, you just need to inject the CAS address into your Deployment manifest. To showcase that, we're creating a new Deployment:

cat > attested-app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attested-hello-world
spec:
  selector:
    matchLabels:
      run: attested-hello-world
  replicas: 1
  template:
    metadata:
      labels:
        run: attested-hello-world
    spec:
      containers:
      - name: attested-hello-world
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: SCONE_CAS_ADDR
          value: cas.hello-scone
        - name: SCONE_CONFIG_ID
          value: hello-k8s-scone/application
        - name: SCONE_LAS_ADDR
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: SCONE_ALLOW_DLOPEN
          value: "2"
        resources:
          limits:
            sgx.intel.com/enclave: 1
---
apiVersion: v1
kind: Service
metadata:
  name: attested-hello-world
  labels:
    run: attested-hello-world
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    run: attested-hello-world
EOF

NOTE: The environment variables defined in the Kubernetes manifest won't be considered once your app is attested. The environment will be retrieved from your session file.

Deploy your attested app:

kubectl create -f attested-app.yaml -n hello-scone

Once again, forward your local port 8082 to the service port:

kubectl port-forward svc/attested-hello-world 8082:8080 -n hello-scone &

The attested application will be available at your http://localhost:8082:

$ curl localhost:8082
Hello World!
$GREETING is: hello from SCONE!!!

TLS with certificates auto-generated by CAS

The CAS service can also generate secrets and certificates automatically. Combined with access policies, it means that such auto-generated secrets and certificates will be seen only by certain applications. No human (e.g. a system operator) will ever see them: they only exist inside of SGX enclaves.

To showcase such feature, let's use the same application as the last example. But now, we serve the traffic over TLS, and we let CAS generate the server certificates.

Start by rewriting our application to serve with TLS:

cat > app/server-tls.py << EOF
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import os
import socket
import ssl


class HTTPHelloWorldHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        """say "Hello World!" and the value of \`GREETING\` env. variable."""
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello World!\n\$GREETING is: %s\n' % (os.getenv('GREETING', 'no greeting :(').encode()))


httpd = HTTPServer(('0.0.0.0', 4443), HTTPHelloWorldHandler)


httpd.socket = ssl.wrap_socket(httpd.socket,
                               keyfile="/app/key.pem",
                               certfile="/app/cert.pem",
                               server_side=True)

print("Starting server")

httpd.serve_forever()
EOF

Our updated Dockerfile looks like this:

cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 4443
COPY app /app
CMD [ "python3" ]
EOF

Build the updated image:

export IMAGE=<YOUR REGISTRY>/<YOUR REPO>:hello-k8s-scone0.2
docker build --pull . -t $IMAGE && docker push $IMAGE

The magic is done in the session file. The certificates are defined in the secrets field, and are injected into the filesystem through images.injection_files:

cat > session-tls-certs.yaml << EOF
name: hello-k8s-scone-tls-certs
version: "0.3.11"

security:
  attestation:
    tolerate: [debug-mode, hyperthreading, outdated-tcb, software-hardening-needed, insecure-configuration]
    ignore_advisories: "*"

services:
   - name: application
     image_name: application_image
     mrenclaves: [$MRENCLAVE]
     command: python3 /app/server-tls.py
     pwd: /
     environment:
        GREETING: hello from SCONE with TLS and auto-generated certs!!!

images:
   - name: application_image
     injection_files:
       - path:  /app/cert.pem
         content: \$\$SCONE::SERVER_CERT:crt\$\$
       - path: /app/key.pem
         content: \$\$SCONE::SERVER_CERT:privatekey\$\$

secrets:
   - name: SERVER_KEY
     kind: private-key
   - name: SERVER_CERT
     kind: x509
     private_key: SERVER_KEY
EOF

Post the session file to your CAS:

scone session create --cas 127.0.0.1:8081 session-tls-certs.yaml

The steps to run your app are similar to before: let's create a Kubernetes manifest and submit it.

cat > attested-app-tls-certs.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attested-hello-world-tls-certs
spec:
  selector:
    matchLabels:
      run: attested-hello-world-tls-certs
  replicas: 1
  template:
    metadata:
      labels:
        run: attested-hello-world-tls-certs
    spec:
      containers:
      - name: attested-hello-world-tls-certs
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 4443
        env:
        - name: SCONE_CAS_ADDR
          value: cas.hello-scone
        - name: SCONE_CONFIG_ID
          value: hello-k8s-scone-tls-certs/application
        - name: SCONE_LAS_ADDR
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: SCONE_ALLOW_DLOPEN
          value: "2"
        resources:
          limits:
            sgx.intel.com/enclave: 1
---
apiVersion: v1
kind: Service
metadata:
  name: attested-hello-world-tls-certs
  labels:
    run: attested-hello-world-tls-certs
spec:
  ports:
  - port: 4443
    protocol: TCP
  selector:
    run: attested-hello-world-tls-certs
EOF
kubectl create -f attested-app-tls-certs.yaml -n hello-scone

Now it's time to access your app. Again, forward the traffic to its service:

kubectl port-forward svc/attested-hello-world-tls-certs 8083:4443 -n hello-scone &

And send a request:

$ curl -k https://localhost:8083
Hello World!
$GREETING is: hello from SCONE with TLS and auto-generated certs!!!

Encrypt your source code

Moving further, you can run the exact same application, but now the server source code will be encrypted using SCONE's file protection feature, and only the Python interpreter that you register will be able to read them (after being attested by CAS).

Encrypt the source code and filesystem

Let's encrypt the source code using SCONE's Fileshield. Run a SCONE CLI container with access to the files:

docker run -it --rm -e SCONE_MODE=sim -v $PWD:/tutorial $BASE_IMAGE sh

Inside the container, create an encrypted region /app and add all the files in /tutorial/app (the plain files) to it. Lastly, encrypt the key itself.

cd /tutorial
rm -rf app_image && mkdir -p app_image/app
cd app_image
scone fspf create fspf.pb
scone fspf addr fspf.pb / --not-protected --kernel /
scone fspf addr fspf.pb /app --encrypted --kernel /app
scone fspf addf fspf.pb /app /tutorial/app /tutorial/app_image/app
scone fspf encrypt fspf.pb > /tutorial/app/keytag

The contents of /tutorial/app_image/app are now encrypted. Try to cat them from inside the container:

$ cd /tutorial/app_image/app && ls
server-tls.py
$ cat server-tls.py
$�ʇ(E@��
�����Id�Z���g3uc��mѪ�Z.$�__�!)ҹS��٠���(�� �X�Wϐ�{$���|�זH��Ǔ��!;2����>@z4-�h��3iݾs�t�7�<>H4:����(9=�a)�j?��p����q�ߧ3�}��Hظa�|���w-�q��96���o�9̃�Ev�v��*$����TU/���Ӕ��v�G���T�1<ڹ��C#p|i��AƢ˅_!6���F���w�@
                   �C��J���+81�8�
Build the new image

You can now exit the container. Let's build the server image with our files, now encrypted:

cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 4443
COPY app_image /
CMD [ "python3" ]
EOF

Choose the new image name and build it:

export IMAGE=<YOUR REGISTRY>/<YOUR REPO>:hello-k8s-scone0.3
docker build --pull . -t $IMAGE && docker push $IMAGE
Run in Kubernetes

Time to deploy our server to Kubernetes. Let's setup the attestation first, extract SCONE_FSPF_KEY and SCONE_FSPF_TAG:

export SCONE_FSPF_KEY=$(cat app/keytag | awk '{print $11}')
export SCONE_FSPF_TAG=$(cat app/keytag | awk '{print $9}')
export SCONE_FSPF=/fspf.pb

Create a session file:

cat > session-tls.yaml << EOF
name: hello-k8s-scone-tls
version: "0.3.11"

security:
  attestation:
    tolerate: [debug-mode, hyperthreading, outdated-tcb, software-hardening-needed, insecure-configuration]
    ignore_advisories: "*"

services:
   - name: application
     image_name: application_image
     mrenclaves: [$MRENCLAVE]
     command: python3 /app/server-tls.py
     pwd: /
     environment:
        GREETING: hello from SCONE with encrypted source code and auto-generated certs!!!
     fspf_path: $SCONE_FSPF
     fspf_key: $SCONE_FSPF_KEY
     fspf_tag: $SCONE_FSPF_TAG

images:
   - name: application_image
     injection_files:
       - path:  /app/cert.pem
         content: \$\$SCONE::SERVER_CERT:crt\$\$
       - path: /app/key.pem
         content: \$\$SCONE::SERVER_CERT:privatekey\$\$

secrets:
   - name: SERVER_KEY
     kind: private-key
   - name: SERVER_CERT
     kind: x509
     private_key: SERVER_KEY
EOF

Post the session file to your CAS:

scone session create --cas 127.0.0.1:8081 session-tls.yaml

Create the manifest and submit it to your Kubernetes cluster:

cat > attested-app-tls.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attested-hello-world-tls
spec:
  selector:
    matchLabels:
      run: attested-hello-world-tls
  replicas: 1
  template:
    metadata:
      labels:
        run: attested-hello-world-tls
    spec:
      containers:
      - name: attested-hello-world-tls
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 4443
        env:
        - name: SCONE_CAS_ADDR
          value: cas.hello-scone
        - name: SCONE_CONFIG_ID
          value: hello-k8s-scone-tls/application
        - name: SCONE_LAS_ADDR
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: SCONE_ALLOW_DLOPEN
          value: "2"
        resources:
          limits:
            sgx.intel.com/enclave: 1
---
apiVersion: v1
kind: Service
metadata:
  name: attested-hello-world-tls
  labels:
    run: attested-hello-world-tls
spec:
  ports:
  - port: 4443
    protocol: TCP
  selector:
    run: attested-hello-world-tls
EOF
kubectl create -f attested-app-tls.yaml -n hello-scone

Time to access your app. Forward the traffic to its service:

kubectl port-forward svc/attested-hello-world-tls 8084:4443 -n hello-scone &

And send a request:

$ curl -k https://localhost:8084
Hello World!
$GREETING is: hello from SCONE with encrypted source code and auto-generated certs!!!

Clean up

To clean up all the resources created in this tutorial, simply delete the namespace:

kubectl delete namespace hello-scone