Skip to content

Confidential Computing with SCONE

This tutorial shows how to build and deploy containerized CC App (Confidential Compute App) with the help of SCONE. We show how to run this CC App in a Kubernetes cluster.

This is a deep-dive to introduce some of the concepts of SCONE using a hands-on example. Note that the CC App we build is not yet sufficiently secure: e.g., the Python code of the base image can be manipulated. We will show in the second part of this tutorial how to run this securely and how to simplify some of the steps.

The tutorial is organized as follows:

  • Setup describes the initial setup required to run this example.

  • Hello World! shows how to build a Python Hello-World! and run this in an enclave in a Kubernetes cluster. This enclaved service does not yet know any secrets.

  • Hello World! with remote attestation provides a secret GREETING to Hello-World. For this, we define a first SCONE policy to run Hello-World!: this policy passes the secret as an environment variable. It does not yet use https since it misses a certificate and a private key.

  • Hello World! with TLS certificates auto-generated by SCONE CAS extends the policy to inject a certificate and a private key into the file system of Hello-World! So far, our policy only ensures that the expected Python engine runs but not that actually our Hello-World! program runs.

  • Encrypt your source code: we encrypt the Hello-World! program to ensure not only its confidentiality but also its integrity such that an adversary cannot change our program. SCONE ensures that the filesystem was not modified. Another way to look at this is that the Hello-World! program itself and the location where it is executed, is use to authenticate the program and to get the decryption key and all the secrets like the certificates.

In the second part of this tutorial, we will show

  • how to simplify the setup
  • how to ensure that all Python libraries are encrypted
  • how to ensure that this program can only run on the nodes of your Kubernetes cluster
  • how to provide access control to Hello-World!, i.e., only authorized clients can access the service, and
  • how to run Hello-World! in release mode

Script

An executable version of this tutorial can be cloned as follows:

git clone https://github.com/scontain/hello-world-scone-on-kubernetes.git

The code has some more error handling than given in this page. Also, some aspects of the code are not discussed in this tutorial.

Unless you set variable, IMAGENAME, it will eventually fail with an error message since it will not be able to push a certain image. The reason is that you will not be able to execute an encrypted image of another session.

You need to define environment variable IMAGENAME

export IMAGENAME=...

such that it points to an image name that you are permitted to push.

By default this will be set to

export IMAGENAME=sconecuratedimages/kubernetes:hello-k8s-scone0.3

You can run this demo by executing even if you do not yet have access to the community edition:

cd hello-world-scone-on-kubernetes
./run_hello_world.sh

There will be some error messages if you do not have access to the community edition but they should all be tolerated.

Setup

A Kubernetes cluster and a separate namespace

This tutorial assumes that you have access to a Kubernetes cluster (through kubectl), and that you are deploying everything to a separate namespace, called hello-scone. Namespaces are a good way to separate resources, not to mention how it makes the clean-up process a lot easier at the end:

kubectl create namespace hello-scone

Don't forget to specify it by adding -n hello-scone to your kubectl commands.

Set yourself a workspace

A lot of files are created through this tutorial, and they have to be somewhere. Choose whatever directory you want to run the commands below, but it might be a good idea to create a temporary one:

cd $(mktemp -d)

We use the following base image:

export BASE_IMAGE=sconecuratedimages/apps:python-3.7.3-alpine3.10

Have fun!

Set a Configuration and Attestation Service (CAS)

When remote attestation enters the scene, you will need a CAS to provide attestation, configuration and secrets. Export now the address of your CAS. If you don't have your own CAS, use our public CAS instance one at scone-cas.cf:

export SCONE_CAS_ADDR=scone-cas.cf

Now, generate client certificates (needed to submit new sessions to CAS):

mkdir -p conf
if [[ ! -f conf/client.crt || ! -f conf/client-key.key  ]] ; then
    openssl req -x509 -newkey rsa:4096 -out conf/client.crt -keyout conf/client-key.key  -days 31 -nodes -sha256 -subj "/C=US/ST=Dresden/L=Saxony/O=Scontain/OU=Org/CN=www.scontain.com" -reqexts SAN -extensions SAN -config <(cat /etc/ssl/openssl.cnf \
<(printf '[SAN]\nsubjectAltName=DNS:www.scontain.com'))
fi

Hello World!

Let's start by writing a simple Python webserver: it only returns the message Hello World! and the value of the environment variable GREETING.

mkdir -p app
cat > app/server.py << EOF
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import os


class HTTPHelloWorldHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        """say "Hello World!" and the value of \`GREETING\` env. variable."""
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello World!\n\$GREETING is: %s\n' % (os.getenv('GREETING', 'no greeting :(').encode()))


httpd = HTTPServer(('0.0.0.0', 8080), HTTPHelloWorldHandler)


httpd.serve_forever()
EOF

To build this application with SCONE, simply use our Python 3.7 curated image as base. The Python interpreter will run inside of an enclave!

cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 8080
COPY app /app
CMD [ "python3", "/app/server.py" ]
EOF

Set the name of the image. If you already have an image, skip the building process.

export IMAGE=sconecuratedimages/kubernetes:hello-k8s-scone0.1
docker build --pull . -t $IMAGE && docker push $IMAGE

Deploying the application with Kubernetes is also simple: we just need a deployment and a service exposing it.

Write the Kubernetes manifests:

cat > app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  selector:
    matchLabels:
      run: hello-world
  replicas: 1
  template:
    metadata:
      labels:
        run: hello-world
    spec:
      containers:
      - name: hello-world
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: GREETING
          value: howdy!
        volumeMounts:
        - mountPath: /dev/isgx
          name: dev-isgx
        securityContext:
          privileged: true
      volumes:
      - name: dev-isgx
        hostPath:
          path: /dev/isgx
---
apiVersion: v1
kind: Service
metadata:
  name: hello-world
  labels:
    run: hello-world
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    run: hello-world
EOF

Now, submit the manifests to the Kubernetes API, using kubectl command:

kubectl create -f app.yaml -n hello-scone

NOTE: SGX applications need access to the Intel SGX driver, a char device located at /dev/isgx (in the host). If you are in a cluster context, it means that you need such device mounted into the container, and that's the reasoning behind the addition of the volume dev-isgx to the Kubernetes manifest. If you don't have access, or if you are not sure whether the underlying infrastructure has SGX installed, you can run in simulated mode, by adding SCONE_MODE=sim to the environment. This will emulate an enclave for you by encripting the main memory (Intel SGX security guarantees do not apply here).

NOTE: You need special privileges to access a host device. Kubernetes does not allow a fine-grained access to such devices (like a Docker --device option does), so you need to run your container in privileged mode (i.e. your container has access to ALL host devices). This is defined in the container spec, as a securityContext policy (privileged: true). You can avoid this by using our Kubernetes SGX plugin.

Now that everything is deployed, you can access your Python app running on your cluster. Forward your local 8080 port to the service port:

kubectl port-forward svc/hello-world 8080:8080 -n hello-scone

The application will be available at your http://localhost:8080:

$ curl localhost:8080
Hello World! Environment GREETING is: howdy!

Run with remote attestation

SCONE provides a remote attestation feature, so you make sure your application is running unmodified. It's also possible to have secrets and configuration delivered directly to the attested enclave!

The remote attestation is provided by two components: LAS (local attestation service, runs on the cluster) and CAS (a trusted service that runs elsewhere. We provide a public CAS instance at domain scone-cas.cf).

You can deploy LAS to your cluster with the help of a DaemonSet, deploying one LAS instance per cluster node. As your application has to contact the LAS container running in the same host, we use the default Docker interface (172.17.0.1) as our LAS address.

cat > las.yaml << EOF
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: local-attestation
  labels:
    k8s-app: local-attestation
spec:
  selector:
    matchLabels:
      k8s-app: local-attestation
  template:
    metadata:
      labels:
        k8s-app: local-attestation
    spec:
      hostNetwork: true
      volumes:
      - name: dev-isgx
        hostPath:
          path: /dev/isgx
      containers:
        - name: local-attestation
          image: sconecuratedimages/kubernetes:las
          volumeMounts:
          - mountPath: /dev/isgx
            name: dev-isgx
          securityContext:
            privileged: true
          ports:
          - containerPort: 18766
            hostPort: 18766
EOF
kubectl create -f las.yaml -n hello-scone

To setup remote attestation, you will need a session file and the MRENCLAVE, which is a unique signature of your application. Extract MRENCLAVE of your application by running its container with the environment variable SCONE_HASH set to 1:

MRENCLAVE=`docker run -i --rm -e "SCONE_HASH=1" $IMAGE`

Then create a session file. Please note that we are also defining a secret GREETING. Only the MRENCLAVES registered in this session file will be allowed to see such secret.

cat > session.yaml << EOF
name: hello-k8s-scone
version: "0.2"

services:
   - name: application
     image_name: $IMAGE
     mrenclaves: [$MRENCLAVE]
     command: python3 /app/server.py
     pwd: /
     environment:
        GREETING: hello from SCONE!!!

images:
   - name: $IMAGE
EOF

Now, post the session file to your CAS of choice:

curl -v -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @session.yaml -X POST https://$SCONE_CAS_ADDR:8081/v1/sessions

Once you submit the session file, you just need to inject the CAS address into your Deployment manifest. To showcase that, we're creating a new Deployment:

cat > attested-app.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attested-hello-world
spec:
  selector:
    matchLabels:
      run: attested-hello-world
  replicas: 1
  template:
    metadata:
      labels:
        run: attested-hello-world
    spec:
      containers:
      - name: attested-hello-world
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        env:
        - name: SCONE_CAS_ADDR
          value: $SCONE_CAS_ADDR
        - name: SCONE_CONFIG_ID
          value: hello-k8s-scone/application
        - name: SCONE_LAS_ADDR
          value: 172.17.0.1:18766
        volumeMounts:
        - mountPath: /dev/isgx
          name: dev-isgx
        securityContext:
          privileged: true
      volumes:
      - name: dev-isgx
        hostPath:
          path: /dev/isgx
---
apiVersion: v1
kind: Service
metadata:
  name: attested-hello-world
  labels:
    run: attested-hello-world
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    run: attested-hello-world
EOF

NOTE: The environment defined in the Kubernetes manifest won't be considered once your app is attested. The environment will be retrieved from your session file.

Deploy your attested app:

kubectl create -f attested-app.yaml -n hello-scone

Once again, forward your local port 8081 to the service port:

kubectl port-forward svc/attested-hello-world 8081:8080 -n hello-scone

The attested application will be available at your http://localhost:8081:

$ curl localhost:8081
Hello World! Environment GREETING is: hello from SCONE!!!

TLS with certificates auto-generated by CAS

The CAS service can also generate secrets and certificates automatically. Combined with access policies, it means that such auto-generated secrets and certificates will be seen only by certain applications. No human (e.g. a system operator) will ever see them: they only exist inside of SGX enclaves.

To showcase such feature, let's use the same application as last example. But now, we serve the traffic over TLS, and we let CAS generate the server certificates.

Start by rewriting our application to serve with TLS:

cat > app/server-tls.py << EOF
from http.server import HTTPServer
from http.server import BaseHTTPRequestHandler
import os
import socket
import ssl


class HTTPHelloWorldHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        """say "Hello World!" and the value of \`GREETING\` env. variable."""
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b'Hello World!\n\$GREETING is: %s\n' % (os.getenv('GREETING', 'no greeting :(').encode()))


httpd = HTTPServer(('0.0.0.0', 4443), HTTPHelloWorldHandler)


httpd.socket = ssl.wrap_socket(httpd.socket,
                               keyfile="/app/key.pem",
                               certfile="/app/cert.pem",
                               server_side=True)


httpd.serve_forever()
EOF

Our updated Dockerfile looks like this:

cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 4443
COPY app /app
CMD [ "/usr/local/bin/python3" ]
EOF

Build the updated image:

export IMAGE=sconecuratedimages/kubernetes:hello-k8s-scone0.2
docker build --pull . -t $IMAGE && docker push $IMAGE

The magic is done in the session file. First, extract the MRENCLAVE again:

MRENCLAVE=$(docker run -i --rm --device /dev/isgx -e "SCONE_HASH=1" $IMAGE)

Now, create a Session file to be submitted to CAS. The certificates are defined in the secrets field, and are injected into the filesystem through images.injection_files:

cat > session-tls-certs.yaml << EOF
name: hello-k8s-scone-tls-certs
version: "0.2"

services:
   - name: application
     image_name: $IMAGE
     mrenclaves: [$MRENCLAVE]
     command: python3 /app/server-tls.py
     pwd: /
     environment:
        GREETING: hello from SCONE with TLS and auto-generated certs!!!

images:
   - name: $IMAGE
     injection_files:
       - path:  /app/cert.pem
         content: \$\$SCONE::SERVER_CERT.crt\$\$
       - path: /app/key.pem
         content: \$\$SCONE::SERVER_CERT.key\$\$

secrets:
   - name: SERVER_CERT
     kind: x509
EOF

Post the session file to your CAS of choice:

curl -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @session-tls-certs.yaml -X POST https://$SCONE_CAS_ADDR:8081/session

The steps to run your app are similar to before: let's create a Kubernetes manifest and submit it.

cat > attested-app-tls-certs.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attested-hello-world-tls-certs
spec:
  selector:
    matchLabels:
      run: attested-hello-world-tls-certs
  replicas: 1
  template:
    metadata:
      labels:
        run: attested-hello-world-tls-certs
    spec:
      containers:
      - name: attested-hello-world-tls-certs
        image: $IMAGE
        imagePullPolicy: Always
        ports:
        - containerPort: 4443
        env:
        - name: SCONE_CAS_ADDR
          value: $SCONE_CAS_ADDR
        - name: SCONE_CONFIG_ID
          value: hello-k8s-scone-tls-certs/application
        - name: SCONE_LAS_ADDR
          value: "172.17.0.1"
        volumeMounts:
        - mountPath: /dev/isgx
          name: dev-isgx
        securityContext:
          privileged: true
      volumes:
      - name: dev-isgx
        hostPath:
          path: /dev/isgx
---
apiVersion: v1
kind: Service
metadata:
  name: attested-hello-world-tls-certs
  labels:
    run: attested-hello-world-tls-certs
spec:
  ports:
  - port: 4443
    protocol: TCP
  selector:
    run: attested-hello-world-tls-certs
EOF
kubectl create -f attested-app-tls-certs.yaml -n hello-scone

Now it's time to access your app. Again, forward the traffic to its service:

kubectl port-forward svc/attested-hello-world-tls-certs 8083:4443 -n hello-scone

And send a request:

$ curl -k https://localhost:8083
Hello World!
$GREETING is: hello from SCONE with TLS and auto-generated certs!!!

Encrypt your source code

Moving further, you can run the exact same application, but now the server source code will be encrypted using SCONE's file protection feature, and only the Python interpreter that you register will be able to read them (after being attested by CAS).

Encrypt the source code and filesystem

Let's encrypt the source code using SCONE's Fileshield. Run a SCONE CLI container with access to the files:

docker run -it --rm --device /dev/isgx -v $PWD:/tutorial $BASE_IMAGE sh

Inside the container, create an encrypted region /app and add all the files in /tutorial/app (the plain files) to it. Lastly, encrypt the key itself.

cd /tutorial
rm -rf app_image && mkdir -p app_image/app
cd app_image
scone fspf create fspf.pb
scone fspf addr fspf.pb / --not-protected --kernel /
scone fspf addr fspf.pb /app --encrypted --kernel /app
scone fspf addf fspf.pb /app /tutorial/app /tutorial/app_image/app
scone fspf encrypt fspf.pb > /tutorial/app/keytag

The contents of /tutorial/app_image/app are now encrypted. Try to cat them from inside the container:

$ cd /tutorial/app_image/app && ls
server-tls.py
$ cat server-tls.py
$�ʇ(E@��
�����Id�Z���g3uc��mѪ�Z.$�__�!)ҹS��٠���(�� �X�Wϐ�{$���|�זH��Ǔ��!;2����>@z4-�h��3iݾs�t�7�<>H4:����(9=�a)�j?��p����q�ߧ3�}��Hظa�|���w-�q��96���o�9̃�Ev�v��*$����TU/���Ӕ��v�G���T�1<ڹ��C#p|i��AƢ˅_!6���F���w�@
                   �C��J���+81�8�

Build the new image

You can now exit the container. Let's build the server image with our files, now encrypted:

cat > Dockerfile << EOF
FROM $BASE_IMAGE
EXPOSE 4443
COPY app_image /
CMD [ "/usr/local/bin/python3" ]
EOF

Choose the new image name and build it:

export IMAGE3=sconecuratedimages/kubernetes:hello-k8s-scone0.3
docker build --pull . -t $IMAGE3 && docker push $IMAGE3

Run in Kubernetes

Time to deploy our server to Kubernetes. Let's setup the attestation first, so we make sure that only our application (identified by its MRENCLAVE) will have access to the secrets to read the encrypted filesystem, as well as our secret greeting.

Extract the MRENCLAVE:

MRENCLAVE=$(docker run -i --rm --device /dev/isgx -e "SCONE_HASH=1" $IMAGE3)

Extract SCONE_FSPF_KEY and SCONE_FSPF_TAG:

export SCONE_FSPF_KEY=$(cat app/keytag | awk '{print $11}')
export SCONE_FSPF_TAG=$(cat app/keytag | awk '{print $9}')
export SCONE_FSPF=/fspf.pb

Create a session file:

cat > session-tls.yaml << EOF
name: hello-k8s-scone-tls
version: "0.2"

services:
   - name: application
     image_name: $IMAGE3
     mrenclaves: [$MRENCLAVE]
     command: python3 /app/server-tls.py
     pwd: /
     environment:
        GREETING: hello from SCONE with encrypted source code and auto-generated certs!!!
     fspf_path: $SCONE_FSPF
     fspf_key: $SCONE_FSPF_KEY
     fspf_tag: $SCONE_FSPF_TAG

images:
   - name: $IMAGE3
     injection_files:
       - path:  /app/cert.pem
         content: \$\$SCONE::SERVER_CERT.crt\$\$
       - path: /app/key.pem
         content: \$\$SCONE::SERVER_CERT.key\$\$

secrets:
   - name: SERVER_CERT
     kind: x509
EOF

Post the session file to your CAS of choice:

curl -k -s --cert conf/client.crt --key conf/client-key.key --data-binary @session-tls.yaml -X POST https://$SCONE_CAS_ADDR:8081/session

Create the manifest and submit it to your Kubernetes cluster:

cat > attested-app-tls.yaml << EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attested-hello-world-tls
spec:
  selector:
    matchLabels:
      run: attested-hello-world-tls
  replicas: 1
  template:
    metadata:
      labels:
        run: attested-hello-world-tls
    spec:
      containers:
      - name: attested-hello-world-tls
        image: $IMAGE3
        imagePullPolicy: Always
        ports:
        - containerPort: 4443
        env:
        - name: SCONE_CAS_ADDR
          value: $SCONE_CAS_ADDR
        - name: SCONE_CONFIG_ID
          value: hello-k8s-scone-tls/application
        - name: SCONE_LAS_ADDR
          value: 172.17.0.1
        volumeMounts:
        - mountPath: /dev/isgx
          name: dev-isgx
        securityContext:
          privileged: true
      volumes:
      - name: dev-isgx
        hostPath:
          path: /dev/isgx
---
apiVersion: v1
kind: Service
metadata:
  name: attested-hello-world-tls
  labels:
    run: attested-hello-world-tls
spec:
  ports:
  - port: 4443
    protocol: TCP
  selector:
    run: attested-hello-world-tls
EOF
kubectl create -f attested-app-tls.yaml -n hello-scone

Time to access your app. Forward the traffic to its service:

kubectl port-forward svc/attested-hello-world-tls 8082:4443 -n hello-scone

And send a request:

$ curl -k https://localhost:8082
Hello World!
$GREETING is: hello from SCONE with encrypted source code and auto-generated certs!!!

Clean up

To clean up all the resources created in this tutorial, simply delete the namespace:

kubectl delete namespace hello-scone

Author: Clenimar