Skip to content

Custom Resource Definitions

Custom Resource Definitions

After we have deployed the SCONE operator, we are ready to create SCONE custom resources in the cluster. Before we can deploy them, however, we need to configure them.

Kubernetes custom resources are extensions to the Kubernetes API. They are specified with the help of custom resource definitions or CRDs. The SCONE CRDs are deployed at the time the SCONE operator is installed. You can examine them by executing the following commands:

kubectl describe crd sgxplugins
kubectl describe crd las
kubectl describe crd cas
kubectl describe crd signedpolicies
kubectl describe crd encryptedpolicies
The group, version, and kind of custom resources resulting from the CRDs are summarized in the following table:

Service Group Version Kind Short Name
SCONE CAS services.scone.cloud v1beta1 CAS
SCONE LAS base.scone.cloud v1beta1 LAS
SCONE SGX Plugin base.scone.cloud v1beta1 SGXPlugin sgx
SCONE Signed Policies cas.scone.cloud v1beta1 SignedPolicy spol
SCONE Encrypted Policies cas.scone.cloud v1beta1 EncryptedPolicy epol

Using the following commands, you can list all existing resources of each kind:

kubectl get sgxplugins
kubectl get las
kubectl get cas
kubectl get signedpolicies
kubectl get encryptedpolicies

Since we have not yet created any custom resources, the output of all three commands will be empty. Creating one of these resources will start the corresponding SCONE service in your cluster. Before we create them, however, we want to configure them according to our needs. There is a sample manifest for the custom resource of each service at https://github.com/scontain/operator-samples that you can use as a starting point:

Service Sample Manifest
SCONE CAS ./config/samples/services_v1beta1_cas.yaml
SCONE LAS ./config/samples/base_v1beta1_las.yaml
SCONE SGX Plugin ./config/samples/base_v1beta1_sgxplugin.yaml
SCONE Signed Policies ./config/samples/cas_v1beta1_signedpolicy.yaml
SCONE Encrypted Policies ./config/samples/cas_v1beta1_encryptedpolicy.yaml

Configuring SCONE Services (CAS, LAS, and SGXPlugin)

Common Configuration

The following are configuration parameters that are common to all SCONE services (CAS, LAS, and SGXPlugin)

Name Description Type Default Value
autoUpdate Indicates whether the docker image of the service should automatically be updated to the newest version. This triggers rolling updates of the service pods and assumes that the image has semantic versioning in place. NOTE: autoUpdate is currently disabled in CAS boolean false
image OCI image for the service string most recent SCONE image
imagePullPolicy Indicates when the OCI image should be pulled from the registry Always, Never, IfNotPresent Always
imagePullSecrets ImagePullSecrets for the OCI image. NOTE: The current implementation considers only the first valid secret, even if multiple are supported Object
resources Resource requirements of this SCONE SGX Plugin Object {}
nodeSelector Selects the nodes where the service should run. The NodeSelector must match a node's labels for a pod to be scheduled on that node map[string]string {}

Configuration of the CAS Custom Resource

Name Description Type Default Value
service.type Service type for the CAS pod. Supported values are ClusterIP, LoadBalancer or NodePort v1.ServiceType ClusterIP
service.clientPort Specify a port for the CAS client port integer 8081
service.enclavePort Specify a port for the CAS enclave port integer 18765
service.loadBalancerIP Specify an IP address for your Service if service.type=LoadBalancer and your cloud provider supports it string -
service.nodePorts.clientPort Specify a nodePort for the CAS client port if service.type=NodePort integer -
service.nodePorts.enclavePort Specify a nodePort for the CAS enclave port if service.type=NodePort integer -
service.annotations Specify annotations for the CAS service map[string]string {}
persistence.enabled If set to true, create a PersistentVolumeClaim alongside CAS for persistence boolean false
persistence.mountPath Where to mount CAS persistent volume string /etc/cas/db
persistence.storageClass Define a storageClassName for the provisioned PersistentVolumeClaim. If unset or set to nil (default), use default storageClassName. If set to "-", disable dynamic provisioning string nil
persistence.accessModes PersistentVolume access modes []v1.PersistentVolumeAccessMode ["ReadWriteOnce"]
persistence.size PersistentVolume size resource.Quantity 1Gi

Configuration of the LAS Custom Resource

Name Description Type Default value
hostPort Containers must forward attestation requests to LAS instances collocated in the same node, but this requirement is not achievable through a standard Kubernetes Service abstraction. For this reason, we rely on hostPort to expose LAS and assume that the pods will use the status.hostIP (provided by the Kubernetes API) as SCONE_LAS_ADDR integer 18766
namespace Kubernetes namespace in which the SCONE LAS service should be running string $OPERATOR_NAMESPACE

Configuration of the SGXPlugin Custom Resource

As we can see by executing kubectl explain sgxplugins.spec, we can configure the SCONE SGXPlugin custom resource using the following parameters:

Name Description Type Default value
capacitySGXPods Maximum number of SGX devices to use per Kubernetes node 1<=integer<=10000 Maximum number of pods per node
image Docker image with the SCONE SGX Plugin string registry.scontain.com/ sconecuratedimages/ kubernetes:k8s-sgx-plugin
namespace Kubernetes namespace in which the SCONE SGX Plugin service should be running string $OPERATOR_NAMESPACE

Configuring SCONE Policies (SignedPolicy and EncryptedPolicy)

The custom resources for SCONE Policies have a model that is very similar to the one of the CAS REST API. In this sense, they are not "configurable" but rather an immutable resource that reflects the output of the sign and encrypts SCONE CLI operations. Updating individual fields would lead to an error because it would break integrity.

This means that for every create or update operation on a policy custom resource; the policy owner must re-do the sign or encrypt operations on the unencrypted base policy and then submit the result of the operation to the cluster by creating the corresponding CR. The creation of the CR registers the policy in CAS.

Nonetheless, the custom resources for policies support the following fields.

Fields for the SignedPolicy Custom Resource

Name Description Type
casAddress The target CAS address string
policy The policy contents string
signatures A list of signatures, where each signature is an object comprised of signer and signature fields []Object

Fields for the EncryptedPolicy Custom Resource

Name Description Type
casAddress The target CAS address string
policy The policy contents string
encryptionKey The target CAS public encryption key string

Installing the Services

To install any SCONE services you need to deploy its manifest file. If you configured it to meet your requirements in the previous step, you already have a manifest. If you did not, you could use the sample manifests provided for each service in the above table. To install a service with its manifest at $MANIFEST_FILE, simply execute the following command:

kubectl apply -f $MANIFEST_FILE

Since the SCONE SGX Plugin service is required for the other ones to run, let's use it as an example and install it:

kubectl apply -f ./config/samples/base_v1beta1_sgxplugin.yaml

NOTE: Since the name of the CR in the sample file is sgxplugin-sample, we use that name in the following

Check the status of the SGXPlugin CR using kubectl get or kubectl describe:

kubectl get sgxplugin sgxplugin-sample
kubectl describe sgxplugin sgxplugin-sample

In the next section, we explain all the information you see in the output of those two commands.

NOTE: You can also check the log of the operator to follow the reconciliation process of the creation of the SGXPlugin CR:

kubectl logs -n $OPERATOR_NAMESPACE $OPERATOR_POD_NAME

Metrics, Alerts, and Conditions

As soon as there is a problem with the execution of any of the services, this must be communicated to the responsible party as soon as possible. It must not go unnoticed. This communication is most commonly done by using a metrics provider like Prometheus or by updating a visible state of the corresponding resource appropriately. In the case of the Kubernetes custom resources (CR) for the SCONE services, each CR can have one of the following states: HEALTHY, PENDING, and UNHEALTHY:

State Explanation
HEALTHY The resource has been successfully reconciled by its controller, and no problems were detected.
PENDING The resource is still in the process of reconciling or is waiting for something.
UNHEALTHY An alert or a problem that is not expected to go away on its own or prevents proper functionality has been detected.

The display of the UNHEALTHY state alerts the human operator, or monitoring services, of the fact that further action is required.

To further help diagnose a potential problem, the Status field of the CRs is also populated with so-called Conditions, which describe the state of service-specific conditions in which the CR can be. Conditions are used in the status fields of many Kubernetes objects, e.g. Pods, and most notably they consist of a Type and a Status. The type describes the condition (e.g., UpToDate), and the possible values of the status of a condition are: True, False, and Unknown. Conditions are displayed when performing a kubectl describe on a resource. We describe the conditions we provide in more detail below.

Apart from the state and conditions of a SCONE custom resource, we also provide metrics specific to the service of the corresponding CR.

CAS Specific Metrics

Status Field Description Type Valid Values Invalid Values
version Actual CAS version observed in the CAS pod string

LAS Specific Metrics and Alerts

This section describes the metrics available for the SCONE LAS service.

Status Field Description Type Valid Values Invalid Values
desiredNumberScheduled Desired number of LAS pods for the cluster integer Number of nodes selected by spec.nodeSelector
nodes Details about each cluster node that supports local attestation map[string]Object
numberAvailable Number of nodes that have a LAS pod in Ready state integer Number of nodes selected by spec.nodeSelector -1, ...,(status.desiredNumberScheduled-1)
version Actual LAS version observed in the LAS pods string

Alerts

As the name suggests, an alert indicates to a human being or a monitoring system that something is not working.

Status Field Description Desired value Alerting values
lasReplicaAlert Number of nodes on which LAS is not running, although it was expected to 0 >0
state State of LAS HEALTHY UNHEALTHY

An alerting value of any of the above alerts will cause the LAS CR to go into the UNHEALTHY state. This status.state field is also displayed in the output of the kubectl get las command.

SGXPlugin Specific Metrics and Alerts

This section describes the metrics, conditions, and alerts available for the SCONE SGX Plugin service.

Metrics

Status Field Description Type Valid Values Invalid Values
availableSGXDevices Number of SGX devices that are allocatable but not used integer >0 0, -1
availableSGXDevicesPerNode Number of SGX devices that are allocatable but not used (per node) map[string]integer >0 0, -1
capacitySGXDevices Total number of SGX devices that can be allocated by pods integer >0 0, -1
capacitySGXDevicesPerNode Total number of SGX devices that pods can allocate (per node) map[string]integer >0 0, -1
usedSGXDevices Number of pods that got access to an SGX device integer >=0 -1
usedSGXDevicesPerNode Number of pods that got access to an SGX device (per node) map[string]integer >=0 -1
desiredReplicas Number of desired replicas integer Number of nodes selected by spec.nodeSelector -1
readyReplicas Number of available replicas integer Number of nodes selected by spec.nodeSelector -1, ..., (status.desiredReplicas-1)

The metrics status.desiredReplicas and status.readyReplicas fields are also displayed in the output of the kubectl get sgxplugin command.

Alerts

As the name suggests, an alert indicates to a human being or a monitoring system that something is not working.

Status Field Description Desired value Alerting values
sgxDeviceAlert Number of nodes (on which the SGX Plugin is running) which has no available devices in one or more device category 0 >0, -1
sgxReplicaAlert Number of nodes on which the SGX Plugin is not running, although it was expected to 0 >0, -1
state State of the SGX Plugin HEALTHY UNHEALTHY

An alerting value of any of the above alerts will cause the SGXPlugin CR to go into the UNHEALTHY state. This status.state field is also displayed in the output of the kubectl get sgxplugin command.

Conditions

All Scone custom resources which are installed by helm, i.e., for example, the SGXPlugin and the CAS CRs, also have conditions in their status field. Each condition describes the condition in which a Kubernetes resource currently is. For example, if a resource is running as expected and no problems have been detected, its IsHealty condition would have the value True. On the other hand, if the object currently, for example, is having problems being deployed, its IsHealthy condition would have the value False. If the controller, for some reason, could not retrieve the object via the Kubernetes API, it would have the value Unknown.

When creating a CR of a service, other Kubernetes objects (e.g., ConfigMaps, Deployments, ...) will also be created. We make sure the CR owns each of these objects, i.e., we set their metadata.ownerReferences to contain the CR, and we call these objects the owned objects or OOs of the CR. CRs and owned objects are described by different sets of condition types, namely CR conditions and OO conditions, respectively. If a CR owns many OOs, there will be many conditions of the same OO condition type (e.g., OwnedObjectHealthy), namely one per OO, whereas there will always only be one condition of each CR condition type. To keep the OO conditions from the different OOs apart, we prefix their condition types with a unique description of the OO when displaying them in the status field of the CR.

The following is a table containing all different types of conditions, their meaning, and which state their value will force the CR (i.e., HEALTHY, PENDING, or UNHEALTHY). As soon as any of the conditions has the status Unknown, the status.state of the CR becomes PENDING, and only when all the IsHealthy and OwnedObjectHealthy conditions of the CR and OOs have a status of True, does the status.state of the CR become HEALTHY.

Described Object Condition Type Value Explanation Resulting CR State Resulting Action
(unless higher priority
action can be taken)
Action Priority
CR IsHealthy True The CR is healthy
False The CR is not healthy UNHEALTHY
Unknown Could not be determined PENDING
DeletionPresent True The CR is being deleted PENDING uninstall release 1
remove finalizer from CR 2
False The CR has not been deleted
Unknown Could not be determined PENDING
FinalizerPresent True The CR has the CR controller as a finalizer
False The CR does not have the CR controller as a finalizer PENDING set controller as finalizer of the CR 3
Unknown Could not be determined PENDING
ReleaseExists True The helm release installing the OOs of the CR exists
False The helm release installing the OOs of the CR could not be found PENDING install the release 6
Unknown The command retrieving the release returned an error of a
different kind than a not-found error
PENDING
ReleaseSteady True The status of the release installing the OOs of the CR
indicates that it is not about to change on its own
False The status of the release installing the OOs of the CR
indicates that it is in transition
PENDING wait 7
Unknown Could not be determined PENDING
ReleaseFailed True The helm release installing the OOs of the CR is in Failed state PENDING uninstall the release 8
False The helm release installing the OOs of the CR is not in Failed state
Unknown Could not be determined PENDING
ReleaseDeployed True The helm release installing the OOs of the CR is in Deployed state
False The helm release installing the OOs of the CR is not in Deployed state PENDING upgrade release 9
Unknown Could not be determined PENDING
AtLeastOneOwnedObjectPresent True At least one OO of the CR exists
False No OO of the CR was found PENDING
Unknown Could not be determined due to failure retrieving the OOs PENDING
AllOwnedObjectsPresent True All OOs of the CR exist
False At least one OO of the CR was not found PENDING
Unknown Could not be determined due to failure retrieving the OOs PENDING
OO OwnedObjectHealthy True The OO is healthy and matches the spec of the CR
False The CR is not healthy UNHEALTHY
Unknown Could not be determined PENDING
OwnedObjectPresent True The OO exists
False The OO does not exist PENDING
Unknown The command retrieving the oo returned an error of a
different kind than a not-found error
PENDING
OwnedObjectDeletionPresent True The OO has been deleted PENDING
False The OO has not been deleted
Unknown Could not be determined PENDING
OwnedObjectValidOwners True The OO has zero or one owner
False The OO has more than one owner PENDING remove all owners from the OO 4
Unknown Could not be determined PENDING
OwnsOwnedObject True The OO has the CR as an owner
False The OO does not have the CR as an owner PENDING set the CR as the owner of the OO 5
Unknown Could not be determined PENDING
SoleOwner True The CR is the only owner of the OO
False The OO does not have exactly one owner, which is the CR PENDING
Unknown Could not be determined PENDING
EqualSpecs True No attribute of the OO contradicts the spec of the CR
False At least one attribute of the OO contradicts the spec of the CR PENDING upgrade release 9
Unknown Could not be determined PENDING

The conditions are displayed in the status field of the SGXPlugin CR when performing the kubectl describe sgxplugin command.