Custom Resource Definitions

Custom Resource Definitions

After we have deployed the SCONE operator, we are ready to create SCONE custom resources in the cluster. Before we can deploy them, however, we need to configure them.

Kubernetes custom resources are extensions to the Kubernetes API. They are specified with the help of custom resource definitions or CRDs. The SCONE CRDs are deployed at the time the SCONE operator is installed. You can examine them by executing the following commands:

kubectl describe crd sgxplugins
kubectl describe crd las
kubectl describe crd cas
kubectl describe crd signedpolicies
kubectl describe crd encryptedpolicies

The group, version, and kind of custom resources resulting from the CRDs are summarized in the following table:

Service	Group	Version	Kind	Short Name
SCONE CAS	services.scone.cloud	v1beta1	CAS
SCONE LAS	base.scone.cloud	v1beta1	LAS
SCONE SGX Plugin	base.scone.cloud	v1beta1	SGXPlugin	sgx
SCONE Signed Policies	cas.scone.cloud	v1beta1	SignedPolicy	spol
SCONE Encrypted Policies	cas.scone.cloud	v1beta1	EncryptedPolicy	epol

Using the following commands, you can list all existing resources of each kind:

kubectl get sgxplugins
kubectl get las
kubectl get cas
kubectl get signedpolicies
kubectl get encryptedpolicies

Since we have not yet created any custom resources, the output of all three commands will be empty. Creating one of these resources will start the corresponding SCONE service in your cluster. Before we create them, however, we want to configure them according to our needs. There is a sample manifest for the custom resource of each service at https://github.com/scontain/operator-samples that you can use as a starting point:

Service	Sample Manifest
SCONE CAS	services_v1beta1_cas.yaml
SCONE LAS	base_v1beta1_las.yaml
SCONE SGX Plugin	base_v1beta1_sgxplugin.yaml
SCONE Signed Policies	cas_v1beta1_signedpolicy.yaml
SCONE Encrypted Policies	cas_v1beta1_encryptedpolicy.yaml

Configuring SCONE Services (CAS, LAS, and SGXPlugin)

Common Configuration

The following are configuration parameters that are common to all SCONE services (CAS, LAS, and SGXPlugin)

Name	Description	Type	Default Value
autoUpdate	Indicates whether the docker image of the service should automatically be updated to the newest version. This triggers rolling updates of the service pods and assumes that the image has semantic versioning in place. NOTE: autoUpdate is currently disabled in CAS	boolean	false
image	OCI image for the service	string	most recent SCONE image
imagePullPolicy	Indicates when the OCI image should be pulled from the registry	Always, Never, IfNotPresent	Always
imagePullSecrets	ImagePullSecrets for the OCI image. NOTE: The current implementation considers only the first valid secret, even if multiple are supported	Object
resources	Resource requirements of this SCONE SGX Plugin	Object	{}
nodeSelector	Selects the nodes where the service should run. The NodeSelector must match a node's labels for a pod to be scheduled on that node	map[string]string	{}

Configuration of the CAS Custom Resource

Name	Description	Type	Default Value
service.type	Service type for the CAS pod. Supported values are ClusterIP, LoadBalancer or NodePort	v1.ServiceType	ClusterIP
service.clientPort	Specify a port for the CAS client port	integer	8081
service.enclavePort	Specify a port for the CAS enclave port	integer	18765
service.loadBalancerIP	Specify an IP address for your Service if service.type=LoadBalancer and your cloud provider supports it	string	-
service.nodePorts.clientPort	Specify a nodePort for the CAS client port if service.type=NodePort	integer	-
service.nodePorts.enclavePort	Specify a nodePort for the CAS enclave port if service.type=NodePort	integer	-
service.annotations	Specify annotations for the CAS service	map[string]string	{}
persistence.enabled	If set to true, create a PersistentVolumeClaim alongside CAS for persistence	boolean	false
persistence.mountPath	Where to mount CAS persistent volume	string	/etc/cas/db
persistence.storageClass	Define a storageClassName for the provisioned PersistentVolumeClaim. If unset or set to nil (default), use default storageClassName. If set to "-", disable dynamic provisioning	string	nil
persistence.accessModes	PersistentVolume access modes	[]v1.PersistentVolumeAccessMode	["ReadWriteOnce"]
persistence.size	PersistentVolume size	resource.Quantity	1Gi

Configuration of the LAS Custom Resource

Name	Description	Type	Default value
hostPort	Containers must forward attestation requests to LAS instances collocated in the same node, but this requirement is not achievable through a standard Kubernetes Service abstraction. For this reason, we rely on hostPort to expose LAS and assume that the pods will use the status.hostIP (provided by the Kubernetes API) as SCONE_LAS_ADDR	integer	18766
namespace	Kubernetes namespace in which the SCONE LAS service should be running	string	namespace of SCONE Operator

Configuration of the SGXPlugin Custom Resource

As we can see by executing kubectl explain sgxplugins.spec, we can configure the SCONE SGXPlugin custom resource using the following parameters:

Name	Description	Type	Default value
capacitySGXPods	Maximum number of SGX devices to use per Kubernetes node	1<=integer<=10000	Maximum number of pods per node
image	Docker image with the SCONE SGX Plugin	string	registry.scontain.com/ scone.cloud/ sgx-plugin
namespace	Kubernetes namespace in which the SCONE SGX Plugin service should be running	string	namespace of SCONE Operator

Configuring SCONE Policies (SignedPolicy and EncryptedPolicy)

The custom resources for SCONE Policies have a model that is very similar to the one of the CAS REST API. In this sense, they are not "configurable" but rather an immutable resource that reflects the output of the sign and encrypts SCONE CLI operations. Updating individual fields would lead to an error because it would break integrity.

This means that for every create or update operation on a policy custom resource; the policy owner must re-do the sign or encrypt operations on the unencrypted base policy and then submit the result of the operation to the cluster by creating the corresponding CR. The creation of the CR registers the policy in CAS.

Nonetheless, the custom resources for policies support the following fields.

Fields for the SignedPolicy Custom Resource

Name	Description	Type
casAddress	The target CAS address	string
policy	The policy contents	string
signatures	A list of signatures, where each signature is an object comprised of signer and signature fields	[]Object

Fields for the EncryptedPolicy Custom Resource

Name	Description	Type
casAddress	The target CAS address	string
policy	The policy contents	string
encryptionKey	The target CAS public encryption key	string

Installing the Services

To install any SCONE services you need to deploy its manifest file. If you configured it to meet your requirements in the previous step, you already have a manifest. If you did not, you could use the sample manifests provided for each service in the above table. To install a service with its manifest at $MANIFEST_FILE, simply execute the following command:

kubectl apply -f $MANIFEST_FILE

Since the SCONE SGX Plugin service is required for the other ones to run, let's use it as an example and install it:

kubectl apply -f ./config/samples/base_v1beta1_sgxplugin.yaml

NOTE: Since the name of the CR in the sample file is sgxplugin-sample, we use that name in the following

Check the status of the SGXPlugin CR using kubectl get or kubectl describe:

kubectl get sgxplugin sgxplugin-sample
kubectl describe sgxplugin sgxplugin-sample

In the next section, we explain all the information you see in the output of those two commands.

NOTE: You can also check the log of the operator to follow the reconciliation process of the creation of the SGXPlugin CR:

kubectl logs -n $OPERATOR_NAMESPACE $OPERATOR_POD_NAME

Metrics, Alerts, and Conditions

As soon as there is a problem with the execution of any of the services, this must be communicated to the responsible party as soon as possible. It must not go unnoticed. This communication is most commonly done by using a metrics provider like Prometheus or by updating a visible state of the corresponding resource appropriately. In the case of the Kubernetes custom resources (CR) for the SCONE services, each CR can have one of the following states: HEALTHY, PENDING, and UNHEALTHY:

State	Explanation
HEALTHY	The resource has been successfully reconciled by its controller, and no problems were detected.
PENDING	The resource is still in the process of reconciling or is waiting for something.
UNHEALTHY	An alert or a problem that is not expected to go away on its own or prevents proper functionality has been detected.

The display of the UNHEALTHY state alerts the human operator, or monitoring services, of the fact that further action is required.

To further help diagnose a potential problem, the Status field of the CRs is also populated with so-called Conditions, which describe the state of service-specific conditions in which the CR can be. Conditions are used in the status fields of many Kubernetes objects, e.g. Pods, and most notably they consist of a Type and a Status. The type describes the condition (e.g., UpToDate), and the possible values of the status of a condition are: True, False, and Unknown. Conditions are displayed when performing a kubectl describe on a resource. We describe the conditions we provide in more detail below.

Apart from the state and conditions of a SCONE custom resource, we also provide metrics specific to the service of the corresponding CR.

CAS Specific Metrics

Status Field	Description	Type	Valid Values	Invalid Values
version	Actual CAS version observed in the CAS pod	string

LAS Specific Metrics and Alerts

This section describes the metrics available for the SCONE LAS service.

Status Field	Description	Type	Valid Values	Invalid Values
desiredNumberScheduled	Desired number of LAS pods for the cluster	integer	Number of nodes selected by spec.nodeSelector
nodes	Details about each cluster node that supports local attestation	map[string]Object
numberAvailable	Number of nodes that have a LAS pod in Ready state	integer	Number of nodes selected by spec.nodeSelector	-1, ...,(status.desiredNumberScheduled-1)
version	Actual LAS version observed in the LAS pods	string

Alerts

As the name suggests, an alert indicates to a human being or a monitoring system that something is not working.

Status Field	Description	Desired value	Alerting values
lasReplicaAlert	Number of nodes on which LAS is not running, although it was expected to	0	>0
state	State of LAS	HEALTHY	UNHEALTHY

An alerting value of any of the above alerts will cause the LAS CR to go into the UNHEALTHY state. This status.state field is also displayed in the output of the kubectl get las command.

SGXPlugin Specific Metrics and Alerts

This section describes the metrics, conditions, and alerts available for the SCONE SGX Plugin service.

Metrics

Status Field	Description	Type	Valid Values	Invalid Values
availableSGXDevices	Number of SGX devices that are allocatable but not used	integer	>0	0, -1
availableSGXDevicesPerNode	Number of SGX devices that are allocatable but not used (per node)	map[string]integer	>0	0, -1
capacitySGXDevices	Total number of SGX devices that can be allocated by pods	integer	>0	0, -1
capacitySGXDevicesPerNode	Total number of SGX devices that pods can allocate (per node)	map[string]integer	>0	0, -1
usedSGXDevices	Number of pods that got access to an SGX device	integer	>=0	-1
usedSGXDevicesPerNode	Number of pods that got access to an SGX device (per node)	map[string]integer	>=0	-1
desiredReplicas	Number of desired replicas	integer	Number of nodes selected by spec.nodeSelector	-1
readyReplicas	Number of available replicas	integer	Number of nodes selected by spec.nodeSelector	-1, ..., (status.desiredReplicas-1)

The metrics status.desiredReplicas and status.readyReplicas fields are also displayed in the output of the kubectl get sgxplugin command.

Alerts

As the name suggests, an alert indicates to a human being or a monitoring system that something is not working.

Status Field	Description	Desired value	Alerting values
sgxDeviceAlert	Number of nodes (on which the SGX Plugin is running) which has no available devices in one or more device category	0	>0, -1
sgxReplicaAlert	Number of nodes on which the SGX Plugin is not running, although it was expected to	0	>0, -1
state	State of the SGX Plugin	HEALTHY	UNHEALTHY

An alerting value of any of the above alerts will cause the SGXPlugin CR to go into the UNHEALTHY state. This status.state field is also displayed in the output of the kubectl get sgxplugin command.

Conditions

All Scone custom resources which are installed by helm, i.e., for example, the SGXPlugin and the CAS CRs, also have conditions in their status field. Each condition describes the condition in which a Kubernetes resource currently is. For example, if a resource is running as expected and no problems have been detected, its IsHealty condition would have the value True. On the other hand, if the object currently, for example, is having problems being deployed, its IsHealthy condition would have the value False. If the controller, for some reason, could not retrieve the object via the Kubernetes API, it would have the value Unknown.

When creating a CR of a service, other Kubernetes objects (e.g., ConfigMaps, Deployments, ...) will also be created. We make sure the CR owns each of these objects, i.e., we set their metadata.ownerReferences to contain the CR, and we call these objects the owned objects or OOs of the CR. CRs and owned objects are described by different sets of condition types, namely CR conditions and OO conditions, respectively. If a CR owns many OOs, there will be many conditions of the same OO condition type (e.g., OwnedObjectHealthy), namely one per OO, whereas there will always only be one condition of each CR condition type. To keep the OO conditions from the different OOs apart, we prefix their condition types with a unique description of the OO when displaying them in the status field of the CR.

The following is a table containing all different types of conditions, their meaning, and which state their value will force the CR (i.e., HEALTHY, PENDING, or UNHEALTHY). As soon as any of the conditions has the status Unknown, the status.state of the CR becomes PENDING, and only when all the IsHealthy and OwnedObjectHealthy conditions of the CR and OOs have a status of True, does the status.state of the CR become HEALTHY.

Described Object	Condition Type	Value	Explanation	Resulting CR State	Resulting Action (unless higher priority action can be taken)	Action Priority
CR	IsHealthy	True	The CR is healthy
		False	The CR is not healthy	UNHEALTHY
		Unknown	Could not be determined	PENDING
	DeletionPresent	True	The CR is being deleted	PENDING	uninstall release	1
					remove finalizer from CR	2
		False	The CR has not been deleted
		Unknown	Could not be determined	PENDING
	FinalizerPresent	True	The CR has the CR controller as a finalizer
		False	The CR does not have the CR controller as a finalizer	PENDING	set controller as finalizer of the CR	3
		Unknown	Could not be determined	PENDING
	ReleaseExists	True	The helm release installing the OOs of the CR exists
		False	The helm release installing the OOs of the CR could not be found	PENDING	install the release	6
		Unknown	The command retrieving the release returned an error of a different kind than a not-found error	PENDING
	ReleaseSteady	True	The status of the release installing the OOs of the CR indicates that it is not about to change on its own
		False	The status of the release installing the OOs of the CR indicates that it is in transition	PENDING	wait	7
		Unknown	Could not be determined	PENDING
	ReleaseFailed	True	The helm release installing the OOs of the CR is in Failed state	PENDING	uninstall the release	8
		False	The helm release installing the OOs of the CR is not in Failed state
		Unknown	Could not be determined	PENDING
	ReleaseDeployed	True	The helm release installing the OOs of the CR is in Deployed state
		False	The helm release installing the OOs of the CR is not in Deployed state	PENDING	upgrade release	9
		Unknown	Could not be determined	PENDING
	AtLeastOneOwnedObjectPresent	True	At least one OO of the CR exists
		False	No OO of the CR was found	PENDING
		Unknown	Could not be determined due to failure retrieving the OOs	PENDING
	AllOwnedObjectsPresent	True	All OOs of the CR exist
		False	At least one OO of the CR was not found	PENDING
		Unknown	Could not be determined due to failure retrieving the OOs	PENDING
OO	OwnedObjectHealthy	True	The OO is healthy and matches the spec of the CR
		False	The CR is not healthy	UNHEALTHY
		Unknown	Could not be determined	PENDING
	OwnedObjectPresent	True	The OO exists
		False	The OO does not exist	PENDING
		Unknown	The command retrieving the oo returned an error of a different kind than a not-found error	PENDING
	OwnedObjectDeletionPresent	True	The OO has been deleted	PENDING
		False	The OO has not been deleted
		Unknown	Could not be determined	PENDING
	OwnedObjectValidOwners	True	The OO has zero or one owner
		False	The OO has more than one owner	PENDING	remove all owners from the OO	4
		Unknown	Could not be determined	PENDING
	OwnsOwnedObject	True	The OO has the CR as an owner
		False	The OO does not have the CR as an owner	PENDING	set the CR as the owner of the OO	5
		Unknown	Could not be determined	PENDING
	SoleOwner	True	The CR is the only owner of the OO
		False	The OO does not have exactly one owner, which is the CR	PENDING
		Unknown	Could not be determined	PENDING
	EqualSpecs	True	No attribute of the OO contradicts the spec of the CR
		False	At least one attribute of the OO contradicts the spec of the CR	PENDING	upgrade release	9
		Unknown	Could not be determined	PENDING

The conditions are displayed in the status field of the SGXPlugin CR when performing the kubectl describe sgxplugin command.