KubeSerial
THIS IS STILL WORK IN PROGRESS
Work with serial devices in Kubernetes
THIS IS STILL WORK IN PROGRESS
KubeSerial is a set of Kubernetes controllers and resources that make working with serial devices in Kubernetes clusters easy. It decouples node that device is connected to from workload that's using it.
How it works
KubeSerial uses Device Monitors to monitor all cluster nodes for Devices. Once Device is detected, it will schedule Device Gateway to specific cluster node. Device Gateway exposes this device as TCP server. Once Device Gateway is available, it can be used by Manager. There are 2 supported modes for Managers:
- Use predefined Manager that will be scheduled when Device is available
- Use annotation to inject device to any workload
Quick Start
- Requirements
- Install with Helm
- Get device attributes
- Update helm values file
- Update your helm release with new values
- Validate that everything is working
Requirements
- k8s cluster
- CertManager installed and Cert Issuer configured
Install with Helm
Add help repo
First you'll need to add helm repository that stores KubeSerial charts
> helm repo add baraniewski https://baraniewski.com/charts/
Install CRDs
Due to way in which helm handles CRDs, they are managed using separate chart.
> helm upgrade --install kubeserial-crds baraniewski/kubeserial-crds
Create minimal values file
In order to make webhook work, you'll need to specify which Cert Issuer should be used for SSL cert used by webhook. To do this, create values file with following structure (change values depending on your setup):
certManagerIssuer:
name: selfsigned-issuer
kind: ClusterIssuer
Set proper name and kind of your Issuer or ClusterIssuer
Install Controller
> helm upgrade --install kubeserial baraniewski/kubeserial
Get device attributes
To find out values of idVendor and idProduct for your device, connect it to your computer, locate where it is (let's say /dev/ttyUSB0) and run:
> udevadm info -q all -n /dev/ttyUSB0 --attribute-walk
Look for them from the top.
Update helm values file
Now you're ready to create configuration for your devices. Here you can find example of 2 devices - one is using predefined manager, one doesn't. To learn about the difference, please refer to manager configuration docs
Add this config to your values file:
kubeserial:
serialDevices:
- idProduct: "6001"
idVendor: "0403"
manager: octoprint
name: ender3
- idProduct: ea60
idVendor: 10c4
name: sonoff-zigbee
Update your helm release with new values
> helm upgrade kubeserial baraniewski/kubeserial -f my-values.yaml
Validate that everything is working
You should see 3 workloads in your cluster - controler manager, device injector webhook and device monitor:
$ ➜ kubectl get pods
NAME READY STATUS RESTARTS AGE
kubeserial-7d58555d4c-97nqk 1/1 Running 0 5m
kubeserial-device-injector-cc7696b59-mfdn5 1/1 Running 0 5m
kubeserial-monitor-rjs82 2/2 Running 0 5m
Number of kubeserial-monitor pods should match number of your cluster nodes.
You should also see your devices:
$ ➜ kubectl get serialdevice
NAME READY AVAILABLE NODE
ender3 True False
sonoff-zigbee True False
You're all set with basic configuration of KubeSerial. Please read through rest of docs to configure it to your needs.
Configuration
KubeSerial
apiVersion: app.kubeserial.com/v1alpha1
kind: KubeSerial
metadata:
annotations:
...
labels:
...
name: kubeserial
namespace: kubeserial
spec:
ingress:
enabled: false
serialDevices:
- idProduct: "6001"
idVendor: "0403"
manager: octoprint
name: ender3
- idProduct: ea60
idVendor: 10c4
name: sonoff-zigbee
Device
Spec
apiVersion: app.kubeserial.com/v1alpha1
kind: SerialDevice
metadata:
annotations:
...
labels:
...
name: ender3
spec:
idProduct: "6001"
idVendor: "0403"
manager: octoprint
name: ender3
Status conditions
apiVersion: app.kubeserial.com/v1alpha1
kind: SerialDevice
...
status:
conditions:
- lastHeartbeatTime: "2022-06-05T23:49:02Z"
lastTransitionTime: "2022-06-05T23:49:02Z"
message: ""
reason: NotValidated
status: "False"
type: Available
- lastHeartbeatTime: "2022-06-05T23:49:02Z"
lastTransitionTime: "2022-06-05T23:49:02Z"
message: ""
reason: AllChecksPassed
status: "True"
type: Ready
- lastHeartbeatTime: "2022-06-05T23:49:02Z"
lastTransitionTime: "2022-06-05T23:49:02Z"
message: ""
reason: NotValidated
status: "False"
type: Free
Managers
In order to learn more about concept of Manager take a look at Manager docs
Manager configuration is not required, although it's the easiest way to work with serial devices using KubeSerial.
There are 2 ways to work with managers. You can use them both at the same time for different devices, but not for the same one.
Manager scheduled by KubeSerial
In this approach, you'll need to create Manager type object which will hold spec of manager software you want to run and bind it with SerialDevice. Then, when KubeSerial detects that device is connected, it will schedule management software for you and create link to the device inside. Once device is disconnected, everything is cleaned up. Learn how to configure it by reading the docs
Manager scheduled externaly
In this approach, you add annotation to your Pod and KubeSerial mutating webhook will update pod spec when it's created to inject connection to device. Learn how to configure it by reading the docs
Manager scheduled by KubeSerial
apiVersion: app.kubeserial.com/v1alpha1
kind: Manager
metadata:
annotations:
meta.helm.sh/release-name: kubeserial
meta.helm.sh/release-namespace: kubeserial
creationTimestamp: "2022-06-05T23:48:43Z"
generation: 1
labels:
app.kubernetes.io/instance: kubeserial
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: kubeserial
app.kubernetes.io/version: 0.0.1-8c68648
helm.sh/chart: kubeserial-0.0.1-8c68648
name: octoprint
resourceVersion: "43158"
uid: c908bad7-8716-4f50-aa3f-aeb91bebe71f
spec:
config: |
accessControl:
enabled: false
plugins:
announcements:
_config_version: 1
channels:
_blog:
read_until: 1573642500
_important:
read_until: 1521111600
_octopi:
read_until: 1573722900
_plugins:
read_until: 1573862400
_releases:
read_until: 1574699400
discovery:
upnpUuid: ef35acc7-a859-4947-980d-d5edb10508e4
softwareupdate:
_config_version: 6
tracking:
enabled: false
deviceProfiles:
default: _default
serial:
additionalPorts:
- /dev/devices/ender3
autoconnect: true
baudrate: 0
port: /dev/device
server:
firstRun: false
onlineCheck:
enabled: true
pluginBlacklist:
enabled: false
seenWizards:
corewizard: 3
cura: null
tracking: null`
configPath: /data/config.yaml
image:
repository: janekbaraniewski/octoprint
tag: 1.3.10
runCmd: mkdir /root/.octoprint && cp /data/config.yaml /root/.octoprint/config.yaml
&& /OctoPrint-1.3.10/run --iknowwhatimdoing --port 80
Manager scheduled externaly
Components
Controller
Manages operator components by observing state of each of the devices.
Controller Loops
KubeSerial Controller
SerialDevice Controller
ManagerScheduleRequest Controller
graph TD;
A-->B;
A-->C;
B-->D;
C-->D;
SerialDeviceMonitor
Monitors cluster nodes waiting for specified serial devices to be connected and updates their state.
How it works
SerialDevice Gateway
Exposed specific device in cluster network over TCP.
Manager
Creates deployment with management software, mounts your device over the network and gives you access through ingress rule.
Development
Intro
Most things you'd need for local development are covered in Makefile. Just run make help to see available commands:
➜ make help
Usage:
make <target>
General
help Display this help.
Development
fmt Run go fmt against code.
vet Run go vet against code.
test Run tests.
test-fswatch Use fswatch to watch source files and run tests on chamnge
Run
run Run codegen and start controller from your host.
Docker
kubeserial-docker-local Build image for local development, tag local, supports only builder platform
kubeserial-docker-all Build and push image for all target platforms
device-monitor-docker-local Build image for local development, tag local, supports only builder platform
device-monitor-docker-all Build and push image for all target platforms
injector-webhook-docker-local Build image for local development, tag local, supports only builder platform
injector-webhook-docker-all Build and push image for all target platforms
Helm
update-kubeserial-chart-version Update version used in chart. Requires VERSION var to be set
update-kubeserial-crds-chart-version Update version used in chart. Requires VERSION var to be set
helm-lint Run chart-testing to lint kubeserial chart.
Kind
Minikube
minikube Start local cluster, build image and deploy
minikube-start Start minikube cluster
minikube-set-context Set context to use minikube cluster
minikube-deploy Deploy the app to local minikube
Deployment
uninstall Uninstall release.
deploy-dev Install dev release in current context/namespace.
Docs
docs-deps Install mdbook (requires rust and cargo) + plugins
docs-serve Build docs, start server and open in browser
Build
kubeserial Build manager binary.
device-monitor Build device monitor binary
injector-webhook Build sidecar injector webhook binary binary
all Run codegen and build all components.
Running tests
After any change run
$ make test
to run tests suite
You can also find it helpful to just run tests every time there is a change in project source files, for this run
$ make test-fswatch
which will run test target every time it detects change using fswatch (fswatch must be installed)
Building images
There are sets of 2 targets for each image that is a part of this project:
*-docker-local
which builds docker image for local development
and
*-docker-all
which builds docker image for all target architectures.
For local development you'll only need to build local images
Building images localy
Docker buildx builder with support for platforms listed in TARGET_PLATFORMS is required
If you want to build all images using your local Docker, run
$ make docker-local
This will execute all *-docker-local targets and build images using your local Docker.
Running minikube
TODO