Pekko in Kubernetes, application level solution for Digital Transformation

2024-06-13

kubernetes, pekko, udp, IIoT

Digital Reform

There was a vary circumstances from digital governance, industry, manufacture and medicine, and the booming of
new business models has stimulated enterprises to accelerate digital transformation for better company management
and customer service, which further promotes the upgrading and optimization of the Chinese digital economy.

Solution - Cloud Native

What’s internet essentially?

移动互联网络应该是
一个异步的由事件流驱动的巨大的状态机

There are many reason that we are migrating business service into Cloud Native. One of most important reason is that we
need to reduce our costs. Furthermore, as our productivity increases, we will have a vision on the IoT service.

Facets of design in IoT-a good product requires integrated thinking across all of these

the digital reform indicate that in the future, the connected things shall be self-diagnosis, reliable, lower-bandwidth,
low-carbon, more intelligent, and have a better version management.

Why Pekko if you have kubernetes?

Kubernetes is not the application layer.

As we know, K8S is covering the cluster orchestration, supporting auto-scaling and resilience on the nodes or the pods
level.

How the auto-scaling and self-healing will be handled within the application instances or node?

Kubernetes will not be able to help with that, another layer such as Pekko need to look after the application
communication, remoting, concurrency & parallel processing, responding to application failure, implementing resilience
design patterns such as bulk-heading and circuit-breakers on the application node level.

Non-function design in Kubernetes is stupid, such like GEO-Red, distributed-lock for business, transactions, entity-id
tracing, overload protect, data desensitization, Monitoring… etc.

In short, Kubernetes is looking after elasticity, auto-scaling and resilience outside the nodes while Pekko and other
reactive application tools are looking after those within the application layer and the nodes.

After all, Kubernetes will not be able to fix a poorly designed application that is prone to failure and lacking
resilience and elasticity.

We need a REAL reactive manifesto application, and below Lightbend diagrams
help with understanding the concept:

Cloud Native

Cloud Pekko

So, how to build stateful, resilient and responsive applications in the Cloud?

Sharding your X

Pekko roles as Kubernetes resource mapping

We use both Kubernetes Statefulset and Deployment. As we can use Statefulset for the seed nodes, mark as role seed,
and Deployment for scaling out worker nodes, marks role worker. In this way, we can maintaion stable and ordered DNS
names for the seed nodes, while retain the ability to start multiple Pekko nodes in parallel when scale with
Deployment.

1
2
3
4
{
  seeds  : Statefulset
  workers: Deployment
}

We will setup env from Kubernetes and deliver into application.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cluster {
  # This can also be defined as Java system properties when starting the JVM using the following syntax:
  #
  # -Dpekko.cluster.seed-nodes.0=pekko://Pekko@seed1:2551
  # -Dpekko.cluster.seed-nodes.1=pekko://Pekko@seed2:2551
  #
  seed-nodes = [
    "pekko://Pekko@127.0.0.1:2551",
    "pekko://Pekko@127.0.0.1:2552"]
  seed-nodes = ${?SEED_NODES}
  roles = ["seed", "sharding"]
  roles = ${?ROLES}
  sharding.role = "sharding"
  sharding.role = ${?SHARDING_ROLE}
}

Statefulset for entry -> seeds

especially, mTLS in istio. We will map it for traffic entries as a headless service. And the seed-ed service won’t
do anything. The resource definition looks like a regular service, but clusterIP set to None:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: v1
kind: Service
metadata:
 name: pekko-seed
spec:
  ports:
  - port: 2551
    protocol: TCP
    targetPort: 2551
  - port: 5684
    protocol: UDP
    targetPort: 5684
  selector:
    run: pekko-seed
  clusterIP: None

Every Pekko seed node instance in the Statefulset can be addressed via DNS as ${SEED_NODE_NAME}.pekko-seed.

Then, create Statefulset of seed nodes:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  labels:
    run: pekko-seed
  name: pekko-seed
spec:
  serviceName: pekko-seed
  replicas: 2
  selector:
    matchLabels:
      run: pekko-seed
  template:
    metadata:
      labels:
        run: pekko-seed
    spec:
      containers:
      - name: pekko-seed
        image: localhost:32000/pekko-jdk17-slim:latest
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: SEED_NODES.0
          value: pekko://Pekko@pekko-seed-0.pekko-seed:2551
        - name: SEED_NODES.1
          value: pekko://Pekko@pekko-seed-1.pekko-seed:2551
        - name: ROLES.0
          value: seed
        - name: SHARDING_ROLE
          value: sharding
        - name: HOST_NAME
          value: $(POD_NAME).kaxis-seed
        - name: HOST_PORT
          value: "4684"
        command: ["/bin/sh", "-c", "HOST_NAME=${POD_NAME}.pekko-seed java -jar /app/app.jar"]
        livenessProbe:
          tcpSocket:
            port: 2551
        ports:
        - containerPort: 2551
          protocol: TCP
        - containerPort: 5684
          protocol: UDP

First of all, every seed node will have a stable name (e.g. pekko-seed-0, pekko-ssed-1). This name is known as the
Kubernetes pod name. And then use it to construct the addressable DNS name, such as pekko-seed-0.pekko-seed.

Once deployment the Pekko seed nodes:

1
kubectl apply -f pekko-seeds.yaml

We will seed the seeds that were created sequentially:

1
2
3
4
kubectl get pods
NAME                          READY     STATUS    RESTARTS   AGE
pekko-seed-0                   1/1       Running   0          15h
pekko-seed-1                   1/1       Running   0          15h

As mentioned before, the seed nodes won’t do anything, below diagram show it just play the role for traffic transform,
cluster communication and cluster init.

This part is very-very important!! for vary business service, such like coap, diameter protocol based on udp that
Kubernetes doesn’t provide a full-stack support for loadbalancer.

Deployment for loadbancer -> workers

Next, deploy the worker nodes using Deployment. It use the Downward API to assign the Pod IP address as te Pekko node’s
hostname.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pekko-worker
spec:
  selector:
    matchLabels:
      app: pekko-worker
  replicas: 3
  template:
    metadata:
      labels:
        app: pekko-worker
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: pekko-worker
          image: localhost:32000/pekko-jdk17-slim:latest
          imagePullPolicy: Always
          resources:
            requests:
              memory: "1Gi"
              cpu: "250m"
            limits:
              memory: "2Gi"
              cpu: "500m"
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SEED_NODES.0
              value: pekko://Pekko@pekko-seed-0.pekko-seed:4684
            - name: SEED_NODES.1
              value: pekko://Pekko@pekko-seed-1.pekko-seed:4684
            - name: ROLES.0
              value: sharding
            - name: SHARDING_ROLE
              value: sharding
            - name: HOST_NAME
              value: "$(POD_NAME)"
            - name: HOST_PORT
              value: "4684"
          livenessProbe:
            httpGet:
              port: 6684
              path: health/alive
            initialDelaySeconds: 30
            periodSeconds: 20
          readinessProbe:
            httpGet:
              port: 6684
              path: health/ready
            initialDelaySeconds: 30
            periodSeconds: 20
          ports:
            - containerPort: 5684
              name: tcp-udp
            - containerPort: 4684
              name: http-gossip
      dnsPolicy: ClusterFirst
      imagePullSecrets:
        - name: regcred

Once deloyed, you can validate that all nodes are up and running:

1
2
3
4
5
6
7
8
$ kubectl get pods
NAME                           READY     STATUS    RESTARTS   AGE
pekko-seed-0                    1/1       Running   0          8s
pekko-seed-1                    1/1       Running   0          6s
pekko-worker-2263404214-8c266   1/1       Running   0          8s
pekko-worker-2263404214-9ws3k   1/1       Running   0          8s
pekko-worker-2263404214-f2tp3   1/1       Running   0          8s
pekko-worker-2263404214-lkvz3   1/1       Running   0          8s

You can validate that the nodes have joined the cluster by inspecting the logs of any of the Pekko nodes. For
example, to tail the first seed node’s log:

1
2
3
4
5
6
7
kubect logs -f pekko-seed-0
[INFO] [02/12/2017 15:36:53.568] [main] [pekko.remote.Remoting] Starting remoting
[INFO] [02/12/2017 15:36:53.707] [main] [pekko.remote.Remoting] Remoting started; listening on addresses :[pekko.tcp://Pekko@pekko-seed-0.akka-seed:2551]
...
[INFO] [02/12/2017 15:37:05.101] [ClusterSystem-pekko.actor.default-dispatcher-16] [pekko.cluster.Cluster(pekko://ClusterSystem)] Cluster Node [pekko.tcp://Pekko@pekko-seed-0.pekko-seed:2551] - Node [pekko.tcp://Pekko@pekko-seed-1.pekko-seed:2551] is JOINING, roles []
[INFO] [02/12/2017 15:37:06.854] [ClusterSystem-pekko.actor.default-dispatcher-16] [pekko://ClusterSystem/user/$a] Member is Up: Member(address = akka.tcp://Pekko@10.44.2.10:2551, status = Up)
...

Also, you can simply attach Kubernetes HPA:

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: pekko-worker
spec:
  maxReplicas: 3
  minReplicas: 1
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: pekko-worker
  targetCPUUtilizationPercentage: 15

Health check

Pekko already implementation cluster health check management:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# management
management {
  http.hostname = 127.0.0.1
  http.hostname = ${?HOST_NAME}
  http.bind-hostname = 0.0.0.0
  http.bind-hostname = ${?HOST_NAME}
  health-checks {
    readiness-path = "health/ready"
    liveness-path = "health/alive"
    readiness-checks {
      # Default health check for cluster.
      cluster-membership = "org.apache.pekko.management.cluster.javadsl.ClusterMembershipCheck"
    }
  }
}

Summary

Please check my own repo in GitHub, if you need more investigate.