Extremely fast Kubernetes jobs orchestration based on Lambdas
Command Palette
Search for a command to run...
Comments
Join the discussionNo comments yet. Be the first to comment.
Search for a command to run...
No comments yet. Be the first to comment.
The combination of Kubernetes and Helm is widely used for service orchestration, provisioning, and release management. The approach of nodes and pods is flexible and efficient for scheduling predictable workflows.
However, the downside of this flexibility is that it involves two layers of orchestration: one for virtual machines (nodes) and the other for containers (pods). This can limit the scalability of bootstrap time, since the controller needs to bring up the node first and then schedule pods.
When cold-start time is crucial, and jobs are spawned in random quantities, making it difficult to optimize node size, using vendor-locked solutions like AWS Lambda directly may seem like an option. However, this approach makes release management more complicated as deployments now need to be done in two different spots, which can lead to potential issues.
To address this need, Kubernetes provides a feature called "Jobs" to abstract this kind of activity. While Kubernetes doesn't offer extensive tools to control the run flow of these jobs, it allows controllers to extend their functionality to address this issue.
The way Kubernetes abstracts this kind of activity is "Jobs". It doesn't provide a lot of tools to control their run flow, but it allows controllers to extend it to address the issue
Kueue is an example of a feature-rich controller that expands Kubernetes with resources, allowing queues, concurrency limits, and resource limitations per abstract group, etc.
Given a Kubernetes cluster is well-setup, horizontal autoscaling is most likely used. And thus we end up with the following scaling process:
A service is creating Job resource in Kubernetes using API using certain meta fields Kueue operates with. Job is not created started.
Kueue picks jobs according to concurrency rules and tries to schedule the workflow as pod(s).
Pods don't have enough resources to start up, so they request horizontal auto-scaling, which starts a Kubernetes node
When the node is up and in sync, pods start initialization and then eventually run
In a certain timeout during the absence of jobs, the node is decommissioned
Now, from a theoretical and design standpoint, this solution is very general and flexible. It also leverages an impressive orchestration engine used for other workflows besides jobs. However, due to its abstract and multi-layered nature, it performs sluggishly. At point 1, when you enqueue a job, it takes at least a minute just to start the container. In certain edge cases, it can easily take several minutes. Oh no!
Before we discuss the ultimate solution, it's important to introduce Fargate. Fargate is a unique alternative to a compute engine that was specifically designed to address the current issue. It seamlessly integrates with Kubernetes and allows you to designate Fargate as the executor of a pod using labels. Unlike classic orchestration methods, Fargate aims to minimize dependent entities. Instead of having scaling nodes running pods, Fargate operates by providing something akin to a dedicated node for each pod.
So, where does this leave us?
A service creates a Job resource in Kubernetes using specific meta fields that Kueue operates with, but the Job is not initially started
Kueue selects jobs based on concurrency rules and attempts to schedule the workflow as pod(s).
The pod is created directly on its own virtual node provided by Fargate.
When the job is finished, the Fargate node is utilized along with the Pod.
By eliminating an unnecessary step and streamlining the orchestration process, we have a specialized service dedicated to providing on-demand resources. Does it work quickly? Not quite. While it's an improvement over the classic scheduler, it's also noticeably more expensive and still takes approximately 60 seconds to get started.
Lambdas are designed for rapid scaling. Here are the stats. They are incompatible with Kubernetes orchestration, though.
Introducing EKS ACK, a set of controllers created to map AWS services from vendors into Kubernetes Resource Objects. This enables the management of resources via Helm releases. Let's break down what this means for us in practical terms:
ACK continually tries to synchronize your Kubernetes Resource definition with the actual cloud service entries. This encompasses the complete cycle of creating, modifying, and removing resources.
Helm manages resources associated with a microservice within the context of versioned releases. As a result, within a release, we can define ACK Resources, and any changes to these resources will prompt the controller to ensure that the cloud environment reflects the changes.
Kubernetes Secrets are integrated into the design and can only be accessed via the Kubernetes API during runtime. Since Lambda does not have a concept of secrets, this is the approach used.
At the core of a Helm release is a snapshot of the application prebuilt as a container image. This same image can be utilized as a Lambda runtime by adjusting the ENTRYPOINT and CMD.
We are going to provision SQS Queue, Lambda, and binding between the two using ACK and Helm integration:
Note that we create two queues to implement DLQ (dead queue).
apiVersion: sqs.services.k8s.aws/v1alpha1
kind: Queue
metadata:
name: queue
namespace: service
annotations:
services.k8s.aws/region: {{ .Values.region }}
spec:
visibilityTimeout: "180"
queueName: Service
redrivePolicy: |
{
"deadLetterTargetArn": "arn:aws:sqs:{{ .Values.region }}:{{ .Values.account }}:ServiceDead",
"maxReceiveCount": "5"
}
policy: |
{
"Version": "2012-10-17",
"Id": "Service",
"Statement": [
{
"Sid": "__sns",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": [
"sqs:SendMessage",
"sqs:SendMessageBatch"
],
"Resource": "*"
},
{
"Sid": "__service",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::{{ .Values.account }}:user/service"
},
"Action": "sqs:SendMessage",
"Resource": "*"
}
]
}
---
apiVersion: sqs.services.k8s.aws/v1alpha1
kind: Queue
metadata:
name: queue-dead
namespace: service
annotations:
services.k8s.aws/region: {{ .Values.region }}
spec:
queueName: ServiceDead
policy: |
{
"Version": "2012-10-17",
"Id": "Service",
"Statement": [
{
"Sid": "SNS",
"Effect": "Allow",
"Principal": {
"Service": "sns.amazonaws.com"
},
"Action": [
"sqs:SendMessage",
"sqs:SendMessageBatch"
],
"Resource": "*"
},
{
"Sid": "__service",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::{{ .Values.account }}:user/service"
},
"Action": "sqs:SendMessage",
"Resource": "*"
}
]
}
apiVersion: lambda.services.k8s.aws/v1alpha1
kind: Function
metadata:
name: service
namespace: service
annotations:
services.k8s.aws/region: {{ .Values.region }}
spec:
name: Service
packageType: Image
code:
imageURI: {{ .Values.image }}
role: arn:aws:iam::{{ .Values.account }}:role/{{ lower .Values.stage }}.ack-lambda-exec
description: Service The Great
timeout: 180
memorySize: {{ .Values.memory }}
imageConfig:
command:
- app/lambdas/service.handler
entryPoint:
- /lambda_entry.sh
vpcConfig:
securityGroupIDs:
{{- range .Values.security_groups }}
- {{ . | quote }}
{{- end }}
subnetIDs:
{{- range .Values.subnets }}
- {{ . | quote }}
{{- end }}
apiVersion: lambda.services.k8s.aws/v1alpha1
kind: EventSourceMapping
metadata:
name: service
namespace: service
annotations:
services.k8s.aws/region: {{ .Values.region }}
spec:
scalingConfig:
maximumConcurrency: {{ .Values.maximumConcurrency }}
eventSourceARN: arn:aws:sqs:{{ .Values.region }}:{{ .Values.account }}:Service
functionName: Service
batchSize: 1
We will use Container Lambda to make deployment primitive compatible to Kubernetes. In this example we will use JS wrapper (pick up any language from documentation).
#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
exec /usr/local/bin/aws-lambda-rie /usr/local/bin/npx aws-lambda-ric $@
else
exec /usr/local/bin/npx aws-lambda-ric $@
fi
We will need a package to properly wrap our execution handler:
npm install aws-lambda @types/aws-lambda --save-prod
Then we need a app/lambdas/action.ts with the following content:
import type { Handler, SQSEvent } from "aws-lambda";
import { getSecrets } from "../services/secrets.js";
export const handler: Handler = async (event: SQSEvent) => {
const secrets = await getSecrets();
//...
};
Compatibility layer is added via:
RUN npm install -g aws-lambda-ric
RUN curl -Lo /usr/local/bin/aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie && chmod +x /usr/local/bin/aws-lambda-rie
COPY ./deploy/shared/lambda_entry.sh /lambda_entry.sh
RUN chmod +x /lambda_entry.sh
The built Dockerfile is supposed to go to ECR from where it's going to be pulled by AWS Lambda during deployment.
Using Helm Chart and Docker container, we now deliverable that's automatically provisioned from operator controller. We got versioning, kubernetes compatibility and the incredible speed of Lambda.
Yay :).