Laravel

Laravel in Kubernetes Part 8 - Deploying Laravel Queue workers in Kubernetes

In this post we will cover deploying Laravel Queue workers in Laravel.

Deploying Laravel Queue workers in Kubernetes, makes it fairly easy to scale out workers when jobs start piling up, and releasing resources when there is lower load on the system.

Queue connection update

We need to make sure the Queue Workers can connect to our Redis instance.

Update the ConfigMap and Secret in the common/ directory to have the new Redis Details, and switch the Queue Driver to Redis

Updating the ConfigMap

Update the details in the common/app-config.yml for Redis and the Queue Driver

apiVersion: v1
kind: ConfigMap
metadata:
  name: laravel-in-kubernetes
data:
  QUEUE_CONNECTION: "redis"
  REDIS_HOST: "XXX"
  REDIS_PORT: "XXX"

We can apply the new ConfigMap

$ kubectl apply -f common/app-config.yml
configmap/laravel-in-kubernetes configured

Updating the Secret

Update the details in the common/app-secret.yml to contain the new Redis connection details.

apiVersion: v1
kind: Secret
metadata:
  name: laravel-in-kubernetes
type: Opaque
stringData:
  REDIS_PASSWORD: "XXX" # If you have no password set, you can set this to an empty string

We can apply the new Secret, and then move on to running the actual Queues.

$ kubectl apply -f common/app-secret.yml
secret/laravel-in-kubernetes configured

Queue directory

First thing we'll need is a new directory in our deployment repo called queue-workers.

Here is where we will configure our queue-workers.

$ mkdir -p queue-workers

Creating the deployment

Next, we need to create a Deployment for our queue workers, which will run them and be able to scale them for us.

In the queue-workers directory, create a new file called deployment-default.yml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: laravel-in-kubernetes-queue-worker-default
  labels:
    tier: backend
    layer: queue-worker
    queue: default
spec:
  replicas: 1
  selector:
    matchLabels:
      tier: backend
      layer: queue-worker
      queue: default
  template:
    metadata:
      labels:
        tier: backend
        layer: queue-worker
        queue: default
    spec:
      containers:
        - name: queue-worker
          image: [your_registry_url]/cli:v0.0.1
          command:
            - php
          args:
            - artisan
            - queue:work
            - --queue=default
            - --max-jobs=200
          ports:
            - containerPort: 9000
          envFrom:
            - configMapRef:
                name: laravel-in-kubernetes
            - secretRef:
                name: laravel-in-kubernetes

This deployment will deploy the queue workers for the default queue only. We will cover adding more queues further down in the post.

Let's apply the new Queue worker, and check that the pod is running correctly.

$ kubectl apply -f queue-workers/deployment-default.yml 
deployment.apps/laravel-in-kubernetes-queue-worker-default created

$ kubectl get pods
NAME                                                          READY   STATUS    RESTARTS   AGE
laravel-in-kubernetes-fpm-856dcb9754-trf65                    1/1     Running   0          10h
laravel-in-kubernetes-queue-worker-default-594bc6f4bb-8swdw   1/1     Running   0          9m38s
laravel-in-kubernetes-webserver-5877867747-zm7zm              1/1     Running   0          10h

That's it. The Queue workers are running correctly.

Separate queues

Our current deployment is running the queue for only the default queue.

If we'd like to add additional workers for more queues, we can simply add another deployment file called deployment-{queue-name}.yml, update the queue label with the new name, and update the --queue flag to the new queue name.

Once we apply that, we'll have a second group of queue workers to run our other queue.

We can also run a single queue

If you have not built in multiple queues into your application, you can also remove the --queue flag from the queue-worker deployment to have it run all of the queued jobs.