redis-cluster-operator

Redis Cluster Operator manages Redis Cluster atop Kubernetes.

The operator itself is built with the Operator framework.

Each master node and its slave nodes is managed by a statefulSet, create a headless svc for each statefulSet, and create a clusterIP service for all nodes.

Each statefulset uses PodAntiAffinity to ensure that the master and slaves are dispersed on different nodes. At the same time, when the operator selects the master in each statefulset, it preferentially select the pod with different k8s nodes as master.

Table of Contents

• Overview
• Features
• Getting Started
  • Prerequisites
  • To Deploy on the Cluster
  • To Uninstall
• Project Distribution
  • Build Installer
  • Using the Installer
• Usage
  • Deploy a Sample Redis Cluster
  • Scaling Up the Redis Cluster
  • Scaling Down the Redis Cluster
  • Backup and Restore
  • Prometheus Discovery
  • Create Redis Cluster with password
  • Persistent Volume
  • Custom Configuration
  • Custom Service
  • Custom Resource
  • Cluster Cleanup Job
• Contributing
• End to End Tests

Features

• Customize the number of master nodes and the number of replica nodes per master

• Password

• Safely Scaling the Redis Cluster

• Persistent Volume

• Custom Configuration

Getting Started

Prerequisites

• go version v1.22.0+
• docker version 17.03+.
• kubectl version v1.11.3+.
• Access to a Kubernetes v1.11.3+ cluster.

To Deploy on the cluster

Build and push your image to the location specified by IMG:

make docker-build docker-push IMG=tyk/redis-cluster-operator:tag

NOTE: This image ought to be published in the personal registry you specified. And it is required to have access to pull the image from the working environment. Make sure you have the proper permission to the registry if the above commands don’t work.

Install the CRDs into the cluster:

make install

Deploy the Manager to the cluster with the image specified by IMG:

make deploy IMG=<some-registry>/redis-cluster-operator:tag

Create instances of your solution You can apply the samples (examples) from the config/sample:

kubectl apply -k config/samples/

NOTE: Ensure that the samples has default values to test it out.

To Uninstall

Delete the instances (CRs) from the cluster:

kubectl delete -k config/samples/

Delete the APIs(CRDs) from the cluster:

make uninstall

UnDeploy the controller from the cluster:

make undeploy

Project Distribution

Following are the steps to build the installer and distribute this project to users.

1. Build the installer for the image built and published in the registry:

make build-installer IMG=tyk/redis-cluster-operator:tag

NOTE: The makefile target mentioned above generates an 'install.yaml' file in the dist directory. This file contains all the resources built with Kustomize, which are necessary to install this project without its dependencies.

2. Using the installer

Users can just run kubectl apply -f to install the project, i.e.:

kubectl apply -f https://raw.githubusercontent.com/<org>/redis-cluster-operator/<tag or branch>/dist/install.yaml

Usage

Deploy a sample Redis Cluster

NOTE: Only the redis cluster that use persistent storage(pvc) can recover after accidental deletion or rolling update.Even if you do not use persistence(like rdb or aof), you need to set pvc for redis.

$ kubectl apply -f config/samples/example/redis.kun_v1alpha1_distributedrediscluster_cr.yaml

Verify that the cluster instances and its components are running.

$ kubectl get distributedrediscluster
NAME                              MASTERSIZE   STATUS    AGE
example-distributedrediscluster   3            Scaling   11s

$ kubectl get all -l redis.kun/name=example-distributedrediscluster
NAME                                          READY   STATUS    RESTARTS   AGE
pod/drc-example-distributedrediscluster-0-0   1/1     Running   0          2m48s
pod/drc-example-distributedrediscluster-0-1   1/1     Running   0          2m8s
pod/drc-example-distributedrediscluster-1-0   1/1     Running   0          2m48s
pod/drc-example-distributedrediscluster-1-1   1/1     Running   0          2m13s
pod/drc-example-distributedrediscluster-2-0   1/1     Running   0          2m48s
pod/drc-example-distributedrediscluster-2-1   1/1     Running   0          2m15s

NAME                                        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
service/example-distributedrediscluster     ClusterIP   172.17.132.71   <none>        6379/TCP,16379/TCP   2m48s
service/example-distributedrediscluster-0   ClusterIP   None            <none>        6379/TCP,16379/TCP   2m48s
service/example-distributedrediscluster-1   ClusterIP   None            <none>        6379/TCP,16379/TCP   2m48s
service/example-distributedrediscluster-2   ClusterIP   None            <none>        6379/TCP,16379/TCP   2m48s

NAME                                                     READY   AGE
statefulset.apps/drc-example-distributedrediscluster-0   2/2     2m48s
statefulset.apps/drc-example-distributedrediscluster-1   2/2     2m48s
statefulset.apps/drc-example-distributedrediscluster-2   2/2     2m48s

$ kubectl get distributedrediscluster
NAME                              MASTERSIZE   STATUS    AGE
example-distributedrediscluster   3            Healthy   4m

Scaling Up the Redis Cluster

Increase the masterSize to trigger the scaling up.

apiVersion: redis.kun/v1alpha1
kind: DistributedRedisCluster
metadata:
  annotations:
    # if your operator run as cluster-scoped, add this annotations
    redis.kun/scope: cluster-scoped
  name: example-distributedrediscluster
spec:
  # Increase the masterSize to trigger the scaling.
  masterSize: 4
  ClusterReplicas: 1
  image: redis:5.0.4-alpine

Scaling Down the Redis Cluster

Decrease the masterSize to trigger the scaling down.

apiVersion: redis.kun/v1alpha1
kind: DistributedRedisCluster
metadata:
  annotations:
    # if your operator run as cluster-scoped, add this annotations
    redis.kun/scope: cluster-scoped
  name: example-distributedrediscluster
spec:
  # Decrease the masterSize to trigger the scaling.
  masterSize: 3
  ClusterReplicas: 1
  image: redis:5.0.4-alpine

Backup and Restore

NOTE: Only Ceph S3 object storage and PVC is supported now

Backup

$ kubectl create -f config/samples/example/backup-restore/redisclusterbackup_cr.yaml

Restore from backup

$ kubectl create -f config/samples/example/backup-restore/restore.yaml

Prometheus Discovery

$ kubectl create -f config/samples/example/prometheus-exporter.yaml

Create Redis Cluster with password

$ kubectl create -f config/samples/example/custom-password.yaml

Persistent Volume

$ kubectl create -f config/samples/example/persistent.yaml

Custom Configuration

$ kubectl create -f config/samples/example/custom-config.yaml

Custom Service

$ kubectl create -f config/samples/example/custom-service.yaml

Custom Resource

$ kubectl create -f config/samples/example/custom-resources.yaml

Cluster Cleanup Job

RedisClusterCleanup CRD

• Purpose:
  Defines the cleanup job configuration for Redis clusters.
• Key Fields:
  • Namespaces: List of namespaces to target for cleanup.
  • Schedule: Cron schedule determining when the cleanup job runs.
  • KeyPatterns: Patterns used to identify keys eligible for inspection.
  • ExpirationRegexes: List of regex strings to extract expiration timestamps from key values.
  • SkipPatterns: Patterns that, if present in a key's value, will cause the key to be skipped.
  • ScanBatchSize: Number of keys to scan in one batch.
  • ExpiredThreshold: Minimum count of expired keys that triggers batch deletion.
  • Suspend: Boolean flag to temporarily disable the cleanup job.

Create Cleanup Job

$ kubectl create -f config/samples/redis_v1alpha1_redisclustercleanup.yaml

Contributing

NOTE: Run make help for more information on all potential make targets

More information can be found via the Kubebuilder Documentation

End to End Tests

see e2e