docs: Added Keys Source Connector documentation

Author: jruaux

docs/guide/src/docs/asciidoc/_links.adoc

@@ -1,4 +1,5 @@
 :link_redis_enterprise: link:https://redis.com/redis-enterprise-software/overview/[Redis Enterprise]
 :link_lettuce_uri:      link:https://github.com/lettuce-io/lettuce-core/wiki/Redis-URI-and-connection-details#uri-syntax[Redis URI Syntax]
 :link_redis_notif:      link:https://redis.io/docs/manual/keyspace-notifications[Redis Keyspace Notifications]
-:link_manual_install:   link:https://docs.confluent.io/home/connect/community.html#manually-installing-community-connectors/[Manually Installing Community Connectors]
+:link_manual_install:   link:https://docs.confluent.io/home/connect/community.html#manually-installing-community-connectors/[Manually Installing Community Connectors]
+:link_redis_keys:       https://redis.io/commands/keys/[Redis KEYS]

docs/guide/src/docs/asciidoc/docker.adoc

@@ -1,7 +1,7 @@
 [[_docker]]
 = Quick Start with Docker
 
-This guide provides a hands-on look at the functionality of the Redis Kafka Source and Sink Connectors:
+This section provides a hands-on look at the functionality of the Redis Kafka Source and Sink Connectors:
 
 * The *redis-sink* connector reads data from a Kafka topic and writes it to a Redis stream
 * The *redis-source* connector reads data from a Redis stream and writes it to a Kafka topic

docs/guide/src/docs/asciidoc/keyreader.adoc

@@ -43,7 +43,7 @@ Use the following properties to write Kafka records as Redis hashes:
 
 [source,properties]
 ----
-redis.type=HASH
+redis.command=HSET
 key.converter=<string or bytes> <1>
 value.converter=<Avro or JSON> <2>
 ----
@@ -58,7 +58,7 @@ Use the following properties to write Kafka records as Redis strings:
 
 [source,properties]
 ----
-redis.type=STRING
+redis.command=SET
 key.converter=<string or bytes> <1>
 value.converter=<string or bytes> <2>
 ----
@@ -73,7 +73,7 @@ Use the following properties to write Kafka records as RedisJSON documents:
 
 [source,properties]
 ----
-redis.type=JSON
+redis.command=JSONSET
 key.converter=<string, bytes, or Avro> <1>
 value.converter=<string, bytes, or Avro> <2>
 ----
@@ -88,7 +88,7 @@ Use the following properties to store Kafka records as Redis stream messages:
 
 [source,properties]
 ----
-redis.type=STREAM
+redis.command=XADD
 redis.key=<stream key> <1>
 value.converter=<Avro or JSON> <2>
 ----
@@ -102,15 +102,14 @@ Use the following properties to add Kafka record keys to a Redis list:
 
 [source,properties]
 ----
-redis.type=LIST
-redis.key=<key name> <1>
-key.converter=<string or bytes> <2>
-redis.push.direction=<LEFT or RIGHT> <3>
+redis.command=<LPUSH or RPUSH> <1>
+redis.key=<key name> <2>
+key.converter=<string or bytes> <3>
 ----
 
-<1><<_collection_key,List key>>
-<2> <<_key_string,String>> or <<_key_bytes,bytes>>: Kafka record keys to push to the list
-<3> `LEFT`: LPUSH (default), `RIGHT`: RPUSH
+<1> `LPUSH` or `RPUSH`
+<2> <<_collection_key,List key>>
+<3> <<_key_string,String>> or <<_key_bytes,bytes>>: Kafka record keys to push to the list
 
 The Kafka record value can be any format.
 If a value is null then the member is removed from the list (instead of pushed to the list).
@@ -121,7 +120,7 @@ Use the following properties to add Kafka record keys to a Redis set:
 
 [source,properties]
 ----
-redis.type=SET
+redis.command=SADD
 redis.key=<key name> <1>
 key.converter=<string or bytes> <2>
 ----
@@ -138,7 +137,7 @@ Use the following properties to add Kafka record keys to a Redis sorted set:
 
 [source,properties]
 ----
-redis.type=ZSET
+redis.command=ZADD
 redis.key=<key name> <1>
 key.converter=<string or bytes> <2>
 ----
@@ -155,7 +154,7 @@ Use the following properties to write Kafka records as RedisTimeSeries samples:
 
 [source,properties]
 ----
-redis.type=TIMESERIES
+redis.command=TSADD
 redis.key=<key name> <1>
 ----
 

docs/guide/src/docs/asciidoc/sink.adoc

@@ -2,21 +2,30 @@
 = Source Connector Guide
 :name: Redis Kafka Source Connector
 
-The {name} reads from a Redis stream and publishes messages to a Kafka topic.
+The {name} includes 2 source connectors:
+
+* <<_stream_source,Stream>>
+* <<_keys_source,Keys>>
+
+[[_stream_source]]
+== Stream
+
+The stream source connector reads from a Redis stream and publishes messages to a Kafka topic.
 It includes the following features:
 
-* <<_source_at_least_once_delivery,At least once delivery>>
-* <<_source_at_most_once_delivery,At most once delivery>>
-* <<_source_tasks,Multiple tasks>>
-* <<_stream_reader,Stream Reader>>
+* <<_stream_source_at_least_once_delivery,At least once delivery>>
+* <<_stream_source_at_most_once_delivery,At most once delivery>>
+* <<_stream_source_tasks,Multiple tasks>>
+* <<_stream_source_schema,Schema>>
+* <<_stream_source_config,Configuration>>
 
-== Delivery Guarantees
+=== Delivery Guarantees
 
 The {name} can be configured to ack stream messages either automatically (at-most-once delivery) or explicitly (at-least-once delivery).
 The default is at-least-once delivery.
 
-[[_source_at_least_once_delivery]]
-=== At-Least-Once
+[[_stream_source_at_least_once_delivery]]
+==== At-Least-Once
 
 In this mode, each stream message is acknowledged after it has been written to the corresponding topic.
 
@@ -25,8 +34,8 @@ In this mode, each stream message is acknowledged after it has been written to t
 redis.stream.delivery=at-least-once
 ----
 
-[[_source_at_most_once_delivery]]
-=== At-Most-Once
+[[_stream_source_at_most_once_delivery]]
+==== At-Most-Once
 
 In this mode, stream messages are acknowledged as soon as they are read.
 
@@ -35,19 +44,14 @@ In this mode, stream messages are acknowledged as soon as they are read.
 redis.stream.delivery=at-most-once
 ----
 
-[[_source_tasks]]
-== Multiple Tasks
-Use configuration property `tasks.max` to have the change stream handled by multiple tasks.
+[[_stream_source_tasks]]
+=== Multiple Tasks
+Reading from the stream is done through a consumer group so that multiple instances of the connector configured via the `tasks.max` can consume messages in a round-robin fashion.
+
 The connector splits the work based on the number of configured key patterns.
 When the number of tasks is greater than the number of patterns, the number of patterns will be used instead.
 
-
-[[_stream_reader]]
-== Stream Reader
-The {name} reads messages from a stream and publishes to a Kafka topic.
-Reading is done through a consumer group so that <<_source_tasks,multiple instances>> of the connector configured via the `tasks.max` can consume messages in a round-robin fashion.
-
-
+[[_stream_source_schema]]
 === Message Schema
 
 ==== Key Schema
@@ -66,16 +70,19 @@ The value schema defines the following fields:
 |body  |Map of STRING|Stream message body
 |====
 
+[[_stream_source_config]]
 === Configuration
 
 [source,properties]
 ----
+connector.class=com.redis.kafka.connect.RedisStreamSourceConnector
 redis.stream.name=<name> <1>
 redis.stream.offset=<offset> <2>
 redis.stream.block=<millis> <3>
 redis.stream.consumer.group=<group> <4>
 redis.stream.consumer.name=<name> <5>
-topic=<name> <6>
+redis.stream.delivery=<mode> <6>
+topic=<name> <7>
 ----
 
 <1> Name of the stream to read from.
@@ -85,10 +92,41 @@ topic=<name> <6>
 <5> Name of the stream consumer (default: `consumer-${task}`).
 May contain `${task}` as a placeholder for the task id.
 For example, `foo${task}` and task `123` => consumer `foo123`.
-<6> Destination topic (default: `${stream}`).
+<6> Delivery mode: `at-least-once`, `at-most-once` (default: `at-least-once`).
+<7> Destination topic (default: `${stream}`).
 May contain `${stream}` as a placeholder for the originating stream name.
 For example, `redis_${stream}` and stream `orders` => topic `redis_orders`.
 
-//[[key-reader]]
-//include::_keyreader.adoc[]
+[[_keys_source]]
+== Keys
+
+The keys source connector captures changes happening to keys in a Redis database and publishes keys and values to a Kafka topic.
+The data structure key will be mapped to the record key, and the value will be mapped to the record value.
+
+[WARNING]
+====
+The keys source connector does not guarantee data consistency because it relies on Redis keyspace notifications which have no delivery guarantees.
+It is possible for some notifications to be missed, for example in case of network failures.
+
+Also, depending on the type, size, and rate of change of data structures on the source it is possible the source connector cannot keep up with the change stream.
+For example if a big set is repeatedly updated the connector will need to read the whole set on each update and transfer it over to the target database.
+With a big-enough set the connector could fall behind and the internal queue could fill up leading up to updates being dropped.
+Some preliminary sizing using Redis statistics and `bigkeys`/`memkeys` is recommended.
+If you need assistance please contact your Redis account team.
+====
+
+[[_keys_source_config]]
+=== Configuration
+[source,properties]
+----
+connector.class=com.redis.kafka.connect.RedisKeysSourceConnector
+redis.keys.patterns=<glob> <1>
+redis.keys.timeout=<millis> <2>
+topic=<name> <3>
+----
+<1> Key pattern to subscribe to. This is the key portion of the pattern that will be used to listen to keyspace events. See {link_redis_keys} for pattern details.
+For example `foo:*` translates to pubsub channel `$$__$$keyspace@0$$__$$:foo:*` and will capture changes to keys `foo:1`, `foo:2`, etc.
+Use comma-separated values for multiple patterns (`foo:*,bar:*`)
+<2> Idle timeout in millis. Duration after which the connector will stop if no activity is encountered.
+<3> Name of the destination topic.