Migration on Confluent Platform

This guide follows the Quickstart for Confluent Platform.

Steps

  1. Open the Confluent Control Center instance at http://localhost:9021/clusters and find the registered source or sink connector. Once found, delete it.

  2. Look at the logs for the connect container (docker-compose logs -f connect). A migrated configuration will be printed to the logs as part of the connector deletion/shutdown. Example:

    connect  | [2024-09-04 09:18:40,066] INFO The migrated settings for 5.1 version of Neo4j Source Connector 'Neo4jSourceConnectorAVRO' is: `{
connect  |   "connector.class" : "org.neo4j.connectors.kafka.source.Neo4jConnector",
connect  |   "neo4j.authentication.basic.password" : "",
connect  |   "neo4j.uri" : "bolt://neo4j:7687",
connect  |   "neo4j.query" : "MATCH (ts:TestSource) WHERE ts.timestamp > $lastCheck RETURN ts.name AS name, ts.surname AS surname, ts.timestamp AS timestamp",
connect  |   "neo4j.query.streaming-property" : "timestamp",
connect  |   "value.converter.schema.registry.url" : "http://schema-registry:8081",
connect  |   "task.class" : "streams.kafka.connect.source.Neo4jSourceTask",
connect  |   "neo4j.authentication.basic.username" : "neo4j",
connect  |   "name" : "Neo4jSourceConnectorAVRO",
connect  |   "neo4j.query.topic" : "my-topic",
connect  |   "value.converter" : "io.confluent.connect.avro.AvroConverter",
connect  |   "key.converter" : "io.confluent.connect.avro.AvroConverter",
connect  |   "key.converter.schema.registry.url" : "http://schema-registry:8081",
connect  |   "neo4j.query.poll-interval" : "5000ms",
connect  |   "neo4j.start-from" : "USER_PROVIDED",
connect  |   "neo4j.start-from.value" : 1725441505774
connect  | }` (streams.kafka.connect.source.Neo4jSourceService)

  3. Create a new JSON file (source/sink)_migrated.neo4j.json and copy the migrated example config to this file along with the connector name. The configuration should contain a name property and the migrated configuration should be nested in the config key.

  4. Validate that the new configuration contains all the relevant keys compared with the previous version of the connector configuration.

    1. See Source Configuration Settings and Sink Configuration Settings for descriptions and examples of the new configuration options.

    2. If migrating the Source component, make note of the neo4j.start-from.value that has been set to the last checked offset.

    3. Replace the sensitive values.

      Sensitive values in the original configuration are not printed in the migrated configuration. These keys need to be filled in with the appropriate values. The affected configuration keys are neo4j.authentication.basic.password and neo4j.authentication.kerberos.ticket.

  5. Download the new connector version following the Installation guide and remove the original 5.0.x connector version. Make sure to copy the new plugin into the ./plugins/ folder created during the Docker setup.

  6. Restart the Kafka Connect Worker by running docker-compose restart connect. This allows the Kafka Connect platform to pick up the new plugin from the ./plugins/ folder.

  7. Continue following the Quickstart for Confluent Platform to deploy the plugin to Kafka Connect with the new configuration.

  8. Once the connector is up and running, validate it is successfully running.