Working with Schema Registry

Neo4j is a schema-less database, which poses challenges when messages need to be converted to schema formats like Avro, Protobuf, or JSON Schema for transmission via Kafka. Schema conflicts may arise due to the flexible nature of the graph data. To mitigate conflicts, it’s crucial to establish a structured approach to schema enforcement and validation.

Enforcing Structure

When using schema formats like Avro, Protobuf, or JSON Schema with Neo4j, it’s recommended to invest efforts in enforcing structure within the data. This includes:

  • Type Consistency in Exported Properties: Ensure that properties exported by the Source connector have the same type across all nodes within the label and all relationships within the type.

  • Type and Existence Constraints: Utilize type and existence constraints, which are available in Neo4j Enterprise and AuraDB. Follow Cypher Manual for more details.

Evolving Graph Structure

When evolving the graph data structure, it’s essential to plan ahead, considering Schema Registry compatibility modes. For instance, in the default compatibility mode (BACKWARD), adding new properties is generally safe. However, deletions or type changes should be approached cautiously, as they can introduce conflicts between schema versions.

Depending on specific requirements and data structure, changing the compatibility mode, even up to NONE, may be necessary. However, it’s important to thoroughly test compatibility changes before implementing.

Handling Unstructured Graph Data

In scenarios where graph data is challenging to structure or evolve, it’s advisable to explore whether you can relax compatibility mode and manage schema changes on consumer side. If it’s not feasible, employing a schema-less message format is recommended. However, you will lose schema-based type conversion support when using a schema-less message format.