New server rolling upgrade of a cluster

It is recommended to read the Introduction and Upgrade or patch 5.X before continuing.

This is an example of how to do a rolling upgrade without reducing fault tolerance, by adding servers before removing old ones. This approach is suitable for deployments that use replaceable cloud or container resources.

Before you begin

  1. Ensure that the databases cannot be stopped, created, or dropped during the rolling upgrade by using the following command:

    DENY STOP ON DATABASE * TO PUBLIC
DENY DATABASE MANAGEMENT ON DBMS TO PUBLIC

  2. Verify that all databases are online. See Operations Manual → Start databases for details.

Upgrade steps (to be repeated for each server)

Perform these steps in full for each server, before moving on to the next server. You must ensure that the cluster returns to a stable state before moving on to the next.

  1. Check the cluster is healthy.

    1. Before replacing the server, ensure that the cluster is in a healthy state. Use the following query to check servers are hosting all their assigned databases. The query should return no results:

      SHOW SERVERS YIELD name, hosting, requestedHosting, serverId WHERE requestedHosting <> hosting

      Up to version 5.9, if you have composite databases, they will not appear in requestedHosting, and so all servers are returned by this query. Thus, you must manually compare the hosting and requestedHosting fields, excluding any composite databases. From version 5.10 onwards, composite databases appear in both hosting and requestedHosting fields.

    2. Use the following query to check all databases are in their expected state. The query should return no results:

      SHOW DATABASES YIELD name, address, currentStatus, requestedStatus, statusMessage WHERE currentStatus <> requestedStatus RETURN name, address, currentStatus, requestedStatus, statusMessage

  2. Create the new server.

  3. Add the new server to the cluster:

    1. Start the new server and wait for it to be visible to the cluster.

    2. Use SHOW SERVERS to see the new server, it should have the status Free. Make a note of the new server’s ID.

    3. Add the server to the DBMS by running ENABLE SERVER [new-server-id]. This allows the DBMS to use the server to host databases.

  4. Prepare a server for removal:

    1. Run SHOW SERVERS to retrieve the ID of one of the old servers. The server must be Available.

    2. Move all databases off the old server by running:

      DEALLOCATE DATABASES FROM SERVER [old-server-id]

    3. Wait for all databases to be deallocated. You can use SHOW SERVER [old-server-id] to follow the process. Once the databases complete their move, the hosting column will contain only system. From 5.15 onwards, when all databases are deallocated, the state column will display DEALLOCATED.

    4. Verify that all moved databases are online on their new host by running:

      SHOW DATABASES

  5. Remove the old server:

    1. Shut down the old server.

    2. When the process has stopped, remove the old server by running DROP SERVER [old-server-id]. It may still appear as Dropped in SHOW SERVERS until the process stops.

  6. (Optional) If you have installed any plugins, make sure they are compatible with the new version.

    The latest APOC is available under /labs, whilst GDS, Bloom, and NOM are available under /products. Currently, an incompatible GDS or APOC version will prevent Neo4j startup.

  7. (Optional) Restore the privilege of the PUBLIC role to stop databases:

    REVOKE DENY STOP ON DATABASE * FROM PUBLIC

  8. (Optional) Restore the privilege of the PUBLIC role to create and drop databases:

    REVOKE DENY DATABASE MANAGEMENT ON DBMS FROM PUBLIC

  9. (Optional) If you have started offline databases during the preparation phase for a rolling upgrade, you will need to stop each of them in order to restore them to the original state.

  10. If using the LIST discovery method, after you upgrade all the servers make sure to set dbms.cluster.discovery.endpoints to include only the new servers.

    See also dbms.cluster.discovery.resolver_type.

Manually upgrade the system database

If you are on 5.8 or an earlier version of Neo4j, you must manually upgrade the system database. Connect to the system database on any of the servers and run the following procedure:

CALL dbms.upgrade();

From version 5.9, the system database is automatically upgraded when safe.

Monitor the logs

When Neo4j restarts, it is a good idea to monitor the logs for any errors or warnings caused by the upgrade. The neo4j.log file contains information on the upgrade.