Managing servers in a cluster

As described previously, server-management is completely separate from database-management in a clustered environment. This section describes how to work with servers in a cluster: adding and removing them, as well as altering their metadata.

Server states

A server can exist in five different states within the DBMS:

  • Free

  • Enabled

  • Deallocating

  • Deallocated Introduced in 5.15

  • Cordoned

  • Dropped

server states4

Free state

When a server is discovered by the discovery service (see Cluster server discovery for more information), it is created in the Free state. Servers in this state have a unique automatically generated ID, but are otherwise unconfigured. These free servers are not yet part of the cluster and cannot be allocated to host any databases.

When first discovered, a server’s name defaults to the value of its generated server ID.

Enabled state

A server in the free state needs to be explicitly enabled in order to be considered an active member of the cluster. The command ENABLE SERVER server name is used to transition a server into this Enabled state. The server’s initial name is its ID. If the server is already enabled and the command is executed with the same options, nothing is changed.

The possible options when enabling a server are:

Option Allowed values Description

modeConstraint

PRIMARY, SECONDARY, NONE

Databases may only be hosted on the server in the mode specified by the constraint. None means there is no constraint and any mode is allowed.

allowedDatabases

list of database names, e.g. ["db1", "db2"]

Only databases matching the specified names may be hosted on the server. This may not be specified in combination with deniedDatabases.

deniedDatabases

list of database names, e.g. ["db1", "db2"]

Only databases not matching the specified names may be hosted on the server. This may not be specified in combination with allowedDatabases.

tags

list of server tags, e.g. ["tag1", "tag2"]

List of server tags used during database allocation and for load balancing and routing policies. Introduced in 5.6

When a server is enabled, if no OPTIONS are not provided, the default server values are taken from the settings initial.server.mode_constraint, initial.server.allowed_databases, initial.server.denied_databases, and/or initial.server.tags.

Once enabled, the server may be allocated databases to host.

Deallocating state

When a server is no longer needed, it cannot be removed from the cluster while it is still allocated to host any databases. The command DEALLOCATE DATABASE[S] FROM SERVER[S] server[,…​] is used to transition servers to the Deallocating state, reallocating all their hosted databases to other servers in the cluster. Additionally, servers which are deallocating will not have any further databases allocated to them.

Deallocated state

When a server is in the deallocated state, it no longer hosts any databases besides system and can be removed from the cluster. Additionally, deallocated servers cannot have any further databases allocated to them. Note that there is a known situation in which a previously deallocated offline server can transiently show as deallocating when restarting, it will, however, eventually return to the deallocated state without intervention.

Cordoned state

The Cordoned state is similar to Deallocating in that servers in this state will not be allocated to host additional databases. Unlike Deallocating however, cordoned servers do not lose the databases they already host. It is worth noting that when decreasing the number of allocations of a database, allocations on cordoned servers are removed first.

A server is transitioned from the Enabled state to the Cordoned state by executing the procedure dbms.cluster.cordonServer. A server in the Cordoned state may be transitioned to Deallocating, or back to Enabled.

This state is primarily used for error handling.

Dropped state

Once a server is in state Deallocating and is only hosting the system database, it is safe to drop it. The command DROP SERVER server name logically removes the server from the cluster. However, as long as the server’s Neo4j process is running, it is still visible to the other cluster members in the Dropped state. Once the Neo4j process is stopped, the server finally disappears. Once dropped, a server cannot rejoin a cluster.

The same physical hardware can rejoin the cluster, provided the Neo4j installation has been "reset" (either re-installing, or running neo4j-admin unbind), causing it to receive a new generated server ID on next startup.

Listing servers

The Cypher command SHOW SERVERS displays all current servers running in the cluster, including both servers yet to be enabled (i.e. servers in the Free state) in the DBMS as well as dropped servers.

neo4j@neo4j> SHOW SERVERS;
+------------------------------------------------------------------------------------------------------------------+
| name                                   | address          | state     | health      | hosting                    |
+------------------------------------------------------------------------------------------------------------------+
| "135ad202-5405-4d3c-9822-df39f59b823c" | "localhost:7690" | "Dropped" | "Available" | ["system"]                 |
| "25a7efc7-d063-44b8-bdee-f23357f89f01" | "localhost:7689" | "Enabled" | "Available" | ["system", "foo", "neo4j"] |
| "42a97acc-acf6-40c0-aff2-3993e90db1ff" | "localhost:7691" | "Free"    | "Available" | ["system"]                 |
| "782f0ee2-5474-4250-b905-4cd8b8f586ba" | "localhost:7688" | "Enabled" | "Available" | ["system", "foo", "neo4j"] |
| "8512c9b9-d9e8-48e6-b037-b15b0004ca18" | "localhost:7687" | "Enabled" | "Available" | ["system", "foo", "neo4j"] |
+------------------------------------------------------------------------------------------------------------------+

To display all available information about the servers in the cluster, use SHOW SERVERS YIELD *:

neo4j@neo4j> SHOW SERVERS YIELD *;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| serverId                               | name                                   | address          | httpAddress      | httpsAddress | state          | health      | hosting                    | requestedHosting           | tags | allowedDatabases | deniedDatabases | modeConstraint | version          |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "135ad202-5405-4d3c-9822-df39f59b823c" | "135ad202-5405-4d3c-9822-df39f59b823c" | "localhost:7690" | "localhost:7477" | NULL         | "Deallocating" | "Available" | ["system"]                 | ["system"]                 | []   | []               | []              | "NONE"         | "5.0.0-drop09.0" |
| "25a7efc7-d063-44b8-bdee-f23357f89f01" | "25a7efc7-d063-44b8-bdee-f23357f89f01" | "localhost:7689" | "localhost:7476" | NULL         | "Enabled"      | "Available" | ["system", "foo", "neo4j"] | ["system", "foo", "neo4j"] | []   | []               | []              | "NONE"         | "5.0.0-drop09.0" |
| "42a97acc-acf6-40c0-aff2-3993e90db1ff" | "42a97acc-acf6-40c0-aff2-3993e90db1ff" | "localhost:7691" | "localhost:7478" | NULL         | "Free"         | "Available" | ["system"]                 | []                         | []   | []               | []              | "NONE"         | "5.0.0-drop09.0" |
| "782f0ee2-5474-4250-b905-4cd8b8f586ba" | "782f0ee2-5474-4250-b905-4cd8b8f586ba" | "localhost:7688" | "localhost:7475" | NULL         | "Enabled"      | "Available" | ["system", "foo", "neo4j"] | ["system", "foo", "neo4j"] | []   | []               | []              | "NONE"         | "5.0.0-drop09.0" |
| "8512c9b9-d9e8-48e6-b037-b15b0004ca18" | "8512c9b9-d9e8-48e6-b037-b15b0004ca18" | "localhost:7687" | "localhost:7474" | NULL         | "Enabled"      | "Available" | ["system", "foo", "neo4j"] | ["system", "foo", "neo4j"] | []   | []               | []              | "NONE"         | "5.0.0-drop09.0" |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

The table of results shows information about the servers:

Column Description Type Default output Full output

name

Name of the server.

STRING

serverId

Id of the server.

STRING

address

Bolt address of the server (if enabled).

STRING

httpAddress

Http address of the server (if enabled).

STRING

httpsAddress

Https address of the server (if enabled).

STRING

state

Information of the state of the server: free, enabled, deallocating,cordoned, or dropped.

STRING

health

The availability of the server: available or unavailable.

STRING

hosting

A list of databases currently hosted on the server.

LIST<STRING>

requestedHosting

A list of databases that should be hosted on the server. Composite databases do not currently appear in this list, though they do appear in hosting for all servers.

LIST<STRING>

tags

Tags are user provided strings that can be used while allocating databases.

LIST<STRING>

allowedDatabases

A list of databases allowed to be hosted on the server.

LIST<STRING>

deniedDatabases

A list of databases not allowed to be hosted on the server.

LIST<STRING>

modeConstraint

Constraint for the allocator to allocate only databases in this mode on the server.

STRING

version

Neo4j version the server is running.

STRING

Add a server to the cluster

To add a server to a running cluster (see Deploy a basic cluster for more information on how to set up a basic cluster), configure it to discover other existing cluster members. There are several different ways to do this, see Cluster server discovery. Once the new server is configured to discover the cluster’s members, it can be started.

Once started, the new server appears in the output of SHOW SERVERS with the Free state. Copy the server’s name from SHOW SERVERS and enable it:

neo4j@neo4j> ENABLE SERVER '42a97acc-acf6-40c0-aff2-3993e90db1ff';

The ENABLE command can take several options:

neo4j@neo4j> ENABLE SERVER '25a7efc7-d063-44b8-bdee-f23357f89f01' OPTIONS
    {modeConstraint:'PRIMARY', allowedDatabases:['foo'], tags:['eu','eu-west']};

modeConstraint is used to control whether a server can be used to host a database in only primary or secondary mode. allowedDatabases and deniedDatabases are collections of database names that filter which databases may be hosted on a server. The allowedDatabases and deniedDatabases are mutually exclusive and if both are specified, an error is returned.

Optionally, it is possible to automatically enable free servers by setting the initial.dbms.automatically_enable_free_servers to true. This can be changed after startup using the dbms.cluster.setAutomaticallyEnableFreeServers procedure.

Server tags are used during database allocation and when configuring load balancing and replication policies. They cannot contain duplicates, so tags:['eu', 'eu'] will return an error. Server tags also cannot contain commas. When altering server tags via cypher, the encoding is done via UTF-8.

allowedDatabases and deniedDatabases do not affect Composite databases, they are always available everywhere.

If no options are set, a server can host any database in any mode. Servers can also provide default values for these options via their neo4j.conf files when enabled.

initial.server.mode_constraint='PRIMARY'
initial.server.allowed_databases='foo'
initial.server.denied_databases='bar','baz'
initial.server.tags=eu,eu-west

If conflicting options are provided between neo4j.conf and the ENABLE SERVER command, those provided to ENABLE SERVER are used.

The input for server tags is a comma-separated list that cannot have duplicates.

Neo4j .conf files use Latin1 for their encoding by default. Therefore, for server tags that need a larger character set (e.g. Chinese or Arabic), it is recommended to use Cypher to alter server tags.

From Neo4j 5.8.0 , .conf files can use UTF-8 by setting the environment variable NEO4J_CONFIG_FILE_CHARSET=utf8. This allows setting server tags with the larger character set via the config.

Hosting databases on added servers

Once enabled, a server does not automatically host databases unless:

  • New databases are created.

  • Existing database topologies are altered to request more hosts.

  • Another server is transitioned to the Deallocating state.

  • You explicitly rebalance the databases across the cluster.

The command REALLOCATE DATABASE[S] can be used to rebalance database allocations across the cluster, adding some to the newly added server(s), see Reallocate databases for more information.

Removing a server from the cluster

Removing a server from the cluster requires two steps: deallocating, then dropping.

Deallocating databases from a server

In preparation for removing a server from the cluster, set it to not host any databases with DEALLOCATE DATABASES FROM SERVER 'name' (see Deallocating state for more information).

Either the server ID or its name can be used with the DEALLOCATE DATABASES command:

neo4j@neo4j> DRYRUN DEALLOCATE DATABASES FROM SERVER '135ad202-5405-4d3c-9822-df39f59b823c';

When deallocating databases from servers, it is important to be mindful of the topology for each database to ensure that there are sufficient servers left in the cluster to satisfy the topologies of each database. Attempting to deallocate database(s) from a server that would result in less available servers than required fails with an error and no changes are made.

For example, if the cluster contains 5 servers and a database foo has a topology requiring 3 primaries and 2 secondaries, then it is not possible to deallocate any of the original 5 servers, without first enabling a 6th, or altering the desired topology of foo to require fewer servers overall.

The command can be used with DRYRUN to get a view of how the databases would be moved from the deallocated server(s).

neo4j@neo4j> DRYRUN DEALLOCATE DATABASES FROM SERVER '135ad202-5405-4d3c-9822-df39f59b823c';
+------------------------------------------------------------------------------------------------------------------------------------------+
| database | fromServerName | fromServerId                           | toServerName | toServerId                             | mode        |
+------------------------------------------------------------------------------------------------------------------------------------------+
| "db1"    | "server-3"     | "135ad202-5405-4d3c-9822-df39f59b823c" | "server-5"   | "00000003-b30a-434e-b9bf-1a5c8009773a" | "secondary" |
+------------------------------------------------------------------------------------------------------------------------------------------+

Deallocation is currently prevented in the following situations.

  • If a database the server is hosting is offline.

  • If the server is hosting a database with an allocation of 1 primary.

  • If a quorum of servers hosting the database in primary mode are cordoned.

Once the command has been executed, the server changes state to deallocating and it cannot readily be enabled again, see Dropped state for more information.

Dropping a server

Once DEALLOCATE DATABASES is executed for a server, its databases begin being moved. It is important not to attempt the next step before SHOW SERVERS reports that the deallocating server is in the (Deallocated state).

For example, do not drop the server 135ad202-5405-4d3c-9822-df39f59b823c given the following output:

neo4j@neo4j> SHOW SERVERS;
+------------------------------------------------------------------------------------------------------------------+
| name                                   | address          | state          | health      | hosting               |
+------------------------------------------------------------------------------------------------------------------+
| "135ad202-5405-4d3c-9822-df39f59b823c" | "localhost:7690" | "Deallocating" | "Available" | ["system", "foo"]     |
+------------------------------------------------------------------------------------------------------------------+

The deallocation process may take some time, as foo must be successfully copied and started on a new server before it is stopped on 135ad202-5405-4d3c-9822-df39f59b823c in order to preserve the availability and fault tolerance of foo.

Once SHOW SERVERS reflects that the server is deallocated and thus no longer hosts foo, the server may be dropped. Either the server ID or its name can be used:

neo4j@neo4j> DROP SERVER '135ad202-5405-4d3c-9822-df39f59b823c';

Once this command has been executed successfully, the neo4j process on the server in question may be stopped.

Controlling a server’s metadata

Altering server options

A running server can have its options modified using the ALTER SERVER 'name' SET OPTIONS { option: value } command. Either the ID or the name of the server can be used.

For example, to prevent a server from hosting databases in PRIMARY, execute the following:

neo4j@neo4j> ALTER SERVER '25a7efc7-d063-44b8-bdee-f23357f89f01' SET OPTIONS {modeConstraint:'SECONDARY'};

Altering servers may cause databases to be moved, and should be performed with care. For example, if the server 25a7efc7-d063-44b8-bdee-f23357f89f01 hosts database foo in primary mode when the above command is executed, then another server must begin hosting foo in primary mode.

Likewise, if ALTER SERVER '25a7efc7-d063-44b8-bdee-f23357f89f01' SET OPTIONS {allowedDatabases:['bar','baz']}; is executed, then foo is forced to move.

The possible options when altering a server are:

Option Allowed values Description

modeConstraint

PRIMARY, SECONDARY, NONE

Databases may only be hosted on the server in the mode specified by the constraint. None means there is no constraint and any mode is allowed.

allowedDatabases

list of database names, e.g. ["db1", "db2"]

Only databases matching the specified names may be hosted on the server. This may not be specified in combination with deniedDatabases.

deniedDatabases

list of database names, e.g. ["db1", "db2"]

Only databases not matching the specified names may be hosted on the server. This may not be specified in combination with allowedDatabases.

tags

list of server tags, e.g. ["tag1", "tag2"]

List of server tags used during database allocation and for load balancing and routing policies. Introduced in 5.6

allowedDatabases and deniedDatabases do not affect Composite databases, they are always available everywhere.

As with the DEALLOCATE DATABASES FROM SERVER …​ command, if the alteration of a server’s options renders it impossible for the cluster to satisfy one or more of the databases' topologies, then the command fails and no changes are made.

Input provided to SET OPTIONS {…​} replaces all existing options, rather than being combined with them. For instance if SET OPTIONS {modeConstraint:'SECONDARY'} is executed followed by SET OPTIONS {allowedDatabases:['foo']}, the execution of the second ALTER removes the mode constraint.

Renaming a server

When first discovered, a server’s name defaults to the value of its generated server ID. However, as long as the server is enabled, this can be changed later using the following command:

neo4j@neo4j> RENAME SERVER '25a7efc7-d063-44b8-bdee-f23357f89f01' TO 'eu-server-4';

This only affects the name of the server; the ID of the server remains fixed as 25a7efc7-d063-44b8-bdee-f23357f89f01. Keep in mind that the name of the server must be unique among existing servers.

Error handling

Occasionally, servers in a cluster may suffer issues such as network partitions or process crashes. These easiest way to observe these server failures is by executing SHOW SERVERS and checking for 'Unavailable' in the health column.

An Available health status does not indicate that a server is functioning perfectly, only that other servers in the cluster are able to make contact with it. For more in depth monitoring of cluster and server health, see Monitor servers.

If the issue with the Unavailable server proves permanent, then the server should be removed. However, if the issue is temporary then it likely is not desirable to remove these servers entirely as this causes all their hosted databases to be moved. Instead it is preferable to prevent those servers from being allocated any new databases to host, either as a result of databases being created or moved.

This is known as cordoning the server in question, and can be achieved by executing the following procedure against the system database:

neo4j@neo4j> CALL dbms.cluster.cordonServer('25a7efc7-d063-44b8-bdee-f23357f89f01');

SHOW SERVERS should then reflect that the server in question is now in Cordoned state.

Once the issue with the server has been resolved, the server can be returned to its previous Enabled state as follows:

neo4j@neo4j> ENABLE SERVER '25a7efc7-d063-44b8-bdee-f23357f89f01';

An unavailable server which has not been cordoned may still be allocated to host new databases. When the server recovers it observes that it is due to host these databases and begin catching up from some other available server (if one exists). However, in the meantime those databases have reduced fault tolerance or, worse, reduced availability. See Disaster Recovery for more details.