Managing databases in a cluster
The number of both primary and secondary servers to host a database can be set when the database is created and altered after creation.
The command CREATE DATABASE
can be used to specify the initial topology and ALTER DATABASE
can be used to change the topology once the database is created.
If a database is no longer needed, the command DROP DATABASE
deletes the database from the cluster.
CREATE DATABASE
The command to create a database in a cluster is not significantly different from the command to create a database in a non-clustered environment (see Create, start, and stop databases for more information on database management on single servers).
The difference in a clustered environment is that the topology can be specified, i.e. how many primaries and secondaries are desired for the database.
To create a database foo
with 3 servers hosting the database in primary mode and 2 servers in secondary mode, the command looks like this:
CREATE DATABASE foo TOPOLOGY 3 PRIMARIES 2 SECONDARIES
The command can only be executed successfully if the cluster’s servers are able to satisfy the specified topology. If they are not, the command results in an error. For example, if the cluster’s servers are set up with mode constraints to contain two primaries and three secondaries, or if only four servers exist, the command fails with an error.
If |
A |
ALTER DATABASE
To alter the topology of or read/write access to a database after it has been created, use the command ALTER DATABASE
.
Alter database topology
To change the topology of the database foo
from the previous example, the command can look like this:
ALTER DATABASE foo SET TOPOLOGY 2 PRIMARIES 1 SECONDARY
Like the CREATE DATABASE
command, this command results in an error if the cluster does not contain sufficient servers to satisfy the requested topology.
Additionally, ALTER DATABASE
is optionally idempotent and also results in an error if the database does not exist.
It is possible to append the command with IF EXISTS
to make sure that no error is returned if the database does not exist.
When there is more than one possible permutation of the specified topology, Neo4j uses an allocator to decide how to spread the database across the cluster.
Note, like CREATE DATABASE
, the ALTER DATABASE
command allocates the database and there is no requirement to execute REALLOCATE DATABASES
unless there is a desire to re-balance databases across all servers that are part of the cluster.
This normally happens when the cluster is configured with more servers than the sum of the number of primaries and secondaries for any one database.
It is not possible to automatically transition to a topology with a single primary host. Attempting to do so results in an error.
However, it is possible to manually do this transition. The first step is to back up the database, see Backup and restore for more information. Once the database is backed up, the next step is to drop the database, see Delete databases for more information. The last step is to either seed a cluster from the backup with the new topology, or to restore the backup on a single server. See Seed a cluster further on for information on seeding.
Also, it is possible to automatically transition from a topology with a single primary host to multiple primary hosts. Keep in mind that during such a transition, the database will be unavailable for a short period of time.
ALTER DATABASE
commands are optionally idempotent, with the default behavior to fail with an error if the database does not exist.
Appending IF EXISTS
to the command ensures that no error is returned and nothing happens should the database not exist.
If the ALTER DATABASE
command decreases the number of allocations of a database, allocations on cordoned servers are removed first.
ALTER DATABASE nonExisting IF EXISTS SET TOPOLOGY 1 PRIMARY 0 SECONDARY
0 rows
Deallocate databases
To relieve the load of a specific server(s), you can use one of the following procedures to deallocate databases causing the pressure from the server(s):
You must have the |
For example, server01
hosts two small databases, foo
and bar
, and one very large database baz
, while other servers contain fewer or smaller databases, and server01
is under pressure.
You can use one of the following approaches to deallocate baz
from server01
or to deallocate a number of databases from server01
:
// With dry run
neo4j@system> CALL dbms.cluster.deallocateDatabaseFromServer("server01", "baz", true);
// Without dry run
neo4j@system> CALL dbms.cluster.deallocateDatabaseFromServer("server01", "baz");
// With dry run
neo4j@system> CALL dbms.cluster.deallocateDatabaseFromServers(["server01", "server02"], "baz", true);
// Without dry run
neo4j@system> CALL dbms.cluster.deallocateDatabaseFromServers(["server01", "server02"], "baz");
// With dry run
neo4j@system> CALL dbms.cluster.deallocateNumberOfDatabases("server01", 3, true);
// Without dry run
neo4j@system> CALL dbms.cluster.deallocateNumberOfDatabases("server01", 3);
Reallocate databases
To rebalance all database allocations across the cluster, for example, because you added new servers, use either procedures or Cypher commands to reallocate databases onto the new servers.
Reallocate databases using a procedure
You can use the procedure dbms.cluster.reallocateDatabase
to rebalance a specific database across the cluster, or dbms.cluster.reallocateNumberOfDatabases
to rebalance a number of database allocations across the cluster and relieve overloaded servers.
Note that if the cluster is already balanced, no reallocations will happen when running these procedures.
These procedures do not require a server name and can be executed with or without a dry run.
You must have the |
For example, you add three new servers and want to move a very large database, baz
, from all the servers containing it to the new servers.
// With dry run
neo4j@system> CALL dbms.cluster.reallocateDatabase("baz", true);
// Without dry run
neo4j@system> CALL dbms.cluster.reallocateDatabase("baz");
// With dry run
neo4j@system> CALL dbms.cluster.reallocateNumberOfDatabases(3, true);
// Without dry run
neo4j@system> CALL dbms.cluster.reallocateNumberOfDatabases(3);
Reallocate databases using a Cypher command
You can use the Cypher command REALLOCATE DATABASES
to rebalance all database allocations across the cluster and relieve overloaded servers.
This command can also be used with DRYRUN
to preview the new allocation of databases.
|
|
neo4j@neo4j> DRYRUN REALLOCATE DATABASES;
+----------------------------------------------------------------------------------------------------------------------------------------+
| database | fromServerName | fromServerId | toServerName | toServerId | mode |
+----------------------------------------------------------------------------------------------------------------------------------------+
| "bar" | "server-1" | "00000000-27e1-402b-be79-d28047a9418a" | "server-5" | "00000003-b76c-483f-b2ca-935a1a28f3db" | "primary" |
| "bar" | "server-3" | "00000001-7a21-4780-bb83-cee4726cb318" | "server-4" | "00000002-14b5-4d4c-ae62-56845797661a" | "primary" |
+----------------------------------------------------------------------------------------------------------------------------------------+
Recreate a database
Neo4j 5.24 introduces the dbms.cluster.recreateDatabase()
procedure, which allows you:
-
To change the database store to a specified backup, while keeping all the associated privileges for the database.
-
To make your database write-available again after it has been lost (for example, due to a disaster).
The recreate procedure works only for real user databases and not for composite databases, or the Remember that the recreate procedure results in downtime while the stores get updated. The time is unbounded and may depend on different factors — for example, the size of the store, network speed, etc. |
The database in question can be in an online
or offline
state when it is recreated, but a successful operation starts the database regardless of its previous state.
If your database has Change Data Capture (CDC) enabled, the CDC chain will stop when the database is recreated, even though CDC remains enabled in the recreated database. To restore CDC functionality, follow the guide on how to initialize CDC applications from an existing database.
Before recreating a database, any eventual quarantined states need to be addressed. For more information, see Standard databases → Error handling.
You need the CREATE DATABASE
and DROP DATABASE
privileges to run the recreate procedure.
To check if the recreation is successful, use the SHOW DATABASES
command and verify that all allocations have been started.
Additionally, you have the option to modify the topology during the recreation process. However, note that the store format, access, and enrichment cannot be altered during recreation.
Seeding options
The store to be used during the recreation of a database can be defined in different ways. One method uses a backup, while others use available allocations in the cluster.
You can use either seedURI
or seedingServers
to specify the source from which the database should be recreated.
-
If you define neither, an error is thrown.
-
If you define both of them, then
seedingServers
must be an empty list. See Undefined servers with fallback backup for more details. -
If
seedingServers
is not empty andseedURI
is also defined, an error will occur.
Use backup as a seed
If you provide a URI to a backup or a dump, the stores on all allocations will be replaced by the backup or the dump at the given URI.
The new allocations can be put on any ENABLED
server in the cluster.
See Seed from URI for more details.
CALL dbms.cluster.recreateDatabase("neo4j", {seedURI: "s3:/myBucket/myBackup.backup"});
Use available servers as a seed
After the recreation is complete, the database will have the latest data store from the seeding servers.
Recreation is based on remaining stores or a store defined by the user. This means that stores which were lost or not defined are not used for the recreation. If not used stores were more up to date than the used ones, this results in data loss. |
Specified servers
You can specify a set of available servers. The stores on all allocations will be synchronized to the most up-to-date store from the defined servers. The number of defined servers cannot exceed the number of total allocations in the desired topology.
CALL dbms.cluster.recreateDatabase("neo4j", {seedingServers: ["serverId1", "serverId2", "serverId3"]});
Undefined servers
If you provide an empty list of seeding servers and do not specify a seedURI
, Neo4j automatically selects all available allocations of the database as seeders.
The store will be replaced by the most up-to-date seeder available in the cluster.
CALL dbms.cluster.recreateDatabase("neo4j", {seedingServers: []});
Undefined servers with fallback backup
If both an empty list of seeding servers and a seedURI
are provided, Neo4j finds all available allocations of the database and use those as seeders.
However, if no available servers can be found, the database is recreated based on the backup or the dump defined by the URI.
This means the store is replaced by the most up-to-date seeder if available; otherwise, the backup is used.
CALL dbms.cluster.recreateDatabase("neo4j", {seedingServers: [], seedURI: "s3:/myBucket/myBackup.backup"});
Change the topology
There is an option to define a new topology when recreating a database. This can be beneficial during a disaster, if enough servers are not available to recreate the database with the original topology. When altering the total number of allocations down during a recreation, it is important to remember that the number of seeding servers cannot exceed the number of total allocations of the database. This also holds true when using recreate with an empty list of seeders. If there are more available servers in the cluster hosting the database than the number of new allocations, the recreation will fail.
CALL dbms.cluster.recreateDatabase("neo4j", {seedingServers: [], primaries: 3, secondaries: 0});
Seed a cluster
There are two different ways to seed a cluster with data. The first option is to use a designated seeder, where a designated server is used to create a backed-up database on other servers in the cluster. The other options is to seed the cluster from URI, where all servers to host a database are seeded with an identical seed from an external source specified by the URI. Keep in mind that using a designated seeder can be problematic in some situations as it is not known in advance how a database is going to be allocated to the servers in a cluster. Also, this method relies on the seed already existing on one of the servers.
Designated seeder
In order to designate a server in the cluster as a seeder, a database backup is transferred to that server using the neo4j-admin database restore
command.
Subsequently, that server is used as the source for other cluster members to copy the backed-up database from.
This example creates a user database called foo
, hosted on three servers in primary mode.
The foo
database should not previously exist on any of the servers in the cluster.
If a database with the same name as your backup already exists, use the command DROP DATABASE
to delete it and all users and roles associated with it.
-
Restore the
foo
database on one server. In this example, theserver01
member is used.bin/neo4j-admin database restore --from-path=/path/to/foo-backup-dir foo
-
Find the server ID of
server01
by logging in to Cypher Shell and runningSHOW SERVERS
. Cross-reference the address to find the server ID. Use any database to connect.SHOW SERVERS YIELD serverId, name, address, state, health, hosting;
+-----------------------------------------------------------------------------------------------------------------------------------------------------+ | serverId | name | address | state | health | hosting | +-----------------------------------------------------------------------------------------------------------------------------------------------------+ | "25a7efc7-d063-44b8-bdee-f23357f89f01" | "25a7efc7-d063-44b8-bdee-f23357f89f01" | "localhost:7689" | "Enabled" | "Available" | ["system", "neo4j"] | | "782f0ee2-5474-4250-b905-4cd8b8f586ba" | "782f0ee2-5474-4250-b905-4cd8b8f586ba" | "localhost:7688" | "Enabled" | "Available" | ["system", "neo4j"] | | "8512c9b9-d9e8-48e6-b037-b15b0004ca18" | "8512c9b9-d9e8-48e6-b037-b15b0004ca18" | "localhost:7687" | "Enabled" | "Available" | ["system", "neo4j"] | +-----------------------------------------------------------------------------------------------------------------------------------------------------+
In this case, the address for
server01
islocalhost:7687
and thus, the server ID is8512c9b9-d9e8-48e6-b037-b15b0004ca18
. -
On one of the servers, use the
system
database and create the databasefoo
using the server ID ofserver01
. The topology offoo
is stored in thesystem
database and when you create it, it is allocated according to the default topology (which can be shown withCALL dbms.showTopologyGraphConfig
). This may be different from the topology offoo
when it was backed up. If you want to ensure a certain allocation across the cluster, you can specify the desired topology with theTOPOLOGY
clause in theCREATE DATABASE
command. SeeCREATE DATABASE
for more information.CREATE DATABASE foo TOPOLOGY [desired number of primaries] PRIMARIES [desired number of secondaries] SECONDARIES OPTIONS {existingData: 'use', existingDataSeedServer: '8512c9b9-d9e8-48e6-b037-b15b0004ca18'};
-
Verify that the
foo
database is online on the desired number of servers, in the desired roles. If thefoo
database is of considerable size, the execution of the command can take some time.SHOW DATABASE foo;
+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | name | type | aliases | access | address | role | writer | requestedStatus | currentStatus | statusMessage | default | home | constituents | +------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | "foo" | "standard" | [] | "read-write" | "localhost:7687" | "primary" | FALSE | "online" | "online" | "" | FALSE | FALSE | [] | | "foo" | "standard" | [] | "read-write" | "localhost:7688" | "primary" | FALSE | "online" | "online" | "" | FALSE | FALSE | [] | | "foo" | "standard" | [] | "read-write" | "localhost:7689" | "primary" | TRUE | "online" | "online" | "" | FALSE | FALSE | [] | +------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 9 rows available after 3 ms, consumed after another 1 ms
Seed from URI
This method seeds all servers with an identical seed from an external source, specified by the URI. The seed can be either a backup or a dump from an existing database. The sources of seeds are called seed providers.
The mechanism is pluggable, allowing new sources of seeds to be supported (see Java Reference → Implement custom seed providers for more information). The product has built-in support for seed from a mounted file system (file), FTP server, HTTP/HTTPS server, Amazon S3, Google Cloud Storage (from Neo4j 5.25), and Azure Cloud Storage (from Neo4j 5.25).
Amazon S3, Google Cloud Storage, and Azure Cloud Storage are supported by default, but the other providers require configuration of |
The URI of the seed is specified when the CREATE DATABASE
command is issued:
CREATE DATABASE foo OPTIONS {existingData: 'use', seedURI:'s3://myBucket/myBackup.backup'}
Download and validation of the seed is only performed as the new database is started.
If it fails, the database is not available and it has the statusMessage
: Unable to start database
of the SHOW DATABASES
command.
neo4j@neo4j> SHOW DATABASES;
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| name | type | aliases | access | address | role | writer | requestedStatus | currentStatus | statusMessage | default | home | constituents |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| "seed3" | "standard" | [] | "read-write" | "localhost:7682" | "unknown" | FALSE | "online" | "offline" | "Unable to start database `DatabaseId{3fe1a59b[seed3]}`" | FALSE | FALSE | [] |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
To determine the cause of the problem, it is recommended to look at the debug.log
.
Seed providers
The URLConnectionSeedProvider
supports the following:
-
file:
-
ftp:
-
http:
-
https:
The S3SeedProvider
supports:
-
s3:
Neo4j 5 comes bundled with necessary libraries for AWS S3 connectivity.
Therefore, if you use |
The S3SeedProvider
requires additional configuration.
This is specified with the seedConfig
option.
This option expects a comma-separated list of configurations.
Each configuration value is specified as a name followed by =
and the value, as such:
CREATE DATABASE foo OPTIONS { existingData: 'use', seedURI: 's3:/myBucket/myBackup.backup', seedConfig: 'region=eu-west-1' }
S3SeedProvider
also requires passing in credentials.
These are specified with the seedCredentials
option.
Seed credentials are securely passed from the Cypher command to each server hosting the database.
For this to work, Neo4j on each server in the cluster must be configured with identical keystores.
This is identical to the configuration required by remote aliases, see Configuration of DBMS with remote database alias.
If this configuration is not performed, the seedCredentials
option fails.
CREATE DATABASE foo OPTIONS { existingData: 'use', seedURI: 's3:/myBucket/myBackup.backup', seedConfig: 'region=eu-west-1', seedCredentials: [accessKey];[secretKey] }
Where accessKey
and secretKey
are provided by AWS.
The CloudSeedProvider
supports:
-
s3:
-
gs:
-
azb:
Neo4j uses the AWS SDK v2 to call the APIs on AWS using AWS URLs.
Alternatively, you can override the endpoints so that the AWS SDK can communicate with alternative storage systems, such as Ceph, Minio, or LocalStack, using the system variables |
-
Install the AWS CLI by following the instructions in the AWS official documentation — Install the AWS CLI version 2.
-
Create an S3 bucket and a directory to store the backup files using the AWS CLI:
aws s3 mb --region=us-east-1 s3://myBucket aws s3api put-object --bucket myBucket --key myDirectory/
For more information on how to create a bucket and use the AWS CLI, see the AWS official documentation — Use Amazon S3 with the AWS CLI and Use high-level (s3) commands with the AWS CLI.
-
Verify that the
~/.aws/config
file is correct by running the following command:cat ~/.aws/config
The output should look like this:
[default] region=us-east-1
-
Configure the access to your AWS S3 bucket by setting the
aws_access_key_id
andaws_secret_access_key
in the~/.aws/credentials
file and, if needed, using a bucket policy. For example:-
Use
aws configure set aws_access_key_id aws_secret_access_key
command to set your IAM credentials from AWS and verify that the~/.aws/credentials
is correct:cat ~/.aws/credentials
The output should look like this:
[default] aws_access_key_id=this.is.secret aws_secret_access_key=this.is.super.secret
-
Additionally, you can use a resource-based policy to grant access permissions to your S3 bucket and the objects in it. Create a policy document with the following content and attach it to the bucket. Note that both resource entries are important to be able to download and upload files.
{ "Version": "2012-10-17", "Id": "Neo4jBackupAggregatePolicy", "Statement": [ { "Sid": "Neo4jBackupAggregateStatement", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": [ "arn:aws:s3:::myBucket/*", "arn:aws:s3:::myBucket" ] } ] }
-
-
Create database from
myBackup.backup
.CREATE DATABASE foo OPTIONS { existingData: 'use', seedURI: 's3:/myBucket/myBackup.backup' }
-
Ensure you have a Google account and a project created in the Google Cloud Platform (GCP).
-
Install the
gcloud
CLI by following the instructions in the Google official documentation — Install the gcloud CLI. -
Create a service account and a service account key using Google official documentation — Create service accounts and Creating and managing service account keys.
-
Download the JSON key file for the service account.
-
Set the
GOOGLE_APPLICATION_CREDENTIALS
andGOOGLE_CLOUD_PROJECT
environment variables to the path of the JSON key file and the project ID, respectively:export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json" export GOOGLE_CLOUD_PROJECT=YOUR_PROJECT_ID
-
Authenticate the
gcloud
CLI with the e-mail address of the service account you have created, the path to the JSON key file, and the project ID:gcloud auth activate-service-account service-account@example.com --key-file=$GOOGLE_APPLICATION_CREDENTIALS --project=$GOOGLE_CLOUD_PROJECT
For more information, see the Google official documentation — gcloud auth activate-service-account.
-
Create a bucket in the Google Cloud Storage using Google official documentation — Create buckets.
-
Verify that the bucket is created by running the following command:
gcloud storage ls
The output should list the created bucket.
-
-
Create database from
myBackup.backup
.CREATE DATABASE foo OPTIONS { existingData: 'use', seedURI: 'gs:/myBucket/myBackup.backup' }
-
Ensure you have an Azure account, an Azure storage account, and a blob container.
-
You can create a storage account using the Azure portal.
For more information, see the Azure official documentation on Create a storage account. -
Create a blob container in the Azure portal.
For more information, see the Azure official documentation on Quickstart: Upload, download, and list blobs with the Azure portal.
-
-
Install the Azure CLI by following the instructions in the Azure official documentation — Azure official documentation.
-
Authenticate the neo4j or neo4j-admin process against Azure using the default Azure credentials.
See the Azure official documentation on default Azure credentials for more information.az login
Then you should be ready to use Azure URLs in either neo4j or neo4j-admin.
-
To validate that you have access to the container with your login credentials, run the following commands:
# Upload a file: az storage blob upload --file someLocalFile --account-name accountName - --container someContainer --name remoteFileName --auth-mode login # Download the file az storage blob download --account-name accountName --container someContainer --name remoteFileName --file downloadedFile --auth-mode login # List container files az storage blob list --account-name someContainer --container someContainer --auth-mode login
-
Create database from
myBackup.backup
.CREATE DATABASE foo OPTIONS { existingData: 'use', seedURI: 'azb://myStorageAccount/myContainer/myBackup.backup' }
For example, in the case of S3SeedProvider
(the default provider), seedCredentials: [accessKey];[secretKey]
where accessKey
and secretKey
are provided by AWS.
Seed provider reference
URL scheme | Seed provider | URI example |
---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Controlling locations with allowed/denied databases
A database can by default be allocated to run on any server in a cluster.
However, it is possible to constrain the servers that specific databases are hosted on.
This is done with ENABLE SERVER
and ALTER SERVER
, described in Managing servers in a cluster.
The following options are available:
-
allowedDatabases
- a set of databases that are allowed to be hosted on a server. -
deniedDatabases
- a set of databases that are denied to be hosted on a server. Allowed and denied are mutually exclusive. -
modeConstraint
- controls in what mode (primary, secondary, or none) databases can be hosted on a server. If not set, there are no mode constraints on the server.
Change the default database
You can use the procedure dbms.setDefaultDatabase("newDefaultDatabaseName")
to change the default database for a DBMS.
-
Ensure that the database to be set as default exists, otherwise create it using the command
CREATE DATABASE <database-name>
. -
Show the name and status of the current default database by using the command
SHOW DEFAULT DATABASE
. -
Stop the current default database using the command
STOP DATABASE <database-name>
. -
Run
CALL dbms.setDefaultDatabase("newDefaultDatabaseName")
against thesystem
database to set the new default database. -
Optionally, you can start the previous default database as non-default by using
START DATABASE <database-name>
.
Be aware that the automatically created initial default database may have a different topology to the default configuration values. See Default database in a cluster for more information. |
Handling errors
Databases can get into error states.
Typically you can observe this with the SHOW DATABASES
command, and use the error handling guidance to help.
In more serious cases you may be dealing with a disaster situation, where the whole DBMS may not be responding correctly, or some specific databases cannot be restored without downtime. Refer to the disaster recovery guide for those situations.