How do I recover from No space left on device
If one does not routinely monitor the disk space usage on a Neo4j server one may encounter a 'No space left on device' (for linux implementation) or a 'Low Disk Space' (Windows Implementations). For linux implementations one should proactively monitor disk space and this can be accomplished similar to what is described here.
Once you have encountered either, the following steps should be considered so as to free up enough space to start the database and allow recovery to complete. It should be noted that as you have consumed all the disk space, you cannot simply compress files to find more space, since that would require writing to the full file system.
-
graph.db path
DO NOT manually delete files in your database path, where the default location is$NEO4J_HOME/data/databases/graph.db
and also noted byconf/neo4j.conf
parameter of dbms.directories.data. Although this path may contain the most data, do not manually remove files from this path, as manually removing files will more than likely corrupt the database and/or prevent future starts.
However if you have another file system on the Neo4j installation which has more free space it is possible to either move the directory in whole to the other file system or commencing with Neo4j 3.4 one can configure dbms.directories.tx_log and this parameter describes where thedata/databases/graph.db/neostore.transaction*
files are recorded. For example if your current Neo4j installation is at/home/software/
, which is where the disk full occurred, and there is another filesystem, for example/home/disk2
which has a signficiant amount of space one can reconfigure the conf/neo4j.conf parameter ofdbms.directories.tx_log
to a value of/home/disk2/tx_logs
and copy the current and existingdata/databases/graph.db/neostore.transaction*
files to/home/disk2/tx_logs
so as to free up space on/home/software
. One could also do the same with regards to copying all ofdata/databases/graph.db
from/home/software
to/home/disk2
and in turn updateconf/neo4j.conf
parameter ofdbms.directories.data
.Under no circumstances should you delete/modify files in the path of graph.db and/or the location of transaction logs specified by configuration dbms.directories.tx_log
-
Local backup copies
Consider if you have written the results ofneo4j-admin backup
to the local file system. If so, and if that copy is not going to be necessary to restore service, can the files be moved/removed from the file system? -
Prior versions of Neo4j software
If you have completed multiple upgrades of Neo4j in the past, it may be the case that you have left the prior version software on the file system. For example if you typically install software into/usr/software/
and have upgraded from Neo4j 3.2.1 to Neo4j 3.5.0, you may have a/usr/software/neo4j-enterprise-3.2.1
and/usr/software/neo4j-enterprise-3.5.0
. If such an older environment exists and provided you are successfully running on the newer version, you may consider moving/removing the prior version, in this example/usr/software/neo4j-enterprise-3.2.1
-
Log files
During the normal course of Neo4j operation, diagnostic logs are written to $NEO4J_HOME/logs/ and specificallydebug.log
, 'neo4j.log',query.log
(provided dbms.logs.query.enabled=true) andsecurity.log
(provided dbms.security.auth_enabled=true), and where the debug.log can be the largest of these files. Given these files are diagnostic logs you may consider moving/removing/truncating these log files. -
Plugins Your
$NEO4J_HOME/plugins
may contain custom plugins (JARs) for Neo4j. Check to see that your plugins are not writing to this path, or if they are, ensure they are properly managing their log files. Additionally, you might consider removing/moving the apoc* jar as it can be easily restored. -
Prune transaction log files through the Neo4j product
If you have freed up a reasonable amount of space, prior to start you may want to configure theconf/neo4j.conf' parameter of dbms.tx_log.rotation.retention_policy to a very small value (example `dbms.tx_log.rotation.retention_policy=100M
). In doing so after a sucessful start and then subsequent checkpoint (which defaults to every 900s), a transaction log pruning/rotation will occur and it should remove all but the last transaction log. If you do not want to wait those 900s (15 minutes) and if you are running Neo4j 3.5.6 or later one can manually force a checkpoint by calling stored proceduredbms.checkpoint()
;
One issue with this approach is that more than likely upon your next backup, if it is an incremental backup, it will revert to a full backup since the transaction logs between the last backup and the next backup are not without a gap. If this approach is taken, after sufficient disk space has been freed up thedbms.tx_log.rotation.retention_policy
parameter should be reverted back to the value it was previously set to. -
Metrics Commencing with Neo4j 3.5.0 logging of metrics as CSV files is enabled by default and the .CSV files are recorded in the location as described by conf/neo4j.conf parameter of
dbms.directories.metrics
which defaults to $NEO4J_HOM]E/metrics. Additionally, conf/neo4j.conf parameters of metrics.csv.rotation.keep_number and metrics.csv.rotation.size describe the number of CSV to keep for a given metric as well as the rotation size for said metric. -
Prepare for growth
Even after freeing up sufficient space you will still need to prepare for the future in terms of making sure you have sufficient disk space for continued database growth. This is more an OS-related responsibility but should not be overlooked. -
Ask for guidance from Neo4j If you are still unable to free up space, ask for guidance from Neo4j.
After Neo4j has safely restarted it is strongly encouraged that you run a database consistency check. This can be performed
by either running a backup bin/neo4j-admin backup …. ….. --check-consistency=true
on a running instance or on a stopped
Neo4j database by running bin/neo4j-admin check-consistency --database=graph.db
Is this page helpful?