Gitlab CE can be installed in a TrueNAS jail.
In addition to the redundancy provided by the NAS pools, the Gitlab server can also be backed up. This can be useful when we need to reinstall a new instance of Gitlab and wish to restore the repositories.
Creating a Backup
A backup of a git server can be created by running the command from the shell:
su -l git -c “cd /usr/local/www/gitlab-ce && RAILS_ENV=production NODE_ENV=production bundle exec rake gitlab:backup:create”
This creates a backup file in the /usr/local/www/gitlab-ce/tmp/backups/ folder,
with /usr/local/www/gitlab-ce/ being the installation folder,
and the relative folder path tmp/backups is set in config/gitlab.yml file.
The backup file name that is created has a format of:
xxxxxxxxxx_yyyy_mm_dd_VERSION_gitlab_backup.tar
where xxxxxxxxxx_yyyy_mm_dd is part of a <backup-id> defining when the backup was created,
VERSION indicates the git version. When restoring the backup, it can only be restored onto the same version as used to create the backup.
The current server version can be seen after logging in and clicking the help button.
For the full version, this can be seen by reading the contents of the files in the installation folder:
GITALY_SERVER_VERSION – such as 16.7.4
GITLAB_ELASTICSEARCH_INDEXER_VERSION – such as 4.5.0
GITLAB_KAS_VERSION – such as v16.8.0-rc1
GITLAB_PAGES_VERSION – such as 16.7.4
GITLAB_SHELL_VERSION – such as 14.32.0
GITLAB_WORKHORSE_VERSION – such as 16.7.4
Reading the Backup Location Path
The “/tmp/backups” part of the path comes from config/gitlab.yml file.
We can read this with help of a tool named yq, which is a tool to parse yaml files. Install it like this:
pkg install -q textproc/yq
Using a script to do this, we can change the current path to /usr/local/www/gitlab-ce/ and then calling yq, with the result being saved to a variable named backuppath:
backuppath=$(yq ‘.production.backup.path’ config/gitlab.yml)
This sets backuppaths to “tmp/backups” (with quotes). We need to remove the quotes:
backuppath=”${backuppath%\”}”
backuppath=”${backuppath#\”}”
backuppath now contains tmp/backups.
Get a Copy of the Backup
Get the filename of the latest backup file and save it to a variable:
lastbackupfile=$(ls -Art $backuppath | tail -n 1)
We can then copy this backup file to a different computer in case this one dies.
Create a folder named backups which is accessible by the TrueNAS root user, and copy the backup into this folder. External scripts can copy the file from this folder.
mkdir -p backups
cp $backuppath/lastfilebackup backups
Schedule a Recurring Backup
We can create a cron job on the Gitlab server to create the backup from the command line.
With the previous steps saved to a shell script named dobackup.sh (made executable with a call to chmod +x dobackup.sh), we can create the cronjob as follows:
Move the Backup to a New Physical Location
In case the Gitlab installation and file system become unrecoverable, its good to have the backup available on a different physical location.
The first step can be something like the following call. This will move the backup outside the Gitlab jail.
mv /mnt/Main2/iocage/jails/Gitlab/root/usr/local/www/gitlab-ce/backups/*.tar /mnt/Main2
This can also be schedules with a cron task through TrueNAS:
Backup Secrets.yml and Gitlab.rb
After running the backup on the Gitlab server, we are prompted that we should manually back up the gitlab-secrets.json and gitlab.rb files.
Reading the INSTALLATION_TYPE file shows it contains “source”. ie self-compiled from source code.
Back up GitLab | GitLab shows that for self compiled installations, the backup should be a minimum of:
config/secrets.yml
config/gitlab.yml
A copy of these files should also be saved.
It seems that the default comments created after a backup job is run are not applicable for self-hosted/self-compiled instances.
(more to come)