GitLab基础：新版本下使用gitlab-rake进行备份和恢复

骑猪看日落 2022-11-26 11:55 839阅读 0赞

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center_pic_center]

从GitLab12.1开始，可以直接使用rake进行备份，如果之前的项目中使用的gitlab-rake进行备份和恢复的方式是否可用这个观点进行了如下实验，通过本文进行简单记录。

# 备份源环境准备 #

容器化方式，直接使用docker-compose，启动如下服务

[root@host131 gitlab]# cat docker-compose.yml 
    version: '2'    
    services:
      # Version Control service: Gitlab
      gitlab:
        image: gitlab/gitlab-ce:12.10.5-ce.0
        ports:
          - "32001:80"
          - "30022:22"
          - "30443:443"
        volumes:
          - ./log/:/var/log/gitlab
          - ./data/:/var/opt/gitlab
          - ./conf/:/etc/gitlab
        restart: "no"
    [root@host131 gitlab]#

修改root用户，并创建如下项目  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center]

# 同步数据 #

使用rsync将各数据卷拷贝至备份目标机器，首先同步数据目录

[root@host131 ~]# rsync -r gitlab/data 192.168.163.132:/root/gitlab/data
    root@192.168.163.132's password: 
    skipping non-regular file "data/gitaly/gitaly.socket"
    skipping non-regular file "data/gitaly/internal_sockets/internal.sock"
    skipping non-regular file "data/gitaly/internal_sockets/ruby.0"
    skipping non-regular file "data/gitaly/internal_sockets/ruby.1"
    skipping non-regular file "data/gitlab-rails/sockets/gitlab.socket"
    skipping non-regular file "data/gitlab-workhorse/socket"
    skipping non-regular file "data/grafana/conf"
    skipping non-regular file "data/grafana/public"
    skipping non-regular file "data/nginx/logs"
    skipping non-regular file "data/postgresql/.s.PGSQL.5432"
    skipping non-regular file "data/redis/redis.socket"
    [root@host131 ~]#

注意事前需要在host132备份目标机器创建对应根目录

[root@host132 ~]# pwd
    /root
    [root@host132 ~]# ls
    anaconda-ks.cfg
    [root@host132 ~]# mkdir gitlab
    [root@host132 ~]# ls gitlab/
    [root@host132 ~]#

不然同步数据的时候会出现类似如下错误提示信息

[root@host131 ~]# rsync -r gitlab/data 192.168.163.132:/root/gitlab/data
    root@192.168.163.132's password: 
    rsync: mkdir "/root/gitlab/data" failed: No such file or directory (2)
    rsync error: error in file IO (code 11) at main.c(657) [Receiver=3.1.2]
    [root@host131 ~]#

*  同步配置数据

[root@host131 ~]# rsync -r gitlab/conf 192.168.163.132:/root/gitlab/conf
    root@192.168.163.132's password: 
    [root@host131 ~]#

*  同步日志数据

[root@host131 ~]# rsync -r gitlab/log 192.168.163.132:/root/gitlab/log
    root@192.168.163.132's password: 
    skipping non-regular file "log/gitlab-rails/sidekiq.log"
    [root@host131 ~]#

PS： 其实也不用这么费劲，直接打个打的tar扔过去就可以了

# 启动备份目标侧GitLab服务 #

使用docker-compose启动之后，界面显示如下，看起来和没有数据的方式一样  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center 1]

# 使用gitlab-rake备份数据 #

[root@host131 ~]# docker exec -it gitlab_gitlab_1 sh
    # gitlab-rake gitlab:backup:create
    2020-08-11 23:26:51 +0000 -- Dumping database ... 
    Dumping PostgreSQL database gitlabhq_production ... [DONE]
    2020-08-11 23:26:56 +0000 -- done
    2020-08-11 23:26:56 +0000 -- Dumping repositories ...
     * root/testproject (@hashed/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b) ... [SKIPPED]
    [SKIPPED] Wiki
    2020-08-11 23:26:57 +0000 -- done
    2020-08-11 23:26:57 +0000 -- Dumping uploads ... 
    2020-08-11 23:26:57 +0000 -- done
    2020-08-11 23:26:57 +0000 -- Dumping builds ... 
    2020-08-11 23:26:57 +0000 -- done
    2020-08-11 23:26:57 +0000 -- Dumping artifacts ... 
    2020-08-11 23:26:57 +0000 -- done
    2020-08-11 23:26:57 +0000 -- Dumping pages ... 
    2020-08-11 23:26:57 +0000 -- done
    2020-08-11 23:26:57 +0000 -- Dumping lfs objects ... 
    2020-08-11 23:26:57 +0000 -- done
    2020-08-11 23:26:57 +0000 -- Dumping container registry images ... 
    2020-08-11 23:26:57 +0000 -- [DISABLED]
    Creating backup archive: 1597188417_2020_08_11_12.10.5_gitlab_backup.tar ... done
    Uploading backup archive to remote storage  ... skipped
    Deleting tmp directories ... done
    done
    done
    done
    done
    done
    done
    done
    Deleting old backups ... skipping
    Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
    and are not included in this backup. You will need these files to restore a backup.
    Please back them up manually.
    Backup task is done.
    # 
    # cd /var/opt/gitlab/backups
    # ls
    1597188417_2020_08_11_12.10.5_gitlab_backup.tar
    # du -k *.tar
    172	1597188417_2020_08_11_12.10.5_gitlab_backup.tar
    #

可以看到数据较少，只有170K，而且提示了一个Warning信息，说一些包含敏感数据的文件gitlab.rb和gitlab-secrets.json文件不再备份之中，需要手动备份和恢复。

# 执行恢复 #

使用restore执行恢复，提示如下错误信息：

[root@host132 backups]# docker exec -it gitlab_gitlab_1 sh
    # cd /var/opt/gitlab/backups
    # ls
    1597188417_2020_08_11_12.10.5_gitlab_backup.tar
    # gitlab-rake gitlab:backup:restore BACKUP=1597188417
    The backup file 1597188417_gitlab_backup.tar does not exist!
    #

这是因为后续tar文件名的格式发生了变化，里面加入了GitLab的版本号，所以之前的命令早不到用于恢复的备份文件，修改文件名如下：

# cp 1597188417_2020_08_11_12.10.5_gitlab_backup.tar 1597188417_gitlab_backup.tar
    #

PS：更为简单的方式是使用如下命令

> 执行命令：gitlab-rake gitlab:backup:restore BACKUP=1597188417\_2020\_08\_11\_12.10.5

在恢复之前需要首先停止服务

# gitlab-ctl stop
    ok: down: alertmanager: 0s, normally up
    ok: down: gitaly: 0s, normally up
    ok: down: gitlab-exporter: 1s, normally up
    ok: down: gitlab-workhorse: 0s, normally up
    ok: down: grafana: 1s, normally up
    ok: down: logrotate: 0s, normally up
    ok: down: nginx: 1s, normally up
    ok: down: postgres-exporter: 0s, normally up
    ok: down: postgresql: 0s, normally up
    ok: down: prometheus: 1s, normally up
    ok: down: redis: 0s, normally up
    ok: down: redis-exporter: 1s, normally up
    ok: down: sidekiq: 0s, normally up
    ok: down: sshd: 0s, normally up
    ok: down: unicorn: 0s, normally up
    #

导入的时候需要注意权限是否正确否则容易出现如下错误

# gitlab-rake gitlab:backup:restore BACKUP=1597188417
    Unpacking backup ... tar: 1597188417_gitlab_backup.tar: Cannot open: Permission denied
    tar: Error is not recoverable: exiting now
    unpacking backup failed
    #

这是因为rsync同步时的文件权限受umask影响的缺省设定，修改即可

# ls -lrt
    total 344
    -rw------- 1 root root 174080 Aug 12 01:23 1597188417_2020_08_11_12.10.5_gitlab_backup.tar
    -rw------- 1 root root 174080 Aug 12 01:26 1597188417_gitlab_backup.tar
    # chmod 644 *
    #

执行备份恢复出现如下错误：

# gitlab-rake gitlab:backup:restore BACKUP=1597188417
    Unpacking backup ... done
    Before restoring the database, we will remove all existing
    tables to avoid future upgrade problems. Be aware that if you have
    custom tables in the GitLab database these tables and all data will be
    removed.
    
    Do you want to continue (yes/no)? yes
    Removing all tables. Press `Ctrl-C` within 5 seconds to abort
    2020-08-12 01:28:07 +0000 -- Cleaning the database ... 
    rake aborted!
    PG::ConnectionBad: could not connect to server: No such file or directory
    	Is the server running locally and accepting
    	connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:27:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:64:in `block (3 levels) in <top (required)>'
    /opt/gitlab/embedded/bin/bundle:23:in `load'
    /opt/gitlab/embedded/bin/bundle:23:in `<main>'
    Tasks: TOP => gitlab:db:drop_tables
    (See full trace by running task with --trace)
    #

提示信息显示无法连接数据库，理论上来说把PostgreSQL启动起来可以继续，这里干脆全部启动

# gitlab-ctl start
    ok: run: alertmanager: (pid 3009) 1s
    ok: run: gitaly: (pid 3017) 0s
    ok: run: gitlab-exporter: (pid 3030) 1s
    ok: run: gitlab-workhorse: (pid 3033) 0s
    ok: run: grafana: (pid 3042) 1s
    ok: run: logrotate: (pid 3050) 0s
    ok: run: nginx: (pid 3056) 0s
    ok: run: postgres-exporter: (pid 3062) 1s
    ok: run: postgresql: (pid 3078) 0s
    ok: run: prometheus: (pid 3088) 1s
    ok: run: redis: (pid 3096) 0s
    ok: run: redis-exporter: (pid 3101) 1s
    ok: run: sidekiq: (pid 3106) 0s
    ok: run: sshd: (pid 3112) 1s
    ok: run: unicorn: (pid 3114) 0s
    # 
    # gitlab-rake gitlab:backup:restore BACKUP=1597188417
    Unpacking backup ... done
    Before restoring the database, we will remove all existing
    tables to avoid future upgrade problems. Be aware that if you have
    custom tables in the GitLab database these tables and all data will be
    removed.
    
    Do you want to continue (yes/no)? yes
    Removing all tables. Press `Ctrl-C` within 5 seconds to abort
    2020-08-12 02:27:25 +0000 -- Cleaning the database ... 
    2020-08-12 02:27:34 +0000 -- done
    2020-08-12 02:27:34 +0000 -- Restoring database ... 
    Restoring PostgreSQL database gitlabhq_production ... SET
    SET
    SET
    SET
    SET
     set_config 
    ------------
     
    (1 row)
    
    SET
    SET
    SET
    SET
    ERROR:  relation "public.u2f_registrations" does not exist
    ERROR:  relation "public.timelogs" does not exist
    ...省略
    ALTER TABLE
    [DONE]
    2020-08-12 02:28:19 +0000 -- done
    2020-08-12 02:28:19 +0000 -- Restoring repositories ...
     * root/testproject ... [DONE]
    2020-08-12 02:28:20 +0000 -- done
    2020-08-12 02:28:20 +0000 -- Restoring uploads ... 
    2020-08-12 02:28:20 +0000 -- done
    2020-08-12 02:28:20 +0000 -- Restoring builds ... 
    2020-08-12 02:28:20 +0000 -- done
    2020-08-12 02:28:20 +0000 -- Restoring artifacts ... 
    2020-08-12 02:28:20 +0000 -- done
    2020-08-12 02:28:20 +0000 -- Restoring pages ... 
    2020-08-12 02:28:20 +0000 -- done
    2020-08-12 02:28:20 +0000 -- Restoring lfs objects ... 
    2020-08-12 02:28:20 +0000 -- done
    This task will now rebuild the authorized_keys file.
    You will lose any data stored in the authorized_keys file.
    Do you want to continue (yes/no)? yes
    
    Warning: Your gitlab.rb and gitlab-secrets.json files contain sensitive data 
    and are not included in this backup. You will need to restore these files manually.
    Restore task is done.
    #

整个过程中数据库数据导入最开始提示了relation不不存在的错误和需要手工处理gitlab.rb等文件的Warning，其余均显示正常完成。首先在目标备份机上的登录，可以看到已经不再提示需要进行密码修改了，输入备份侧的用户名和密码信息点击登录按钮  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center 2]  
可以看到刚刚在host131的GitLab服务上创建的工程项目信息也已经成功恢复了  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center 3]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center_pic_center]: /images/20221124/d13a277c47554a7cac0c8351c1b7226c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center]: /images/20221124/fd0e3b8475a14f648c21960e37ccdcd6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221124/41754a2d0d394bd081607a541ef85158.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221124/52f5f44ae7a846fdb7987e4a04b1115e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xpdW1pYW9jbg_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20221124/68c4773c0a13449995f3b9c40a52fd4a.png