問題

發(fā)現(xiàn)git上傳不了，登錄服務(wù)器，table補齊命令都報錯，發(fā)現(xiàn)根目錄磁盤使用率100%

解決問題

問題排查：

1.使用排查docker使用空間

docker system df?

未見異常

2.清理系統(tǒng)緩存

sudo apt autoremove
rm -rf ~/.cache/thumbnails/*
sudo rm -rf ~/.local/share/Trash/*
....

等等一頓操作下來，還是100%,reboot

發(fā)現(xiàn)界面都進(jìn)不去了！

屮！

3.進(jìn)入終端

Ctrl+Alt+F2

登錄終端

4.查找大文件夾

sudo du -hsx /* | sort -rh | head -10

一層層找進(jìn)去，在/var/lib/docker/containers目錄發(fā)發(fā)現(xiàn)了一個日志文件*-json.log占了幾百個G

我屮艸芔茻?。。。?/p>

刪掉，重啟，系統(tǒng)啟動正常

but，docker gitlab還是起不來，一直starting

一直報錯…

[+] Running 2/1
 ? Network docker_gitlab_default  Created                                                                                                                                                                                              0.1s
 ? Container gitlab               Created                                                                                                                                                                                              0.1s
Attaching to gitlab
gitlab  | Thank you for using GitLab Docker Image!
gitlab  | Current version: gitlab-ce=11.1.4-ce.0
gitlab  | 
gitlab  | Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
gitlab  | And restart this container to reload settings.
gitlab  | To do it use docker exec:
gitlab  | 
gitlab  |   docker exec -it gitlab vim /etc/gitlab/gitlab.rb
gitlab  |   docker restart gitlab
gitlab  | 
gitlab  | For a comprehensive list of configuration options please see the Omnibus GitLab readme
gitlab  | https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
gitlab  | 
gitlab  | If this container fails to start due to permission problems try to fix it by executing:
gitlab  | 
gitlab  |   docker exec -it gitlab update-permissions
gitlab  |   docker restart gitlab
gitlab  | 

gitlab  | Preparing services...
gitlab  | Starting services...
gitlab  | Configuring GitLab package...
gitlab  | /opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
gitlab  | /opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system
gitlab  | JSON file with existing configuration not found inside /opt/gitlab/embedded/nodes.
gitlab  | Skipping config check.
gitlab  | gitlab preinstall: 
gitlab  | gitlab preinstall: This node does not appear to be running a database
gitlab  | gitlab preinstall: Skipping version check, if you think this is an error exit now
gitlab  | gitlab preinstall: 
gitlab  | Configuring GitLab...

使用docker exec -it gitlab update-permissions等操作，然并nuan~

找到最后持續(xù)輸出的打印如下：

gitlab  | ==> /var/log/gitlab/postgresql/current <==
gitlab  | 2023-02-16_07:24:22.32111 FATAL:  lock file "postmaster.pid" is empty
gitlab  | 2023-02-16_07:24:22.32112 HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.
gitlab  | 2023-02-16_07:24:23.32501 FATAL:  lock file "postmaster.pid" is empty
gitlab  | 2023-02-16_07:24:23.32502 HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.
gitlab  | 2023-02-16_07:24:24.32886 FATAL:  lock file "postmaster.pid" is empty
gitlab  | 2023-02-16_07:24:24.32887 HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.
gitlab  | 2023-02-16_07:24:25.33276 FATAL:  lock file "postmaster.pid" is empty
gitlab  | 2023-02-16_07:24:25.33278 HINT:  Either another server is starting, or the lock file is the remnant of a previous server startup crash.

進(jìn)入docker 容器，在根目錄find這個文件postmaster.pid,刪之…，重啟容器，一切正常！

切記

磁盤空間滿先不要不要亂刪東西?。?！

一勞永逸解決問題

更改docker日志全局設(shè)置

在主機修改docker配置問題，只對新建docker容器有效。

# 排查Docker容器日志輸出情況 ?docker ?inspect ?--format {{.LogPath}} ?容器名/容器ID ??
vim /etc/docker/daemon.json
{
? ?"registry-mirrors": [
? ? ? ?"https://mirror.ccs.tencentyun.com",
? ? ? ? "https://10odyesl.mirror.aliyuncs.com"
? ],
? "log-driver":"json-file",
? "log-opts": {"max-size":"300m", "max-file":"3"}
}
systemctl restart docker
systemctl daemon-reload?

更改容器配置（docker-compose）

修改 docker-compose.yml，增加如下配置。yml分級按兩個空格縮進(jìn)，可以使用在線yml格式校驗

? ? logging:
? ? ? driver: "json-file"
? ? ? options:
? ? ? ? max-size: "50m"
? ? ? ? max-file: "2

查看配置：docker inspect -f ‘{{.HostConfig.LogConfig}}’ 容器名xxx

總結(jié)

以上為個人經(jīng)驗，希望能給大家一個參考，也希望大家多多支持腳本之家。

最新評論