golang進(jìn)程在docker中OOM后hang住問(wèn)題解析
正文
golang版本:1.16
背景:golang進(jìn)程在docker中運(yùn)行,因?yàn)槭褂脙?nèi)存較多,經(jīng)常在內(nèi)存未達(dá)到docker上限時(shí),就被oom-kill,為了避免程序頻繁被殺,在docker啟動(dòng)時(shí)禁用了oom-kill,但是出現(xiàn)了新的問(wèn)題。
現(xiàn)象:docker內(nèi)存用滿(mǎn)后,golang進(jìn)程hang住,無(wú)任何響應(yīng)(沒(méi)有額外內(nèi)存系統(tǒng)無(wú)法分配新的fd,無(wú)法服務(wù)),即使在程序內(nèi)置了內(nèi)存達(dá)到上限就重啟,也不會(huì)生效,只能kill
因?yàn)閜prof查看進(jìn)程內(nèi)存有很多是能在gc時(shí)釋放的,起初懷疑是golang進(jìn)程問(wèn)題
在hang住之前,先登錄到docker上,寫(xiě)一個(gè)golang測(cè)試程序,只申請(qǐng)一小段內(nèi)存后sleep,啟動(dòng)時(shí)加GODEBUG=GCTRACE=1打印gc信息,發(fā)現(xiàn)mark 階段stw耗時(shí)達(dá)到31s(31823+15+0.11 ms對(duì)應(yīng)STW Mark Prepare,Concurrent Marking,STW Mark Termination)

懷疑是不是申請(qǐng)內(nèi)存失敗后,沒(méi)有觸發(fā)oom退出。在golang標(biāo)準(zhǔn)庫(kù)中查看oom相關(guān)的邏輯
mgcwork.go:374
if s == nil {
systemstack(func() {
s = mheap_.allocManual(workbufAlloc/pageSize, spanAllocWorkBuf)
})
if s == nil {
throw("out of memory")
}
// Record the new span in the busy list.
lock(&work.wbufSpans.lock)
work.wbufSpans.busy.insert(s)
unlock(&work.wbufSpans.lock)
}
mheap分配內(nèi)存使用了mmap,繼續(xù)懷疑是mmap返回的錯(cuò)誤碼在docker中不是非0
func sysMap(v unsafe.Pointer, n uintptr, sysStat *sysMemStat) {
sysStat.add(int64(n))
p, err := mmap(v, n, _PROT_READ| _PROT_WRITE, _MAP_ANON| _MAP_FIXED| _MAP_PRIVATE, -1, 0)
if err == _ENOMEM {
throw("runtime: out of memory")
}
if p != v || err != 0 {
throw("runtime: cannot map pages in arena address space")
}
}
為了對(duì)比驗(yàn)證,用c寫(xiě)一段調(diào)用mmap的代碼,在同一個(gè)docker中同時(shí)跑看下
#include <sys/mman.h>
#include <unistd.h>
#include <errno.h>
#include <stdio.h>
#define BUF_SIZE 393216
void main() {
char *addr;
int i;
for(i=0;i<1000000;i++) {
addr = (char *)mmap(NULL, BUF_SIZE, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
if (addr != MAP_FAILED) {
addr[0] = 'a';
addr[BUF_SIZE-1] = 'b';
printf("i:%d, sz: %d, addr[0]: %c, addr[-1]: %c\n", i, BUF_SIZE, addr[0], addr[BUF_SIZE-1]);
munmap(addr, BUF_SIZE);
} else {
printf("error no: %d\n", errno);
}
usleep(1000000);
}
}
mmap沒(méi)有失敗,而且同樣會(huì)hang住,說(shuō)明不是golang機(jī)制的問(wèn)題,應(yīng)該是阻塞在了系統(tǒng)調(diào)用上。查看調(diào)用堆棧,發(fā)現(xiàn)是hang在了cgroup中
[<ffffffff81224d65>] mem_cgroup_oom_synchronize+0x275/0x340 [<ffffffff811a068f>] pagefault_out_of_memory+0x2f/0x74 [<ffffffff81066bed>] __do_page_fault+0x4bd/0x4f0 [<ffffffff81801605>] async_page_fault+0x45/0x50 [<ffffffffffffffff>] 0xffffffffffffffff
查看go程序,也有相同的調(diào)用堆棧
[<ffffffff81103681>] futex_wait_queue_me+0xc1/0x120 [<ffffffff81104086>] futex_wait+0xf6/0x250 [<ffffffff8110647b>] do_futex+0x2fb/0xb20 [<ffffffff81106d1a>] SyS_futex+0x7a/0x170 [<ffffffff81003948>] do_syscall_64+0x68/0x100 [<ffffffff81800081>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [<ffffffffffffffff>] 0xffffffffffffffff [<ffffffff810f3ffe>] hrtimer_nanosleep+0xce/0x1e0 [<ffffffff810f419b>] SyS_nanosleep+0x8b/0xa0 [<ffffffff81003948>] do_syscall_64+0x68/0x100 [<ffffffff81800081>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [<ffffffffffffffff>] 0xffffffffffffffff [<ffffffff81224c5a>] mem_cgroup_oom_synchronize+0x16a/0x340 [<ffffffff811a068f>] pagefault_out_of_memory+0x2f/0x74 [<ffffffff81066bed>] __do_page_fault+0x4bd/0x4f0 [<ffffffff81801605>] async_page_fault+0x45/0x50 [<ffffffffffffffff>] 0xffffffffffffffff [<ffffffff81224c5a>] mem_cgroup_oom_synchronize+0x16a/0x340 [<ffffffff811a068f>] pagefault_out_of_memory+0x2f/0x74 [<ffffffff81066bed>] __do_page_fault+0x4bd/0x4f0 [<ffffffff81801605>] async_page_fault+0x45/0x50 [<ffffffffffffffff>] 0xffffffffffffffff [<ffffffff81224c5a>] mem_cgroup_oom_synchronize+0x16a/0x340 [<ffffffff811a068f>] pagefault_out_of_memory+0x2f/0x74 [<ffffffff81066bed>] __do_page_fault+0x4bd/0x4f0 [<ffffffff81801605>] async_page_fault+0x45/0x50 [<ffffffffffffffff>] 0xffffffffffffffff
看了下cgroup內(nèi)存控制的代碼,策略是沒(méi)有可用內(nèi)存并且未配置oom kill的程序,會(huì)鎖在一個(gè)等待隊(duì)列里,當(dāng)有可用內(nèi)存時(shí)再?gòu)年?duì)首喚醒。這個(gè)邏輯沒(méi)辦法通過(guò)配置或者其他方式繞過(guò)去。
elixir.bootlin.com/linux/v4.14…
/**
* mem_cgroup_oom_synchronize - complete memcg OOM handling
* @handle: actually kill/wait or just clean up the OOM state
*
* This has to be called at the end of a page fault if the memcg OOM
* handler was enabled.
*
* Memcg supports userspace OOM handling where failed allocations must
* sleep on a waitqueue until the userspace task resolves the
* situation. Sleeping directly in the charge context with all kinds
* of locks held is not a good idea, instead we remember an OOM state
* in the task and mem_cgroup_oom_synchronize() has to be called at
* the end of the page fault to complete the OOM handling.
*
* Returns %true if an ongoing memcg OOM situation was detected and
* completed, %false otherwise.
*/
bool mem_cgroup_oom_synchronize(bool handle)
{
struct mem_cgroup *memcg = current->memcg_in_oom;
struct oom_wait_info owait;
bool locked;
/* OOM is global, do not handle */
if (!memcg)
return false;
if (!handle)
goto cleanup;
owait.memcg = memcg;
owait.wait.flags = 0;
owait.wait.func = memcg_oom_wake_function;
owait.wait.private = current;
INIT_LIST_HEAD(&owait.wait.entry);
prepare_to_wait(&memcg_oom_waitq, &owait.wait, TASK_KILLABLE);
mem_cgroup_mark_under_oom(memcg);
locked = mem_cgroup_oom_trylock(memcg);
if (locked)
mem_cgroup_oom_notify(memcg);
if (locked && !memcg->oom_kill_disable) {
mem_cgroup_unmark_under_oom(memcg);
finish_wait(&memcg_oom_waitq, &owait.wait);
mem_cgroup_out_of_memory(memcg, current->memcg_oom_gfp_mask,
current->memcg_oom_order);
} else {
schedule();
mem_cgroup_unmark_under_oom(memcg);
finish_wait(&memcg_oom_waitq, &owait.wait);
}
if (locked) {
mem_cgroup_oom_unlock(memcg);
/*
* There is no guarantee that an OOM-lock contender
* sees the wakeups triggered by the OOM kill
* uncharges. Wake any sleepers explicitly.
*/
memcg_oom_recover(memcg);
}
cleanup:
current->memcg_in_oom = NULL;
css_put(&memcg->css);
return true;
}
結(jié)論:
docker內(nèi)存耗光后,golang在gc的mark階段,需要申請(qǐng)新的內(nèi)存記錄被標(biāo)記的對(duì)象時(shí),需要調(diào)用mmap,因?yàn)闆](méi)有可用內(nèi)存,就會(huì)被hang在cgroup中,gc無(wú)法完成也就無(wú)法釋放內(nèi)存,就會(huì)導(dǎo)致golang程序一直在stw階段,無(wú)法對(duì)外服務(wù),即使壓力下降也無(wú)法恢復(fù)。最好還是不要關(guān)閉docker的oom-kill
以上就是golang進(jìn)程在docker中OOM后hang住問(wèn)題解析的詳細(xì)內(nèi)容,更多關(guān)于golang進(jìn)程docker OOM hang的資料請(qǐng)關(guān)注腳本之家其它相關(guān)文章!
相關(guān)文章
詳解Golang time包中的time.Duration類(lèi)型
在日常開(kāi)發(fā)過(guò)程中,會(huì)頻繁遇到對(duì)時(shí)間進(jìn)行操作的場(chǎng)景,使用 Golang 中的 time 包可以很方便地實(shí)現(xiàn)對(duì)時(shí)間的相關(guān)操作,本文講解一下 time 包中的 time.Duration 類(lèi)型,需要的朋友可以參考下2023-07-07
Go中阻塞以及非阻塞操作實(shí)現(xiàn)(Goroutine和main Goroutine)
本文主要介紹了Go中阻塞以及非阻塞操作實(shí)現(xiàn)(Goroutine和main Goroutine),文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2024-05-05
Go語(yǔ)言開(kāi)發(fā)技巧必知的小細(xì)節(jié)提升效率
這篇文章主要介紹了Go語(yǔ)言開(kāi)發(fā)技巧必知的小細(xì)節(jié)提升效率,分享幾個(gè)你可能不知道的Go語(yǔ)言小細(xì)節(jié),希望能幫助大家更好地學(xué)習(xí)這門(mén)語(yǔ)言2024-01-01
Golang?Fasthttp選擇使用slice而非map?存儲(chǔ)請(qǐng)求數(shù)據(jù)原理探索
本文將從簡(jiǎn)單到復(fù)雜,逐步剖析為什么?Fasthttp?選擇使用?slice?而非?map,并通過(guò)代碼示例解釋這一選擇背后高性能的原因,有需要的朋友可以借鑒參考下,希望能夠有所幫助,祝大家多多進(jìn)步,早日升職加薪2024-02-02
Go 實(shí)現(xiàn)一次性打包各個(gè)平臺(tái)的可執(zhí)行程序
這篇文章主要介紹了Go 實(shí)現(xiàn)一次性打包各個(gè)平臺(tái)的可執(zhí)行程序,具有很好的參考價(jià)值,希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧2020-12-12
Golang中crypto/cipher加密標(biāo)準(zhǔn)庫(kù)全面指南
本文主要介紹了Golang中crypto/cipher加密標(biāo)準(zhǔn)庫(kù),包括對(duì)稱(chēng)加密、非對(duì)稱(chēng)加密以及使用流加密和塊加密算法,文中通過(guò)示例代碼介紹的非常詳細(xì),需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2024-02-02
Go中RPC遠(yuǎn)程過(guò)程調(diào)用的實(shí)現(xiàn)
本文主要介紹了Go中RPC遠(yuǎn)程過(guò)程調(diào)用的實(shí)現(xiàn),文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2023-07-07

