core dump 问题

`bt` 显示 ??

gdb 版本过低

`thread apply all bt` 中完全无主程序的调试信息

1
2
3
Thread 1 (LWP 2528530):
#0  0x00007f36376b7fff in ?? ()
#1  0x0000000000000000 in ?? ()

gdb program core

考虑 program 二进制文件与 core 文件是否一致。

`thread apply all bt` 中存在主程序的调试信息，明显与主程序地址段不同的显示为 ??

动态链接库缺失

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
// dl.cpp
void dlthrow() {
  throw -1;
}

// main.cpp
void dlthrow();
int main(int argc, char* argv[]) {
  dlthrow();
  return 0;
}

1
2
g++ -shared -fPIC -o libdlthrow.so dl.cpp
g++ -L./ -ldlthrow main.cpp

1
gdb a.out -c core.a.out-2640879-1648111582

1
2
3
4
warning: Can't open file /xxx/yyy/zzz/libdlthrow.so during file-backed mapping note processing
[New LWP 2640879]

warning: Could not load shared library symbols for /xxx/yyy/zzz/libdlthrow.so.

可以通过 export LD_LIBRARY_PATH=/xxx/yyy/lib:$export LD_LIBRARY_PATH进行配置，或 gdb 控制台执行 set solib-search-path /xxx/yyy/zzz/。

`dlopen` 等方式加载的库缺失

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
// dl.cpp
extern "C"
void dlthrow() {
  throw -1;
}

// main.cpp
#include "dlfcn.h"
#include <iostream>

int main(int argc, char* argv[]) {
  void* handler = dlopen("libdlthrow.so", RTLD_LAZY);
  if (handler == nullptr) {
    std::cout << "handler is null" << std::endl;
    return -1;
  }
  (void)dlerror();
  auto dlthrow = reinterpret_cast<void (*)()>(dlsym(handler, "dlthrow"));
  if (dlerror() != nullptr) {
    std::cout << "dlthrow is null" << std::endl;
    return -1;
  }

  dlthrow();
  return 0;
}

1
2
g++ -shared -fPIC -o libdlthrow.so dl.cpp
g++ -ldl main.cpp

gdb 加载 core 文件时提示库缺失，栈显示如下：

1
2
3
4
5
6
7
8
9
#0  0x00007ff7304f9fff in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ff7304fb42a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ff730e120ad in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ff730e10066 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ff730e100b1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ff730e102c9 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ff7302c578b in ?? ()
#7  0x00007ffed5edd540 in ?? ()
#8  0x000055aa3ead8b0a in main ()

如 #6/#7 与 #8 地址段明显不同。

1
info sharedlibrary

1
2
3
4
5
6
7
8
From                To                  Syms Read   Shared Object Library
0x00007ff731103d80  0x00007ff73110494e  Yes (*)     /lib/x86_64-linux-gnu/libdl.so.2
0x00007ff730e0c7e0  0x00007ff730eb4d49  Yes (*)     /usr/lib/x86_64-linux-gnu/libstdc++.so.6
0x00007ff730a82680  0x00007ff730aee8da  Yes (*)     /lib/x86_64-linux-gnu/libm.so.6
0x00007ff730868a90  0x00007ff730878895  Yes (*)     /lib/x86_64-linux-gnu/libgcc_s.so.1
0x00007ff7304e6940  0x00007ff730610613  Yes (*)     /lib/x86_64-linux-gnu/libc.so.6
0x00007ff731307aa0  0x00007ff731322fb0  Yes (*)     /lib64/ld-linux-x86-64.so.2
                                        No          /xxx/yyy/zzz/libdlthrow.so

可以看到 /xxx/yyy/zzz/libdlthrow.so (有些 hack 的场景也可能看不到)。

但通过查看 mmap 映射区，可以通过地址精确的定位缺失的库为 libdlthrow.so (#6 0x00007ff7302c578b 对应地址段 [0x7ff7302c5000, 0x7ff7302c6000) /xxx/yyy/zzz/libdlthrow.so)。

1
info proc map

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Mapped address spaces:

          Start Addr           End Addr       Size     Offset objfile
      0x55aa3ead8000     0x55aa3ead9000     0x1000        0x0 /xxx/yyy/zzz/a.out
      0x55aa3ecd9000     0x55aa3ecda000     0x1000     0x1000 /xxx/yyy/zzz/a.out
      0x55aa3ecda000     0x55aa3ecdb000     0x1000     0x2000 /xxx/yyy/zzz/a.out
      0x7ff7302c5000     0x7ff7302c6000     0x1000        0x0 /xxx/yyy/zzz/libdlthrow.so
      0x7ff7302c6000     0x7ff7304c5000   0x1ff000     0x1000 /xxx/yyy/zzz/libdlthrow.so
      0x7ff7304c5000     0x7ff7304c6000     0x1000        0x0 /xxx/yyy/zzz/libdlthrow.so
      0x7ff7304c6000     0x7ff7304c7000     0x1000     0x1000 /xxx/yyy/zzz/libdlthrow.so
      0x7ff7304c7000     0x7ff73065c000   0x195000        0x0 /lib/x86_64-linux-gnu/libc-2.24.so
      0x7ff73065c000     0x7ff73085c000   0x200000   0x195000 /lib/x86_64-linux-gnu/libc-2.24.so
      0x7ff73085c000     0x7ff730860000     0x4000   0x195000 /lib/x86_64-linux-gnu/libc-2.24.so
      0x7ff730860000     0x7ff730862000     0x2000   0x199000 /lib/x86_64-linux-gnu/libc-2.24.so
      0x7ff730866000     0x7ff73087c000    0x16000        0x0 /lib/x86_64-linux-gnu/libgcc_s.so.1
      0x7ff73087c000     0x7ff730a7b000   0x1ff000    0x16000 /lib/x86_64-linux-gnu/libgcc_s.so.1
      0x7ff730a7b000     0x7ff730a7c000     0x1000    0x15000 /lib/x86_64-linux-gnu/libgcc_s.so.1
      0x7ff730a7c000     0x7ff730a7d000     0x1000    0x16000 /lib/x86_64-linux-gnu/libgcc_s.so.1
      0x7ff730a7d000     0x7ff730b80000   0x103000        0x0 /lib/x86_64-linux-gnu/libm-2.24.so
      0x7ff730b80000     0x7ff730d7f000   0x1ff000   0x103000 /lib/x86_64-linux-gnu/libm-2.24.so
      0x7ff730d7f000     0x7ff730d80000     0x1000   0x102000 /lib/x86_64-linux-gnu/libm-2.24.so
      0x7ff730d80000     0x7ff730d81000     0x1000   0x103000 /lib/x86_64-linux-gnu/libm-2.24.so
      0x7ff730d81000     0x7ff730ef3000   0x172000        0x0 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
      0x7ff730ef3000     0x7ff7310f3000   0x200000   0x172000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
      0x7ff7310f3000     0x7ff7310fd000     0xa000   0x172000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
      0x7ff7310fd000     0x7ff7310ff000     0x2000   0x17c000 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
      0x7ff731103000     0x7ff731106000     0x3000        0x0 /lib/x86_64-linux-gnu/libdl-2.24.so
      0x7ff731106000     0x7ff731305000   0x1ff000     0x3000 /lib/x86_64-linux-gnu/libdl-2.24.so
      0x7ff731305000     0x7ff731306000     0x1000     0x2000 /lib/x86_64-linux-gnu/libdl-2.24.so
      0x7ff731306000     0x7ff731307000     0x1000     0x3000 /lib/x86_64-linux-gnu/libdl-2.24.so
      0x7ff731307000     0x7ff73132a000    0x23000        0x0 /lib/x86_64-linux-gnu/ld-2.24.so
      0x7ff73152a000     0x7ff73152b000     0x1000    0x23000 /lib/x86_64-linux-gnu/ld-2.24.so
      0x7ff73152b000     0x7ff73152c000     0x1000    0x24000 /lib/x86_64-linux-gnu/ld-2.24.so

动态链接库存在部分函数名显示 ??

1
2
3
4
5
6
7
8
#0  0x00007f320d227fff in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f320d22942a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f320db400ad in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007f320db3e066 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007f320db3e0b1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007f320db3e2c9 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x0000555d9b9a47c8 in raiseException() ()
#7  0x0000555d9b9a47dc in main ()

#3 可能原因为栈被破坏。

干扰因素

C/C++ 语言添加编译选项 -g
-g 选项提供了行号信息，-g 选项带来的效果如下：
1
#7 0x000055e9c75f9b0a in main (argc=1, argv=0x7ffdd4124638) at main.cpp:17
不包含 -g 选项时：
1
#7 0x000055aa3ead8b0a in main ()
该选项对于 bt 显示 ?? 并无帮助，但可以使用诸如 list <function-name> 等 gdb 命令。
Go 语言添加编译选项 -gcflags "-N -l"
其中 -N 禁止优化，-l 禁止内联，对于 bt 显示 ?? 亦无帮助。
gdb 加载 Go 插件 runtime-gdb.py
1
source /usr/local/go/src/runtime/runtime-gdb.py
runtime-gdb.py 主要提供了 $len()、$cap()、$dtype() 以及 goroutine 相关的查看，对于 bt 显示 ?? 同样无帮助。
参考：Debugging Go Code with GDB

core dump gdb issues

Contents

core dump 问题

`bt` 显示 ??

gdb 版本过低

`thread apply all bt` 中完全无主程序的调试信息

`thread apply all bt` 中存在主程序的调试信息，明显与主程序地址段不同的显示为 ??

`dlopen` 等方式加载的库缺失

动态链接库存在部分函数名显示 ??

干扰因素

Contents

core dump 问题

bt 显示 ??

gdb 版本过低

thread apply all bt 中完全无主程序的调试信息

thread apply all bt 中存在主程序的调试信息，明显与主程序地址段不同的显示为 ??

dlopen 等方式加载的库缺失

动态链接库存在部分函数名显示 ??

干扰因素

`bt` 显示 ??

`thread apply all bt` 中完全无主程序的调试信息

`thread apply all bt` 中存在主程序的调试信息，明显与主程序地址段不同的显示为 ??

`dlopen` 等方式加载的库缺失