Android tombstone文件是如何生成的

本節內容我們聚焦到androidQ上，分析android中一個用於debug的功能，那就是tombstone，俗稱“墓碑”。現實生活中墓碑一般是給死人準備的，而在android系統中“墓碑”則是給進程準備的。

爲何Android要設計出這樣一個東西呢？ 因爲android系統是運行在Linux Kernel內核之上的，當內核出現異常，則內核異常機制會分辨出是什麼原因，處理不了的直接panic。而對於運行在Linux Kernel內核之上的android系統，如果出現異常，一般會自動重啓android層的，這就導致問題很難復現定位debug，則當android層出現異常，通常會將進程的上下文信息保存到tombstone（墓碑）中，方便後續的debug調試。

               

上圖是一張經典的android系統架構圖，而我們的墓碑主要是給Native 層的進程準備的，主要用於分析NativeCrash。因爲Kernel Crash整個系統直接就panic了，內核會打印出對應的call trace，對於Java層的代碼出錯也會有對應的異常拋出的。所以墓碑主要是給Native層的進程準備的。

Tombstone初識

tombstone到底長啥樣呢？ 當android系統出現異常時，會在/data/tombstones目錄生成對應的tombstone文件

root:/data/tombstones # ls -l
-rw-r----- 1 tombstoned system 3454991 2020-03-13 18:10 tombstone_00
-rw-rw-rw- 1 root       root         0 2020-03-14 10:28 tombstone_01
-rw-r----- 1 root       root   3454991 2020-03-14 10:29 tombstone_02
-rw-r----- 1 root       root   3454991 2020-03-14 10:29 tombstone_03

打開一個文件，看看tombstone到底長啥樣

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'dev/good_dev/2020.3.6_china_dev_test:user/test-keys'
Revision: '0'
ABI: 'arm'
Timestamp: 2020-03-07 02:46:27+0800
pid: 23051, tid: 23051, name: .tencent.qqlive  >>> com.tencent.qqlive <<<
uid: 10256
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xdb3fb000
    r0  db3f9000  r1  00000000  r2  00023780  r3  db3fb000
    r4  cd5ef340  r5  db3f9000  r6  e7a1e260  r7  cd62c6e0
    r8  cd668190  r9  ffc417a8  r10 00000001  r11 d1ffa760
    ip  00000000  sp  ffc41768  lr  b9466590  pc  e79bfd58

backtrace:
      #00 pc 0005dd58  /apex/com.android.runtime/lib/bionic/libc.so (memset_a7+48) (BuildId: dcf0e174e93e33d22f35a631ba9c0de5)
      #01 pc 0001258c  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (LogBuffer::__Clear()+36) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #02 pc 0000ff68  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (appender_open(TAppenderMode, char const*, char const*, char const*)+560) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #03 pc 00010ca8  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (appender_open_with_cache(TAppenderMode, std::string const&, std::string const&, char const*, char const*)+1652) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #04 pc 000051cc  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (Java_com_tencent_mars_xlog_Xlog_appenderOpen+428) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #05 pc 000db673  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/oat/arm/base.odex (art_jni_trampoline+210)

此文件比較長，我們目前只貼一部分，本節的內容不是分析tombstone內容的含義，本節重點分析此文件生成的過程，明白了是如何生成的，後續再分析此文件的內容是什麼含義，以及如何去分析解決此類問題。

NULL指針例子回顧

我們在上一節NULL指針的奇妙之旅中詳細講解了當CPU去訪問NULL指針，操作系統內部的一系列活動，最後在控制檯打印出耳熟能詳的"Segmetation Faule"。在這裏我們回顧下，因爲這個過程可以套用今天的tombstone的生產過程。

• 當CPU去訪問一個虛擬地址，肯定會經過MMU去查對應的虛實關係的
• 一旦虛擬地址是非法的，MMU硬件單元則會觸發異常，CPU則去異常向量表執行對應的異常
• 經過處理後Linux內核對userspace的異常則通過信號的方式通知給對應的進程
• 當進程一旦收到信號，則會執行對應的信號處理函數。
• 信號處理函數的安裝一般會在glibc中做的，glibc會對所有的通用信號做默認的處理的。

回到android系統中，當一個Native的進程觸發了NULL指針，首先CPU會收到對應異常，然後去執行異常，接着會通過發生SIGSEGV的信號，信號處理函數則會去處理信號，處理信號的過程中，則就會保存進程的現場，最後留下墓碑供後人膜拜。

通過上面的描述，我們大概已經推測出tombstone的大致實現流程了，接下來就去驗證猜想了。

 

進程是如何運行起來的

這裏簡單描述下android中一個進程是如何跑起來的。這裏以微信app爲例子說明

• 微信app首先是存儲在UFS，EMMC指令的存儲設備上
• 當用戶去點擊微信app圖標時，操作系統則會將微信app從Flash load到主存中
  • 肯定要去通過fork類似命令去創建對應的進程
  • 進程創建完畢需要通過exec類似的命令去加載微信的內容
• 最後由/system/bin/linker程序負責加載微信程序用到的一些共享庫，
• 最終跳轉到微信程序的入口處執行

以上就是一個簡單的描述一個程序時如何運行起來的，我們直接看下android系統中/system/bin/linker代碼

代碼路徑： /bionic/linker/arch/arm64/begin.S

ENTRY(_start)
  // Force unwinds to end in this function.
  .cfi_undefined x30

  mov x0, sp
  bl __linker_init

  /* linker init returns the _entry address in the main image */
  br x0
END(_start)

首先肯定是跳轉到linker的代碼段去執行，跳轉到__linker_init函數處

/*
 * This is the entry point for the linker, called from begin.S.
 */
extern "C" ElfW(Addr) __linker_init(void* raw_args) {

  tmp_linker_so.base = linker_addr;
  tmp_linker_so.size = phdr_table_get_load_size(phdr, elf_hdr->e_phnum);
  tmp_linker_so.load_bias = get_elf_exec_load_bias(elf_hdr);
  tmp_linker_so.dynamic = nullptr;
  tmp_linker_so.phdr = phdr;
  tmp_linker_so.phnum = elf_hdr->e_phnum;
  tmp_linker_so.set_linker_flag();

  return __linker_init_post_relocation(args, tmp_linker_so);
}

在Linker_init中會根據鏈接地址，以及elf的頭等信息，去重新計算是否需要重定位之類的

static ElfW(Addr) __attribute__((noinline))
__linker_init_post_relocation(KernelArgumentBlock& args, soinfo& tmp_linker_so) {

  // Initialize the linker's static libc's globals
  __libc_init_globals();

  // Initialize the linker's own global variables
  tmp_linker_so.call_constructors();

  ElfW(Addr) start_address = linker_main(args, exe_to_load);

  INFO("[ Jumping to _start (%p)... ]", reinterpret_cast<void*>(start_address));
}

通過linker_main函數返回可執行程序的開始地址，然後跳轉過去執行。我們重點看下linker_main函數

static ElfW(Addr) linker_main(KernelArgumentBlock& args, const char* exe_to_load) {
  ProtectedDataGuard guard;

  // Register the debuggerd signal handler.
#ifdef __ANDROID__
  debuggerd_callbacks_t callbacks = {
    .get_abort_message = []() {
      return __libc_shared_globals()->abort_msg;
    },
    .post_dump = &notify_gdb_of_libraries,
  };
  debuggerd_init(&callbacks);
#endif

重點關注這裏，當時android系統的話，則會去初始化debggerd_init，此函數中會按照信號的默認處理函數的

void debuggerd_init(debuggerd_callbacks_t* callbacks) {

  struct sigaction action;
  memset(&action, 0, sizeof(action));
  sigfillset(&action.sa_mask);
  action.sa_sigaction = debuggerd_signal_handler;
  action.sa_flags = SA_RESTART | SA_SIGINFO;

  // Use the alternate signal stack if available so we can catch stack overflows.
  action.sa_flags |= SA_ONSTACK;
  debuggerd_register_handlers(&action);
}

static void __attribute__((__unused__)) debuggerd_register_handlers(struct sigaction* action) {
  sigaction(SIGABRT, action, nullptr);
  sigaction(SIGBUS, action, nullptr);
  sigaction(SIGFPE, action, nullptr);
  sigaction(SIGILL, action, nullptr);
  sigaction(SIGSEGV, action, nullptr);
#if defined(SIGSTKFLT)
  sigaction(SIGSTKFLT, action, nullptr);
#endif
  sigaction(SIGSYS, action, nullptr);
  sigaction(SIGTRAP, action, nullptr);
  sigaction(DEBUGGER_SIGNAL, action, nullptr);
}

可以看到這裏註冊了一些異常信號，而信號的action處理函數是debuggerd_signal_handler。

當異常發生

比如當Native進程出現了null指針問題，則通過linux內核判斷會發生信號，最終信號由debuggerd_signal_handler函數處理

debuggerd_signal_handler

static void debuggerd_signal_handler(int signal_number, siginfo_t* info, void* context) {

  // Only allow one thread to handle a signal at a time.
  int ret = pthread_mutex_lock(&crash_mutex);
  if (ret != 0) {
    async_safe_format_log(ANDROID_LOG_INFO, "libc", "pthread_mutex_lock failed: %s", strerror(ret));
    return;
  }

  log_signal_summary(info);
}

使用pthread_mutex_lock防止同一時間多個線程同時來處理信號，後面則調用log_signal_summary函數來打印一些信息

  async_safe_format_log(ANDROID_LOG_FATAL, "libc",
                        "Fatal signal %d (%s), code %d (%s%s)%s in tid %d (%s), pid %d (%s)",
                        info->si_signo, get_signame(info), info->si_code, get_sigcode(info),
                        sender_desc, addr_desc, __gettid(), thread_name, self_pid, main_thread_name)

這裏這麼做的目的是防止後面動作出錯，最終不能確定是那個進程出錯的，此處先打印一些關鍵信息。可以從logcat中找到對應的信息

 libc    : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xdb3fb000 in tid 23051 (.tencent.qqlive), pid 23051 (.tencent.qqlive)

• 信號的num，比如信號11代表的是SIGSEGV
• 信號code，SEGV_MAPERR，就代表映射出錯了
• fault addr，出錯時的地址
• tid: 對應的線程ID
• pid: 對應的進程ID，如果一個進程中有好多線程，則每個線程的id是不一樣的。

  pid_t child_pid =
    clone(debuggerd_dispatch_pseudothread, pseudothread_stack,
          CLONE_THREAD | CLONE_SIGHAND | CLONE_VM | CLONE_CHILD_SETTID | CLONE_CHILD_CLEARTID,
          &thread_info, nullptr, nullptr, &thread_info.pseudothread_tid);
  if (child_pid == -1) {
    fatal_errno("failed to spawn debuggerd dispatch thread");
  }

  // Wait for the child to start...
  futex_wait(&thread_info.pseudothread_tid, -1);

  // and then wait for it to terminate.
  futex_wait(&thread_info.pseudothread_tid, child_pid);

接着是通過clone系統調用，clone出一個僞線程pseudothread線程，去dispatch處理信號，這裏則等待子進程的開始以及結束

debuggerd_dispatch_pseudothread

  pid_t crash_dump_pid = __fork();
  if (crash_dump_pid == -1) {
    async_safe_format_log(ANDROID_LOG_FATAL, "libc",
                          "failed to fork in debuggerd signal handler: %s", strerror(errno));
  } else if (crash_dump_pid == 0) {
   
    async_safe_format_buffer(main_tid, sizeof(main_tid), "%d", thread_info->crashing_tid);
    async_safe_format_buffer(pseudothread_tid, sizeof(pseudothread_tid), "%d",
                             thread_info->pseudothread_tid);
    async_safe_format_buffer(debuggerd_dump_type, sizeof(debuggerd_dump_type), "%d",
                             get_dump_type(thread_info));

    execle(CRASH_DUMP_PATH, CRASH_DUMP_NAME, main_tid, pseudothread_tid, debuggerd_dump_type,
           nullptr, nullptr);
    async_safe_format_log(ANDROID_LOG_FATAL, "libc", "failed to exec crash_dump helper: %s",
                          strerror(errno));
    return 1;
  }

在僞線程中通過fork去創建子線程，新創建的子線程中調動通過execle系統調用去執行crash_dump64程序，而父進程則在這裏等待crash_dump進程退出

crash_dump進程

  pid_t forkpid = fork();
  if (forkpid == -1) {
    PLOG(FATAL) << "fork failed";
  } else if (forkpid == 0) {
    fork_exit_read.reset();
  } else {
    // We need the pseudothread to live until we get around to verifying the vm pid against it.
    // The last thing it does is block on a waitpid on us, so wait until our child tells us to die.
    fork_exit_write.reset();
    char buf;
    TEMP_FAILURE_RETRY(read(fork_exit_read.get(), &buf, sizeof(buf)));
    _exit(0);
  }

crash_dump進程則直接通過fork出一個新進程，父進程通過read去等待子進程，而子進程在繼續執行crash_dump的任務

  // Get the process name (aka cmdline).
  std::string process_name = get_process_name(g_target_thread);

  // Collect the list of open files.
  OpenFilesList open_files;
  {
    ATRACE_NAME("open files");
    populate_open_files_list(&open_files, g_target_thread);
  }

• 獲取進程的name，通過/proc/PID/cmdline獲取進程的名字
• 獲取此進程總共打開了多個文件，通過/proc/PID/fd/就可以獲取此進程打開了多少個文件，每個文件都有一個文件描述符fd

{
    ATRACE_NAME("ptrace");
    for (pid_t thread : threads) {
      // Trace the pseudothread separately, so we can use different options.
      if (thread == pseudothread_tid) {
        continue;
      }

      if (!ptrace_seize_thread(target_proc_fd, thread, &error)) {
        bool fatal = thread == g_target_thread;
        LOG(fatal ? FATAL : WARNING) << error;
      }

      ThreadInfo info;
      info.pid = target_process;
      info.tid = thread;
      info.uid = getuid();
      info.process_name = process_name;
      info.thread_name = get_thread_name(thread);

      if (!ptrace_interrupt(thread, &info.signo)) {
        PLOG(WARNING) << "failed to ptrace interrupt thread " << thread;
        ptrace(PTRACE_DETACH, thread, 0, 0);
        continue;
      }

      if (thread == g_target_thread) {
        // Read the thread's registers along with the rest of the crash info out of the pipe.
        ReadCrashInfo(input_pipe, &siginfo, &info.registers, &abort_msg_address,
                      &fdsan_table_address);
        info.siginfo = &siginfo;
        info.signo = info.siginfo->si_signo;
      } else {
        info.registers.reset(unwindstack::Regs::RemoteGet(thread));
        if (!info.registers) {
          PLOG(WARNING) << "failed to fetch registers for thread " << thread;
          ptrace(PTRACE_DETACH, thread, 0, 0);
          continue;
        }
      }

      thread_info[thread] = std::move(info);
    }
  }

for循環遍歷這個進程中的所有線程，對每一個進程中的線程進程ptrace操作，對目標線程讀取其crashinfo。

  // Detach from all of our attached threads before resuming.
  for (const auto& [tid, thread] : thread_info) {
    int resume_signal = thread.signo == DEBUGGER_SIGNAL ? 0 : thread.signo;
    if (wait_for_gdb) {
      resume_signal = 0;
      if (tgkill(target_process, tid, SIGSTOP) != 0) {
        PLOG(WARNING) << "failed to send SIGSTOP to " << tid;
      }
    }

    LOG(DEBUG) << "detaching from thread " << tid;
    if (ptrace(PTRACE_DETACH, tid, 0, resume_signal) != 0) {
      PLOG(ERROR) << "failed to detach from thread " << tid;
    }
  }

讀取crashinfo完畢後，則對每個thead做detach操作

tombstoned_connect

  {
    ATRACE_NAME("tombstoned_connect");
    LOG(INFO) << "obtaining output fd from tombstoned, type: " << dump_type;
    g_tombstoned_connected =
        tombstoned_connect(g_target_thread, &g_tombstoned_socket, &g_output_fd, dump_type);
  }

連接到tombstone進程，通過socket連接的。

  // si_value is special when used with DEBUGGER_SIGNAL.
  //   0: dump tombstone
  //   1: dump backtrace
  if (!fatal_signal) {
    int si_val = siginfo.si_value.sival_int;
    if (si_val == 0) {
      backtrace = false;
    } else if (si_val == 1) {
      backtrace = true;
    } else {
      LOG(WARNING) << "unknown si_value value " << si_val;
    }
  }

根據tombstone傳遞的參數做不同的判斷，當參數=0時代表dump tombstone，等於1時，只dump backtrace

  if (backtrace) {
    ATRACE_NAME("dump_backtrace");
    dump_backtrace(std::move(g_output_fd), &unwinder, thread_info, g_target_thread);
  } else {
    {
      ATRACE_NAME("fdsan table dump");
      populate_fdsan_table(&open_files, unwinder.GetProcessMemory(), fdsan_table_address);
    }

    {
      ATRACE_NAME("engrave_tombstone");
      engrave_tombstone(std::move(g_output_fd), &unwinder, thread_info, g_target_thread,
                        abort_msg_address, &open_files, &amfd_data);
    }
  }

最終tombstone是通過engrave_tombstone來進程生成的。

engrave_tombstone

void engrave_tombstone(unique_fd output_fd, unwindstack::Unwinder* unwinder,
                       const std::map<pid_t, ThreadInfo>& threads, pid_t target_thread,
                       uint64_t abort_msg_address, OpenFilesList* open_files,
                       std::string* amfd_data) {
  // don't copy log messages to tombstone unless this is a dev device
  bool want_logs = android::base::GetBoolProperty("ro.debuggable", false);

  log_t log;
  log.current_tid = target_thread;
  log.crashed_tid = target_thread;
  log.tfd = output_fd.get();
  log.amfd_data = amfd_data;

  _LOG(&log, logtype::HEADER, "*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***\n");
  dump_header_info(&log);
  dump_timestamp(&log, time(nullptr));

  auto it = threads.find(target_thread);
  if (it == threads.end()) {
    LOG(FATAL) << "failed to find target thread";
  }
  dump_thread(&log, unwinder, it->second, abort_msg_address, true);

  if (want_logs) {
    dump_logs(&log, it->second.pid, 50);
  }

  for (auto& [tid, thread_info] : threads) {
    if (tid == target_thread) {
      continue;
    }

    dump_thread(&log, unwinder, thread_info, 0, false);
  }

  if (open_files) {
    _LOG(&log, logtype::OPEN_FILES, "\nopen files:\n");
    dump_open_files_list(&log, *open_files, "    ");
  }

  if (want_logs) {
    dump_logs(&log, it->second.pid, 0);
  }

tombstone文件實例分析

tombstone標誌性log開始： "*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***"

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

dump_header_info打印頭信息

static void dump_header_info(log_t* log) {
  auto fingerprint = GetProperty("ro.build.fingerprint", "unknown");
  auto revision = GetProperty("ro.revision", "unknown");

  _LOG(log, logtype::HEADER, "Build fingerprint: '%s'\n", fingerprint.c_str());
  _LOG(log, logtype::HEADER, "Revision: '%s'\n", revision.c_str());
  _LOG(log, logtype::HEADER, "ABI: '%s'\n", ABI_STRING);
}

示例如下：

Build fingerprint: 'dev/good_dev/2020.3.6_china_dev_test:user/test-keys'
Revision: '0'
ABI: 'arm'

dump_timestamp打印時間信息

static void dump_timestamp(log_t* log, time_t time) {
  struct tm tm;
  localtime_r(&time, &tm);

  char buf[strlen("1970-01-01 00:00:00+0830") + 1];
  strftime(buf, sizeof(buf), "%F %T%z", &tm);
  _LOG(log, logtype::HEADER, "Timestamp: %s\n", buf);
}

打印出差的時間，示例如下：

Timestamp: 2020-03-07 02:46:27+0800

dump_thread_info打印thread信息

static void dump_thread_info(log_t* log, const ThreadInfo& thread_info) {
  // Blacklist logd, logd.reader, logd.writer, logd.auditd, logd.control ...
  // TODO: Why is this controlled by thread name?
  if (thread_info.thread_name == "logd" ||
      android::base::StartsWith(thread_info.thread_name, "logd.")) {
    log->should_retrieve_logcat = false;
  }

  _LOG(log, logtype::HEADER, "pid: %d, tid: %d, name: %s  >>> %s <<<\n", thread_info.pid,
       thread_info.tid, thread_info.thread_name.c_str(), thread_info.process_name.c_str());
  _LOG(log, logtype::HEADER, "uid: %d\n", thread_info.uid);
}

示例如下：

pid: 23051, tid: 23051, name: .tencent.qqlive  >>> com.tencent.qqlive <<<
uid: 10256

dump_signal_info打印信號信息

  _LOG(log, logtype::HEADER, "signal %d (%s), code %d (%s%s), fault addr %s\n",
       thread_info.siginfo->si_signo, get_signame(thread_info.siginfo),
       thread_info.siginfo->si_code, get_sigcode(thread_info.siginfo), sender_desc, addr_desc);

示例如下：

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xdb3fb000

dump_probable_cause打印可能原因信息

      cause = StringPrintf("null pointer dereference");
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0ffc)) {
      cause = "call to kuser_helper_version";
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0fe0)) {
      cause = "call to kuser_get_tls";
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0fc0)) {
      cause = "call to kuser_cmpxchg";
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0fa0)) {
      cause = "call to kuser_memory_barrier";
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0f60)) {
      cause = "call to kuser_cmpxchg64";
    }

if (!cause.empty()) _LOG(log, logtype::HEADER, "Cause: %s\n", cause.c_str());

示例如下：

Cause: null pointer dereference

dump_registers打印寄存器信息

    r0  db3f9000  r1  00000000  r2  00023780  r3  db3fb000
    r4  cd5ef340  r5  db3f9000  r6  e7a1e260  r7  cd62c6e0
    r8  cd668190  r9  ffc417a8  r10 00000001  r11 d1ffa760
    ip  00000000  sp  ffc41768  lr  b9466590  pc  e79bfd58

log_backtrace打印backtrace的信息

backtrace:
      #00 pc 0005dd58  /apex/com.android.runtime/lib/bionic/libc.so (memset_a7+48) (BuildId: dcf0e174e93e33d22f35a631ba9c0de5)
      #01 pc 0001258c  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (LogBuffer::__Clear()+36) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #02 pc 0000ff68  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (appender_open(TAppenderMode, char const*, char const*, char const*)+560) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #03 pc 00010ca8  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (appender_open_with_cache(TAppenderMode, std::string const&, std::string const&, char const*, char const*)+1652) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #04 pc 000051cc  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (Java_com_tencent_mars_xlog_Xlog_appenderOpen+428) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #05 pc 000db673  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/oat/arm/base.odex (art_jni_trampoline+210)

dump_stack打印stack的信息

stack:
         ffc41728  cd422400  [anon:libc_malloc]
         ffc4172c  b946cc2b  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so
         ffc41730  ffc417b0  [stack]
         ffc41734  db3f9000  /data/data/com.tencent.qqlive/files/log/QQLiveLog.mmap2
         ffc41738  db3f9000  /data/data/com.tencent.qqlive/files/log/QQLiveLog.mmap2
         ffc4173c  e7a1e260  [anon:.bss]
         ffc41740  ffc417b0  [stack]
         ffc41744  00001252
         ffc41748  e7a1e260  [anon:.bss]
         ffc4174c  b946cc7b  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so
         ffc41750  00000000

dump_memory_and_code打印memory的信息

memory near r0 (/data/data/com.tencent.qqlive/files/log/QQLiveLog.mmap2):
    db3f8fe0 -------- -------- -------- --------  ................
    db3f8ff0 -------- -------- -------- --------  ................
    db3f9000 02000109 00096a02 00000000 00000000  .....j..........
    db3f9010 bd0aa200 5ef280e3 000000cd 00000000  .......^........
    db3f9020 00000000 00000000 00000000 c6e85800  .............X..
    db3f9030 00000012 00000000 00000000 00000000  ................
    db3f9040 00000000 d5ef2800 838b8a0c 74d2c701  .....(.........t
    db3f9050 05132305 73430323 ccd0cf20 cadcd4ca  .#..#.Cs .......
    db3f9060 1d2e0cd0 6c646d17 03a86a60 622dcd24  .....mdl`j..$.-b
    db3f9070 8c0c8da3 8c0d740c 15cc0d75 0c2c0db4  .....t..u.....,.
    db3f9080 8c0c140c accc4cac b963cc8c 00000000  .....L....c.....
    db3f9090 494affff c92851cd c8554dcc 48512d2f  ..JI.Q(..MU./-QH
    db3f90a0 2d49cccb 14ad7306 0000b80c ffff0000  ..I-.s..........
    db3f90b0 512d4f4a 2c4dcdc8 72180a80 00000001  JO-Q..M,...r....
    db3f90c0 75f2ffff 0f8e0a0c 52b1f20d 294fcdc8  ...u.......R..O)
    db3f90d0 f7d549cd e62a2c4d 00000002 8b02ffff  .I..M,*.........

dump_all_maps打印map的信息

memory map (2172 entries): (fault address prefixed with --->)
    0a125000-0a126fff r--         0      2000  /system/bin/app_process32 (BuildId: 4b3fcbf21f6cb09225ed60ef016d3f87)
    0a127000-0a129fff r-x      2000      3000  /system/bin/app_process32 (BuildId: 4b3fcbf21f6cb09225ed60ef016d3f87)
    0a12a000-0a12afff r--      5000      1000  /system/bin/app_process32 (BuildId: 4b3fcbf21f6cb09225ed60ef016d3f)
    0a12b000-0a12bfff rw-         0      1000
    12c00000-15fbffff rw-         0   33c0000  [anon:dalvik-main space (region space)]
    15fc0000-161bffff rw-         0    200000  [anon:dalvik-main space (region space)]

  dump_log_file(log, pid, "system", tail);打印system log的信息
  dump_log_file(log, pid, "main", tail);打印mainlog的信息

--------- tail end of log main
03-07 02:46:27.072 23051 23051 W System.err: 	at com.tencent.qqlive.ona.base.QQLiveApplication.attachBaseContext(QQLiveApplication.java:1223)
03-07 02:46:27.072 23051 23051 W System.err: 	at com.tencent.qqlive.ona.base.QQLiveApplicationWrapper.attachBaseContext(QQLiveApplicationWrapper.java:197)
03-07 02:46:27.073 23051 23051 W System.err: 	at android.app.Application.attach(Application.java:376)

總結

• 當Native進程發生了異常，比如NULL指針
• 操作系統會去異常向量表的地址去處理異常，然後發送信號
• 在debuggred_init註冊的信號處理函數就會收到處理
• 創建僞線程去啓動crash_dump進程，crash_dump則會獲取當前進程中各個線程的crash信息
• tombstoned進程是開機就啓動的，開機時註冊好了socket等待監聽
• 當在crash_dump中去連接tombstoned進程的時候，根據傳遞的dump_type類型會返回一個/data/tombstones/下文件描述符
• crash_dump進程後續通過engrave_tombstone函數將所有的線程的詳細信息寫入到tombstone文件中
• 則就在/data/tombstones下生成了此次對應的tombstone_XX文件