Writing this down quickly before I forget. When debugging a std::string from GNU libstdc++, the debugger typically won’t show you the actual representation. First, you need to turn off the pretty printer (assuming that it worked in the first place): (gdb) p reregisterSlaveMessage.resource_version_uuid_.ptr_ $13 = (std::string *) 0x7f4d98d5e970 (gdb) p *reregisterSlaveMessage.resource_version_uuid_.ptr_ $14 = "\022\020K|\n\225\064\246CE\222\350\275\315t", <incomplete sequence> (gdb) disable pretty-printer 2 printers disabled 0 of 2 printers enabled (gdb) p *reregisterSlaveMessage.resource_version_uuid_.ptr_ $15 = { static npos = <optimized out>, _M_dataplus = { <:allocator>> = { <:new_allocator>> = {<no data fields>}, <no data fields>}, members of std::basic_string<char std::char_traits>, std::allocator<char> >::_Alloc_hider: _M_p = 0x7f4d98d67068 "\022\020K|\n\225\064\246CE\222\350\275\315t", <incomplete sequence> } } Next, you need to know that the internal structure of std::string is prepended to the actual string data, so you need to cast and subtract from the data pointer to find the length and refcount.

I wanted to know who was removing the Mesos memory cgroups hierarchy and why, so I turned to SystemTap. Here’s my one-liner: sudo stap \ -d /usr/lib/systemd/libsystemd-shared-233.so \ -d /usr/lib64/libc-2.25.so \ -d /usr/lib/systemd/systemd \ -e 'probe kernel.function("sys_rmdir") { printf("%s(%s): %s\n", execname(), pp(), user_string($pathname)); print_ubacktrace(); }' Note that you have to feed in the binaries you expect to see in order to get user stack traces. The corresponding systemd stack trace was:

Verifying Traffic Server with AddressSanitizer is fairly straight-forward. On Linux, you need recent gcc or clang and the libasan library. On OS X, libasan wasn’t present, so I just switched to Linux ;) You should give --enable-asan to configure when you build. The build system will enable ASAN on all the parts that should have it. Then, whenever you run you will get ASAN checking memory state. LeakSanitizer reports leaks from an atexit(3) handler, so you need to ensure that the program exits rather than calls _exit(2) or dumps core.