Maybe libtool is not that bad

I’m considering taking back all the bad things I have said about libtool It turns out that by using libltdl it is possible to generate plugins that can be built a statically or as shared objects. I know it’s not too bad to implement that in a custom build, but as I understood more about libtool, this turns out to be relatively clean. This is the best introduction to using libltdl that I have found.
Read more...

Per-module logging with glog

I spent a few hours trying to implement per-module logging in the Mesos logging toggle. The Google logging library supports the --vmodule flags to toggle the logging level on a per-module basis, which looked promising. You can set this at run time using the SelVLOGLevel API. Unfortunately, the implementation of the VLOG_IS_ON macro is such that you need to set a per-module log level before the logging call site is hit for the first time, so this is clearly intended only for startup.
Read more...

What I learned about Linux AIO today

  1. There is no filesystem that implements the AIO fsync operation.
  2. When performing AIO reads on vboxfs (VirtualBox filesystem) it will return -EPROTO. No idea why that happens.
  3. Neither vboxfs nor tmpfs suport O_DIRECT.
  4. AIO seems to work as advertised on XFS, with or without O_DIRECT.
Read more...

Building the Mesos documentation

I made a minor change to the documentation and I wanted to test it, so I had to figure out how to build the Mesos documentation. I’m doing this in a CentOS 7 VM, but I guess something similar would work on different platforms. First, you need to know that Mesos uses Middleman to build the website from a set of markdown files. Next, you need to know that while the documentation itself is in the main Mesos repository, the Middleman configuration and site tooling is in a separate SVN repository.
Read more...

Building Mesos on OS X

So when you build Mesos on OS X, you have to use Homebrew to install a bunch of dependencies. I was momentarily stumped by the fact that linking the apr package with brew link --force seemed to not make the headers available. Then I realized that you are supposed to use apr-1-config to find the headers location.

Like this:

$ ./configure --prefix=/opt/mesos --with-apr=$(apr-1-config --prefix)
Read more...

Leak detection with tcmalloc

tcmalloc has a built-in leak detection mechanism. It took me a couple of tries to figure out how to work it, even after reading the documentation. At least on Centos 7, the trick is to make sure you install the pprof package as well as gperftools-libs package. You will also need to set the PPROF_PATH environment variable so that the tcmalloc runtime can find proof. If you don’t do this, then the leaks report will not resolve symbols, so the stack traces will not be that useful.
Read more...

Cannot extend ID. It is not part of highstate

I spent quite a while scratching my head over the following error message from Salt:

Cannot extend ID trafficserver in "base:trafficserver.collector". It is not part of the high state.

This actually means that you used a requisite clause like watch_in to inject a dependency into a state that Salt cannot resolve. I filed bug 7336.

Read more...

Creating multiple resources with Salt

I wanted to create a Salt Stack state that manages multiple directories. I figured that there was a way to do this, but could not see a good example in the documentation. Fortunately, the very helpful #salt IRC channel pointed me to the answer:

hierarchy:
   file.directory:
     - user: root
     - group: root
     - mode: 755
     - makedirs: True
     - names:
      - /var/lib/hierarchy
      - /var/lib/hierarchy/a
      - /var/lib/hierarchy/b
      - /var/lib/hierarchy/b/c
      - /var/lib/hierarchy/b/c/d
Read more...

Debugging iPXE from the iLO console

So I’ve been trying to get iPXE chainloading to work and I’ve been using the iLO virtual serial console over SSH to verify and debug. iPXE has a DCHP debug build option which you can enable by doing make bin/undionly.kpxe DEBUG=dhcp. However, when you do this, you will find that each line of output on the iLO virtual serial console output overwrites the previous line, creating a big illegible mess. Fortunately, you can build iPXE with only serial output support, so that you can actually read the debug messages on the iLO virtual serial console.
Read more...