dxu's blog

Optimistic concurrency control in ATS

2017-04-09T17:12:48-07:00

Optimistic concurrency control in ATS Home

Optimistic concurrency control in ATS

Optimistic concurrency control is one of the tools ATS uses to create a high performance logging subsystem. For various reasons, mutexes were deemed to be too expensive to be used in the logging fast path. Instead, ATS uses an optimistic commit/rollback strategy to synchronize shared memory. The most typical use of the commit/rollback strategy in the logging subsystem is when the current LogBuffer is full and we need to allocate a new LogBuffer. Since multiple threads can be writing to the current LogBuffer at any given time, ATS needs a way to synchronize access to the current LogBuffer.

The pointer to the current LogBuffer is defined here:

volatile head_p m_log_buffer; // current work buffer

head_p is defined in lib/ts/ink_queue.h:

#if (defined(__i386__) || defined(__arm__) || defined(__mips__)) && (SIZEOF_VOIDP == 4)
  typedef int32_t version_type;
  typedef int64_t data_type;
#elif TS_HAS_128BIT_CAS
  typedef int64_t version_type;
  typedef __int128_t data_type;
#else
  typedef int64_t version_type;
  typedef int64_t data_type;
#endif

  struct {
    void *pointer;
    version_type version;
  } s;

  data_type data;
} head_p;

where s.pointer is a pointer we want to serialize access for and s.version is a counter to tell when head_p has been modified. The version, at least in the context of a LogBuffer, lets us know how many threads are currently holding a reference to s.pointer.

But why a union? The key insight here is that data_type and struct s are the same size. This means that we can do an atomic CAS on s by simply referring to head_p.data. This lets us avoid complicated bit fiddling while still being able to still do accesses like head_p.s.pointer. But wait, isn’t this undefined? As it turns out, according to the C++ spec, it is in fact

“undefined behavior to read from the member of the union that wasn’t most recently written.”

However, that sentence is quickly followed by

“Many compilers implement, as a non-standard language extension, the ability to read inactive members of a union.”

ATS is relying on non-standard language extensions, whoopee. That being said, ATS has been in use for the better part of two decades, so if I were you I wouldn’t start losing sleep over this just yet.

When we actually want to change the values held in head_p, we obey this pattern:

do {
  INK_QUEUE_LD(old_h, m_log_buffer);
  if (FREELIST_POINTER(old_h) != FREELIST_POINTER(h)) {
    ink_atomic_increment(&buffer->m_references, -1);

    // another thread should be taking care of creating a new
    // buffer, so delete new_buffer and try again
    delete new_buffer;
    break;
  }
} while (!write_pointer_version(&m_log_buffer, old_h, new_buffer, 0));

There’s a lot of macro magic going on here. To spare you the details, here’s a quick summary of what each macro does:

INK_QUEUE_LD(x, y): Atomic copy-by-value of y into x.

FREELIST_POINTER(x): Maps to x.s.pointer.

ink_atomic_increment(x, y): Atomically increments x by y.

write_pointer_version(a, b, c, d): Atomic CAS between a and b with the new value being a head_p with s.pointer = c and s.version = d.

The entire do-while loop of goodness guarantees that anything executed inside of the loop body is done so atomically. This is opportunistic because if another thread comes along and changes m_log_buffer right after we call INK_QUEUE_LD(), the CAS inside write_pointer_version(..) will catch the change and abort the write. The loop repeats until we succeed in atomically performing the actions inside the loop body.

At first this may seem like a better, more lightweight solution over locks, but it does come with certain drawbacks:

If the critical section is highly contested, then performance quickly degrades. Every failed transaction generates more work, and more work generates more failed transactions. On the other hand, mutexes will put the thread to sleep at the added cost of a context switch.
It is easy to create a time to check to time of use bug with this method. If we forget to wrap the expression in a do-while with the correct terminating condition, we expose ourselves to a TOCTTOU bug. As with the C++ language itself, this form of concurrency control gives the programmer a lot of power at the expense of naive safety.

Tracking down a race condition in the Apache Trafficserver logging subsystem

2017-04-22T09:44:10-07:00

Tracking down a race condition in the Apache Trafficserver logging subsystem Home

Tracking down a race condition in the Apache Trafficserver logging subsystem

TODO: make this post more coherent

Lately at work, I’ve been tracking down a log corruption issue inside of Trafficserver’s logging subsystem. The issue is that there’s often random data inserted into the middle of log statements. Specifically, these are access logs that a lot of Yahoo’s analytics are based off of, so this is kind of an important issue.

Now before we get into the details of the bug, there are a few things I have to say about the logging subsystem. I’ve not had too much experience with the rest of the code, but ATS is one of those old code bases where documentation is very sparse. In some cases, the only existing documentation is flat out wrong. Furthermore, when Yahoo open sourced this project about 8 years ago, the project lost all of the commit history. This means that git-blame will take us only as far back as the initial open source commit in 2009. This makes for a very unpleasant time tracking down the reason a particular section of code is the way it is.

Now back to the bug. To make things even more challenging, this corruption issue only occurs under high load. What I’m really trying to say is that this bug not only happens in production, but also in a very specific region of the United States. Fancy that. Needless to say, this isn’t something I can debug live, much less reproduce locally. For those of you that have experienced this pain before, you’re probably screaming “RACE CONDITION”. Unfortunately, you’re probably right. Seeing as I haven’t quite solved this bug yet, I’ll keep this post updated with all of the strategies and dead ends I come across.

The LogBuffer

There’s a lot of interesting design choices that go into making a high performance logging system. I’m not going to claim I’m some expert (yet), so I’ll just jot down the things I’ve noticed and/or surmised.

Heap allocations are expensive.
- Heap allocations are many times more expensive than stack allocations, so we want to be avoiding heap allocations as often as we can. This kind of incentive often leads to things like memory pools and buffer spaces.
Excessivily writing to disk is expensive.
- We don’t want to write each individual log line to disk immediately, since that incurs significant overhead. Instead, we want to buffer a bunch of log entries and flush them all at once. This is where the LogBuffer comes in.

The LogBuffer class is designed to provide a thread-safe mechanism to store log entries before they’re flushed. To reduce system call overhead, LogBuffers are designed to avoid heavy-weight mutexes in favor of using lightweight atomics built on top of compare-and-swap operations. When a caller wants to write into a LogBuffer, the caller “checks out” a segment of the buffer to write into. LogBuffer makes sure that no two callers are served overlapping segments. To illustrate this point, consider this diagram of a buffer:

          +--------------------------------+
          | thread_1's segment             |
          |--------------------------------|
          | thread_2's segment             |
          |                                |
          |                                |
          |--------------------------------|
          | thread_3's segment             |
          |                                |
          |                                |
          |                                |
          |--------------------------------|
          | thread_4's segment             |
          |--------------------------------|
          | <unused>                       |
          |                                |
          |                                |
          |                                |
          |                                |
          |                                |
          |                                |
          |                                |
          +--------------------------------+

In this manner, since no two threads are writing in the other’s segment, we avoid race conditions on the actual logging. This also makes LogBuffer’s critical section extremely small. In fact, the only time we need to enter a critical section is when we do the book keeping to keep track of which segments are checked out.

When a thread wants to write a log entry, it does so by calling the global Log::access(...) function. Log::access(...) in turn does a bunch of work eventually culminating in checking out a buffer segment from an active LogBuffer, serializing the log entry into the LogBuffer segment, and checking the LogBuffer segment back in. Between the inclusive checkout and checkin operations is where I currently believe the bug to be.

Lead 1

The first step I took to try and narrow down this bug was to run TSan on a local instance. TSan suggested that (1) could be an issue:

LogBuffer::LB_ResultCode
LogBuffer::checkout_write(size_t *write_offset, size_t write_size)
{
  ...

  do {
    new_s = old_s = m_state;  // (1)

    if (old_s.s.full) {       // (2)
      break;
    } else {
      ++new_s.s.num_writers;
      new_s.s.offset += actual_write_size;
      ++new_s.s.num_entries;

      ...

      // essentially cas(m_state, old_s, new_s)
      if (switch_state(old_s, new_s)) {   // (3)
        // we succeded in setting the new state
        break;
      }
    }
  } while (--retries);

  ...
}

After reasoning with this code for a bit, I realized that if m_state was changed by another thread between assignments of new_s and old_s, then this could very well explain the data corruption issue I was facing. The reason being is that another thread could have given away the last available segment of the current LogBuffer object and marked m_state as full. However, our current thread’s old_s still reports the state as being NOT full on (2). I let myself celebrate for a few minutes before realizing it couldn’t be this easy. Unfortunately I was right.

My previous assumtion that new_s and old_s could hold different values was wrong. If you decompile the expression new_s = old_s = m_state, you’ll see something like this:

        movl    -20(%rbp), %eax
        movl    %eax, -4(%rbp)
        movl    -4(%rbp), %eax
        movl    %eax, -8(%rbp)

where -4(%rbp) is old_s and -8(%rbp) is new_s. What this snippet of assembly is saying is essentially this:

        %eax = m_state
        old_s = %eax
        %eax = old_s
        new_s = %eax

Too bad for me, this means that old_s and new_s have to be the same value. Dead end :(. However, I’m not convinced the race condition isn’t in LogBuffer just yet.

Lead 2

After delving into the depths of the LogBuffer code, I felt safe in assuming there weren’t any bugs in that area of the codebase. So I took a step back and examined the LogObject code. The way LogObjects and LogBuffers interact is not unnatural. The general gist of it is that each LogObject instance represents one logical logging “object” in ATS. For example, if there was one access log in squid format and another as a named pipe, then ATS would be keeping track of two LogObjects. The details of where all the bits and bytes should go are encapsulated in the LogObject implementation. This is orthogonal to the LogBuffer because as we discussed earlier, the LogBuffer is a mechanism to prevent excessive disk flushes whereas the LogObject is a logical object.

I also recently wrote an architectural overview of the ATS logging subsystem. As of the time of this writing, the pull request is sitting in a review queue in the upstream project. Hopefully when it’s merged, I’ll remember to update it here.

Now to the details of the issue. Every LogObject holds on to at most one LogBuffer at any given time. The LogObject uses the LogBuffer to cache writes into the filesystem/network/OS. It is safe to assume that at some point every LogBuffer will get full. When said LogBuffer is full, we need to 1) flush the LogBuffer to some underlying structure and 2) allocate a new LogBuffer for the LogObject and do an atomic swap. The following code is used to swap the old LogBuffer with the freshly allocated LogBuffer:

 case LogBuffer::LB_FULL_NO_WRITERS:
      // no more room in current buffer, create a new one
      new_buffer = new LogBuffer(this, Log::config->log_buffer_size);

      // swap the new buffer for the old one
      INK_WRITE_MEMORY_BARRIER;
      head_p old_h;

      do {
        INK_QUEUE_LD(old_h, m_log_buffer);
        if (FREELIST_POINTER(old_h) != FREELIST_POINTER(h)) {
          ink_atomic_increment(&buffer->m_references, -1);

          // another thread should be taking care of creating a new
          // buffer, so delete new_buffer and try again
          delete new_buffer;
          break;
        }
      } while (!write_pointer_version(&m_log_buffer, old_h, new_buffer, 0));

     if (FREELIST_POINTER(old_h) == FREELIST_POINTER(h)) {
        ink_atomic_increment(&buffer->m_references, FREELIST_VERSION(old_h) - 1);

        int idx = m_buffer_manager_idx++ % m_flush_threads;
        Debug("log-logbuffer", "adding buffer %d to flush list after checkout", buffer->get_id());
        m_buffer_manager[idx].add_to_flush_queue(buffer);
        Log::preproc_notify[idx].signal();
        buffer = nullptr;
      }

      ...

If we look closely at line 417, we notice that we identify potential race conditions very strangely. So the current thread, thread 1, attemps to swap the old LogBuffer with the new LogBuffer. Suppose that another thread, thread 2, is also doing the same thing at the same time. Only one of these operations should succeed because otherwise we start leaking memory. So naturally, thread 1 needs a mechanism to detect when a LogBuffer has been pulled out from underneath it. On line 417, we see that we identify this pulling out by comparing pointers. It makes the implicit assumption that if the pointer value has changed, then the LogBuffer must have changed. While this logic is sound in the “forward direction” (ie if the pointer changes then the buffer has changed), the converse is not true. Suppose we free the old LogBuffer and then immediately allocate another LogBuffer. It is conceivable that the pointer will remain the same. That is to say, if the LogBuffer has changed, there is no guarantee that the pointer to the new LogBuffer will be any different than the old pointer. You might imagine this can cause a few issues. And while you’re probably not wrong, the truth is equally unappealing.

As it turns out, we always allocate a new LogBuffer before we free the old LogBuffer. On line 409, we allocate the new LogBuffer. Later on in line 432, we free the old LogBuffer. If these operations are sequential (and we have no reason to believe otherwise), then no race condition is possible. While this was an interesting examination into the LogBuffer swapping code, it was ultimately another dead end.

The end… for now?

It’s been about a week and a half into the investigation. I thought it might be good to get some fresh samples of the log corruption to see if anything new or novel has happened to further my debugging, so I asked for some new samples. As it turns out, the issue has stopped happening in production. What this means is that even if I found a potential fix, there would be no way to verify the effectiveness of the fix. That means if there’s not an obvious problem, then there isn’t an obvious solution.

So until this issue crops up again, work on this bug has to be put on hold.

Creating a basic Linux kernel development setup

2017-05-09T21:25:02-07:00

Creating a basic Linux kernel development setup Home

Creating a basic Linux kernel development setup

Linux kernel development is something I’ve been wanting to get into for a long time now. I’ve tried and failed to get started over the years, but I finally think I’m in a place where I can do some real kernel work. But before we can dive into kernel development, we have to have a working development setup.

This document will be a rough outline into

Compiling the kernel
Setting up a initramfs for the kernel to boot into
Booting into our setup

Compiling the kernel

Compiling the kernel is a relatively straightforward task. First, do as you would any other open source project and fork Linus’ kernel repo on Github. Alternatively, you could clone his kernel.org repo, but it doesn’t really matter.

cd into your cloned repo and run make defconfig; make x86_64_defconfig; make kvmconfig. This will generate a pretty good set of default configs (located in .config).

make defconfig generates a default .config
make x86_64_defconfig makes some x86 specific changes to the new .config
make kvmconfig enables some kvm specific kernel options that we will later take advantage of

Finally run make to compile the kernel. Hint: use make -jN, where N is the number of logical cores you have on your CPU for faster compilation.

Setting up initramfs

initramfs is the initial memory only filesystem the Linux kernel uses during boot. It is necessary to solve some of the boot time chicken-or-egg type problems that may arise. Read more from the link if curious. If we were to try and boot into a bare kernel, we would get a kernel panic about not being able to mount a rootfs. This is expected because of the previous reasons. Thus, we need to create our own initramfs.

There are a number of ways to create an initial ramdisk. We are going to roll our own busybox ramdisk. Although it’s not the easiest way, it’s rather transparent and somewhat easy to understand.

First, clone the busybox repo. Then, cd into the directory and run make defconfig. Sounds familiar, right? Next, run make menuconfig and make sure the static binary option is ticked. You may need to dig around in that interface, but it’s there. Finding it is left as an exercise to the reader. Finally, run make; make install. This will put all the symlinked busybox “binaries” into _install of the current directory.

As for the initramfs, create a folder for holding all the intermediate files. For this guide, we’ll assume it’s in ~/dev/initramfs.

mkdir -pv ~/dev/initramfs/x86-busybox
cd !$
mkdir -pv {bin,sbin,etc,proc,sys,usr/{bin,sbin}}
cp -av ~/dev/busybox/_install/* .

What this does is create a basic filesystem layout for our custom Linux kernel to boot into. Obviously some utilities that rely on a more complicated filesystem layout won’t work. Finding out which ones is another exercise left to the reader.

Now we need an init script. All Linux systems need an inital process to startup all the userspace fun, and this is no different. Plop this simple init script into ~/dev/initramfs:

#!/bin/sh

mount -t proc none /proc
mount -t sysfs none /sys

echo -e "\nBoot took $(cut -d' ' -f1 /proc/uptime) seconds\n"

exec /bin/sh

Afterwards, run chmod +x init to give the init script execution bits. After this, we’re ready to cpio the whole initramfs up. This cpio command will (g)zip up all of our files and folders up into something the kernel can work with:

find . -print0 \
    | cpio --null -ov --format=newc \
    | gzip -9 > ~/dev/initramfs/initramfs-busybox-x86.cpio.gz

Booting

Ah, for the moment of truth. Truth to be told, it’s a rather underwhelming command. But the results are exciting!

qemu-system-x86_64 \
    -kernel ~/dev/linux/arch/x86_64/boot/bzImage \
    -initrd ~/dev/initramfs/initramfs-busybox-x86.cpio.gz \
    -enable-kvm

Notice how we use those fancy kvm settings after all. The qemu command should pop open a small window with a small Linux environment running inside. All your classic userspace utilities should be working courtesy of busybox.

Addendums

I used a lot of resources on the internet to arrive at this process, but one resource stands out in particular.

systemd and sd_notify(3)

2017-11-13T07:59:14-08:00

systemd and sd_notify(3) Home

systemd and sd_notify(3)

A little bit of background first: I’ve been spending a lot of time recently mucking around with systemd and its process management features. The latest project I was working on involved implementing hot upgrades on a running server. This means that when used, clients of said server don’t experience any disruption other than a very small latency bubble. No connections are torn down and no sockets are closed client-side. Fancy right? I’ll save the details for how that was done in a later post. For now, we discuss onto the systemd side of the equation.

So implementing 0-downtime was fun and tricky, but the feature itself was largely contained in C++ land, far away from systemd. The systemd interaction with the server binary works a little something like this.

         Process 1                       Process 2
----------------------------|---------------------------------
systemctl start server

<serving clients>

                                   <want to upgrade server>
                                     ./server -upgrade

<hands over client conns>


                                   <receives client conns>
                                            (1)

<exits>
  (2)

Pretty straight forward right? Assuming that Process 1 is a systemd controlled process (meaning systemd started the process), as point (2), systemd is going to detect Process 1 exited, and will systemctl restart the process, negating the hot upgrade we worked so hard to implement. How do we prevent this?

As it turns out, systemd has a little know (to me) service type called Type=notify. This means that the process will link with systemd headers and explicitly tell systemd about the service state and optionally, gasp, the main PID of the running process, typically at point (1).

Proof of concept (aka show me the code)

server.c

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>

int main() {
    pid_t pid = fork();
    if (!pid) {
        printf("We're the new server process!\n");
        sleep(30);  // so we can examine systemd state
    } else if (pid == -1) {
        printf("Fork failed :(\n");
        perror("fork");
    } else {
        printf("We're the parent\n");
        exit(0);
    }
}

Build: cc server.c -o server

example_server.service

[Unit]
Description=Example server process

[Service]
Type=simple
ExecStart=/home/daniel/dev/dxuuu.xyz/examples/systemd-sdnotify/server
Restart=always

[Install]
WantedBy=multi-user.target

Running this service will cause example_server.service to flap. You can confirm that’s happening by checking systemctl status example_server:

● example_server.service - Example server process
   Loaded: loaded (/etc/systemd/system/example_server.service; disabled; vendor preset: disabled)
      Active: failed (Result: start-limit-hit) since Mon 2017-11-13 07:28:29 PST; 4s ago
     Process: 4723 ExecStart=/home/daniel/dev/dxuuu.xyz/examples/systemd-sdnotify/server (code=exited, status=0/SUCCESS)
   Main PID: 4723 (code=exited, status=0/SUCCESS)

Nov 13 07:28:29 maharaja systemd[1]: example_server.service: Service hold-off time over, scheduling restart.
Nov 13 07:28:29 maharaja systemd[1]: example_server.service: Scheduled restart job, restart counter is at 5.
Nov 13 07:28:29 maharaja systemd[1]: Stopped Example server process.
Nov 13 07:28:29 maharaja systemd[1]: example_server.service: Start request repeated too quickly.
Nov 13 07:28:29 maharaja systemd[1]: example_server.service: Failed with result 'start-limit-hit'.
Nov 13 07:28:29 maharaja systemd[1]: Failed to start Example server process.

However, if we use sd_notify(3), we get much better results.

server_sdnotify.c

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>

#include <systemd/sd-daemon.h>

int main() {
    pid_t pid = fork();
    if (!pid) {
        printf("We're the new server process!\n");

        // tell systemd we're ready
        sd_notify(0, "READY=1\n");
        sleep(30);  // so we can examine systemd state
    } else if (pid == -1) {
        printf("Fork failed :(\n");
        perror("fork");
    } else {
        printf("We're the parent\n");

        // tell systemd the child is the main process now
        sd_notifyf(0, "MAINPID=%lu",
                      (unsigned long) pid);
        exit(0);
    }
}

Build: cc server_sdnotify.c -o server_sdnotify -lsystemd

example_server_sdnotify.service

[Unit]
Description=Example server process with sd_notify

[Service]
Type=notify
ExecStart=/home/daniel/dev/dxuuu.xyz/examples/systemd-sdnotify/server_sdnotify
Restart=always
NotifyAccess=all

[Install]
WantedBy=multi-user.target

And now when you run systemctl start server_sdnotify, you’ll see everything works nicely:

● example_server_sdnotify.service - Example server process
   Loaded: loaded (/etc/systemd/system/example_server_sdnotify.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-11-13 07:40:08 PST; 3s ago
 Main PID: 11445 (server_sdnotify)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/example_server_sdnotify.service
           └─11445 /home/daniel/dev/dxuuu.xyz/examples/systemd-sdnotify/server_sdnotify

Nov 13 07:40:08 maharaja systemd[1]: Starting Example server process...
Nov 13 07:40:08 maharaja server_sdnotify[11444]: We're the parent
Nov 13 07:40:08 maharaja systemd[1]: example_server_sdnotify.service: Supervising process 11445 which is not our child. We'll most likely not notice when it exits.
Nov 13 07:40:08 maharaja systemd[1]: Started Example server process.

Note that the “Supervising process 11445 which is not our child” warning is a bit bogus. Since the parent dies after forking the child, the child now belongs to PID 1. As such, systemd can listen to SIGCHLD. If you read through the systemd source code, you can confirm this behavior is true.

std::vector and emplace_back

2018-03-30T18:58:11-07:00

std::vector and emplace_back Home

std::vector and emplace_back

There are some interesting bits about the C++ runtime. Consider this problem: you want to add an element to a vector of type std::vector<std::pair<int, Foo>> with as little overhead as possible. That means one construction – no extra moves, no extra copies, nothing.

For this problem, let us use this implementation of Foo:

class Foo {
public:
    Foo() {
        std::cout << "constructor\n";
    }

    Foo(int x) {
        std::cout << "constructor2\n";
    }

    Foo(const Foo& f) {
        std::cout << "copy constructor\n";
    }

    Foo(Foo&& f) {
        std::cout << "move constructor\n";
    }
};

Obviously we want to use emplace_back to elide the move you would traditionally get from something like vec.push_back(std::move(f)).

int main()
{
  std::vector<std::pair<int, Foo>> list;
  list.emplace_back(3, Foo{});
}

When run:

$ ./a.out
constructor
move constructor

So what happened? Clearly Foo was std::moved at least once. In fact, it happens when emplace_back is evaluated, since we constructed an rvalue Foo{}.

Ok, so we need a different approach. How about forwarding through the arguments to Foo, so Foo is constructed as late as possible?

#include <tuple>

int main()
{
  std::vector<std::pair<int, Foo>> list;
  list.emplace_back(std::piecewise_construct, std::forward_as_tuple(3), std::forward_as_tuple());
}

$ ./a.out
constructor

Here we use std::forward_as_tuple to tell the compiler that we want to call the 0-arg constructor. If using fancy C++ standard library features isn’t your cup of tea, you could alternatively use a dummy constructor:

int main()
{
  std::vector<std::pair<int, Foo>> list;
  list.emplace_back(3, 1);
}

$ ./a.out
constructor2

Great! So we solved our original problem. Or did we? Consider this:

int main()
{
  std::vector<std::pair<int, Foo>> list;
  list.emplace_back(3, 1);
  std::cout << "---\n";
  list.emplace_back(3, Foo{});
  std::cout << "---\n";
  list.emplace_back(3, 1);
}

$ ./a.out
constructor2
---
constructor
move constructor
copy constructor
---
constructor2
copy constructor
copy constructor

Woah, what’s with all the extra copies? Well, as it turns out, when a vector needs to be resized, enough contiguous memory for all elements needs to be allocated. Then, the contents of the old vector need to be move or copy constructed into the new memory. To elide this issue, we can preallocate memory for our vector.

int main()
{
  std::vector<std::pair<int, Foo>> list;
  list.reserve(3);
  list.emplace_back(3, 1);
  std::cout << "---\n";
  list.emplace_back(3, Foo{});
  std::cout << "---\n";
  list.emplace_back(3, 1);
}

$ ./a.out
constructor2
---
constructor
move constructor
---
constructor2

Hey, now this is Looking good. But wait! There’s one outstanding question: what’s with all the copies in the previous snippet? Shouldn’t they be moves? We have a move constructor defined for Foo.

Well, it turns out due to exception-safe guarantees, C++ standard collections will not use non-exception-safe move constructors. So how do we fix this? Add noexcept to the move constructor, like so:

Foo(Foo&& f) noexcept {
  std::cout << "move constructor\n";
}

Then run the example without the list.reserve(3) again:

$ ./a.out
constructor2
---
constructor
move constructor
move constructor
---
constructor2
move constructor
move constructor

Congratulations, now you know more about std::vector than you wanted to know.

Comparing interned strings

2019-06-19T17:41:08-07:00

Comparing interned strings Home

Comparing interned strings

C/C++ has this interesting property where if you define a string literal, the compiler will make sure to only keep one copy of it. This makes sense as it reduces how much static storage is required.

For example, if you have:

const char* s = "asdf";
const char* s2 = "asdf";

the compiler will make sure “asdf” is stored once in the resulting binary.

This opens up an interesting property where you can compare string pointers as if you were calling strcmp. For example:

#include <iostream>

constexpr auto s1 = "string one";
constexpr auto s2 = "string two";
constexpr auto s1_copy = "string one";

int main() {
  if (s1 == s2) {
    std::cout << "s1 == s2" << std::endl;
  } else {
    std::cout << "s1 != s2" << std::endl;
  }

  if (s1 == s1_copy) {
    std::cout << "s1 == s1_copy" << std::endl;
  } else {
    std::cout << "s1 != s1_copy" << std::endl;
  }

  if (s1 == "string one") {
    std::cout << "s1 == string one" << std::endl;
  } else {
    std::cout << "s1 != string one" << std::endl;
  }

  return 0;
}

outputs:

$ ./interned_string
s1 != s2
s1 == s1_copy
s1 == string one

Note

According to cppreference:

The compiler is allowed, but not required, to combine storage for equal or overlapping string literals. That means that identical string literals may or may not compare equal when compared by pointer.

That being said, no sane compiler would omit this.

/proc/[pid] and the missing threads

2020-04-23T13:25:15-07:00

/proc/[pid] and the missing threads Home

`/proc/[pid]` and the missing threads

Suppose you run the following program:

use std::time;
use std::thread;

fn main() {
    println!("My TID: {}", nix::unistd::gettid());

    let child = thread::spawn(move || {
        println!("My thread's TID: {}", nix::unistd::gettid());
        loop {
            thread::sleep(time::Duration::from_secs(1));
        }
    });

    child.join().unwrap();
}

and get the following output:

My TID: 840680
My thread's TID: 840695

Inspecting thread state through /proc works as expected:

$ cat /proc/840680/comm
thread_id

$ cat /proc/840695/comm
thread_id

However, if you happened to browse through /proc/ (via ls or other), you’ll notice a strange inconsistency:

$ ls -l /proc | grep 840680 &> /dev/null; echo $?
0

$ ls -l /proc | grep 840695 &> /dev/null; echo $?
1

In other words, there’s no directory entry for the thread.

Why is this the case? We have to look at the kernel code to find out. First let’s look at where all the entries in /proc are instantiated. Remember that /proc, or procfs, is a virtual file system so there’s not actually anything on disk backing the fileystem. Everything is generated when we request it.

In fs/proc/root.c:

/*
 * This is the root "inode" in the /proc tree..
 */
struct proc_dir_entry proc_root = {
        .low_ino        = PROC_ROOT_INO,
        .namelen        = 5,
        .mode           = S_IFDIR | S_IRUGO | S_IXUGO,
        .nlink          = 2,
        .refcnt         = REFCOUNT_INIT(1),
        .proc_iops      = &proc_root_inode_operations,
        .proc_dir_ops   = &proc_root_operations,
        .parent         = &proc_root,
        .subdir         = RB_ROOT,
        .name           = "/proc",
};

&proc_root_operations seems like a likely suspect for directory operations:

/*
 * The root /proc directory is special, as it has the
 * <pid> directories. Thus we don't use the generic
 * directory handling functions for that..
 */
static const struct file_operations proc_root_operations = {
        .read            = generic_read_dir,
        .iterate_shared  = proc_root_readdir,
        .llseek         = generic_file_llseek,
};

So far the comments confirm our understanding. However, it’s somewhat unclear which callback is called when we run ls versus directly cat a file. Let’s use bpftrace to investigate.

In one terminal:

$ sudo bpftrace -e 'kprobe:generic_read_dir { printf("%s\n", kstack); }'
Attaching 1 probe...

In another terminal:

$ ls -l /proc

Nothing in the first terminal. Let’s try the next function.

$ sudo bpftrace -e 'kprobe:proc_root_readdir { printf("%s\n", kstack); }'
Attaching 1 probe...

Run ls again and we get the following output:

        proc_root_readdir+1
        iterate_dir+323
        ksys_getdents64+156
        __x64_sys_getdents64+22
        do_syscall_64+78
        entry_SYSCALL_64_after_hwframe+68


        proc_root_readdir+1
        iterate_dir+323
        ksys_getdents64+156
        __x64_sys_getdents64+22
        do_syscall_64+78
        entry_SYSCALL_64_after_hwframe+68

Nice, so we know running ls generates a proc_root_readdir callback. Let’s look at the code:

static int proc_root_readdir(struct file *file, struct dir_context *ctx)
{
        if (ctx->pos < FIRST_PROCESS_ENTRY) {
                int error = proc_readdir(file, ctx);
                if (unlikely(error <= 0))
                        return error;
                ctx->pos = FIRST_PROCESS_ENTRY;
        }

        return proc_pid_readdir(file, ctx);
}

FIRST_PROCESS_ENTRY is defined as:

in fs/proc/internal.h:

/*
 * Offset of the first process in the /proc root directory..
 */
#define FIRST_PROCESS_ENTRY 256

and we see proc_readdir incrementing pos in proc_readdir_de (a later callee). So this code probably handles all the non-process entries in /proc and we can ignore it for now and focus on proc_pid_readdir.

In fs/proc/base.c:

/* for the /proc/ directory itself, after non-process stuff has been done */
int proc_pid_readdir(struct file *file, struct dir_context *ctx)
{
        struct tgid_iter iter;
        struct pid_namespace *ns = proc_pid_ns(file_inode(file));
        loff_t pos = ctx->pos;

This code just sets up some variables by pulling context information out. Not really important.

        if (pos >= PID_MAX_LIMIT + TGID_OFFSET)
                return 0;

        if (pos == TGID_OFFSET - 2) {
                struct inode *inode = d_inode(ns->proc_self);
                if (!dir_emit(ctx, "self", 4, inode->i_ino, DT_LNK))
                        return 0;
                ctx->pos = pos = pos + 1;
        }
        if (pos == TGID_OFFSET - 1) {
                struct inode *inode = d_inode(ns->proc_thread_self);
                if (!dir_emit(ctx, "thread-self", 11, inode->i_ino, DT_LNK))
                        return 0;
                ctx->pos = pos = pos + 1;
        }

This code does 3 things:

Impose a limit on the number of entries
Emit the /proc/self entry
Emit the /proc/thread-self entry

Interesting to note but not important for this article.

        iter.tgid = pos - TGID_OFFSET;
        iter.task = NULL;
        for (iter = next_tgid(ns, iter);
             iter.task;
             iter.tgid += 1, iter = next_tgid(ns, iter)) {

Now this is the interesting bit. Now we’re iterating through all the thread group IDs (tgid) via next_tgid. TGIDs are better understood from userspace as the PIDs we see, where each process can have multiple threads (each with their own TID).

             ...
             <fill dcache entry>
             ...
        }
}

There’s more code that follows but it’s not very interesting for us.

So we know why ls /proc does not show threads now. But how does directly accessing /proc/[TID]/comm work?

We follow the same process with bpftrace and try some more functions. Finally, we discover that the following triggers output when we run cat /proc/864518/comm:

$ sudo bpftrace -e 'kprobe:proc_root_lookup / comm == "cat" / { printf("%s\n", kstack); }'
Attaching 1 probe...

        proc_root_lookup+1
        __lookup_slow+140
        walk_component+513
        link_path_walk+759
        path_openat+157
        do_filp_open+171
        do_sys_openat2+534
        do_sys_open+68
        do_syscall_64+78
        entry_SYSCALL_64_after_hwframe+68

Note that we used a filter in our bpftrace script to limit output to our command.

Astute readers might have noted that our cat command used a different TID. That’s because we only trigger output once per lifetime (or some other period of time) of the TID. That’s because the kernel is probably caching directory entries in memory so it doesn’t have to do a full lookup every time.

Now look at proc_root_lookup:

In fs/proc/root.c:

static struct dentry *proc_root_lookup(struct inode * dir, struct dentry * dentry, unsigned int flags)
{
        if (!proc_pid_lookup(dentry, flags))
                return NULL;

        return proc_lookup(dir, dentry, flags);
}

In fs/proc/base.c:

struct dentry *proc_pid_lookup(struct dentry *dentry, unsigned int flags)
{
        struct task_struct *task;
        unsigned tgid;
        struct pid_namespace *ns;
        struct dentry *result = ERR_PTR(-ENOENT);

        tgid = name_to_int(&dentry->d_name);
        if (tgid == ~0U)
                goto out;

Some setup and error checks. Not too interesting.

        ns = dentry->d_sb->s_fs_info;
        rcu_read_lock();
        task = find_task_by_pid_ns(tgid, ns);

This is more interesting: we do a lookup on the requested tgid. Note that tgid here is somewhat improperly named. We’re doing a lookup based on a task which does not have to be a thread group leader.

        if (task)
                get_task_struct(task);
        rcu_read_unlock();
        if (!task)
                goto out;

        result = proc_pid_instantiate(dentry, task, NULL);
        put_task_struct(task);
out:
        return result;
}

The remainder of the function instantiates an inode for /proc/[TID] and most likely populates it as well. Then in proc_root_lookup, proc_lookup probably walks the FS structure and finds the new inode.

Mystery solved.

GDB hacking

2020-07-04T20:05:46-07:00

GDB hacking Home

GDB hacking

This past weekend I spent some time hacking on GDB where the overall goal was to improve rust debugging support. I considered hacking on LLDB but their upstream stance on language support is fairly conservative. The current state is that they want non C-family languages to be maintained out of tree. Anyone who’s done open source software maintenance knows that’s an enormous burden on a team – to say nothing of a single person. In fact, such an out of tree LLDB has been attempted before but the sole maintainer got a new job and the patches ended up being too difficult to rebase. So support was dropped. And that’s how I decided to hack on GDB.

Despite being released over 30 years ago, the codebase is still in fairly great shape. In fact, the entire GDB codebase uses C++11. The current developers/maintainers have obviously taken great care to keep things modern. This is evidenced by the tasteful use of C++11 features (smart pointers, override, limited use of templates).

It took about a day, but I did manage to get a couple small patches out. The only patch worth noting is adding support for raw identifiers, a fairly useless but educational feature. My hope is to steadily send patches for at least a little while.

To end this post, I’ll share one cool trick: debugging GDB with GDB:

$ cd ~/dev/gdb/build/gdb
$ make GDBFLAGS=./gdb run
[...]

(top-gdb) b print_command_1
Breakpoint 3 at 0xd7374: print_command_1. (2 locations)
(top-gdb) r

Starting program: /home/daniel/dev/gdb/build/gdb/gdb
[...]
(gdb) p 0

Thread 1 "gdb" hit Breakpoint 3, print_command_1 (During symbol reading: incomplete CFI data; unspecified registers (e.g., rax) at 0x55555589f362
args=0x555555eb9682 "0", voidprint=1) at ../../gdb/printcmd.c:1198
1198    {
(top-gdb) set print frame-info short-location
(top-gdb) where
#0  print_command_1 (args=..., voidprint=...)
#1  0x00005555556d4592 in cmd_func (cmd=..., args=..., from_tty=...)
#2  0x0000555555994aac in execute_command (p=..., from_tty=...)
#3  0x000055555578e65d in command_handler (command=...)
#4  0x000055555578e9dd in command_line_handler (rl=...)
#5  0x000055555578efde in gdb_rl_callback_handler (rl=...)
#6  0x0000555555a15d90 in rl_callback_read_char ()
#7  0x000055555578dc1e in gdb_rl_callback_read_char_wrapper_noexcept ()
#8  0x000055555578ee91 in gdb_rl_callback_read_char_wrapper (client_data=...)
#9  0x000055555578d950 in stdin_event_handler (error=..., client_data=...)
#10 0x0000555555ade2b6 in gdb_wait_for_event (block=...)
#11 0x0000555555ade55d in gdb_wait_for_event (block=...)
#12 gdb_do_one_event ()
#13 0x00005555558509f5 in start_event_loop ()
#14 captured_command_loop ()
#15 0x0000555555852c75 in captured_main (data=...)
#16 gdb_main (args=...)
#17 0x000055555563aa2c in main (argc=..., argv=...)

The neat part is that something is detecting the gdb-gdb debug and changing the top level gdb prompt to top-gdb. This is quite helpful when debugging changes or tracing code paths.

Playing with C++ templates

2020-07-04T20:09:14-07:00

Playing with C++ templates Home

Playing with C++ templates

Last week I found myself quite sick. Rather than wallow in it, I played around with C++ templates. Until now, I’ve only parameterized classes into generic functions. As a daily user of folly, I’ve spent quite some time puzzling over C++ template magic. It’s time I got my feet wet.

Simple example

We want a generic function to call a standard interface:

#include <iostream>

struct A {
    int a() {
        return 1;
    }
};

struct B {
    int a() {
        return 2;
    }
};

template <typename T>
void foo() {
    T t;
    std::cout << t.a() << "\n";
}

int main() {
    foo<A>();
    foo<B>();
}

When compiled and run:

$ g++ typename.cpp
$ ./a.out
1
2

Note that we may choose to use old-style class instead of typename on line 15. However, for clarity, class should be avoided. Trivia: Bjarne originally reused the class keyword in template declarations to avoid another reserved keyword.

Integers inside templates

Now another example. Did you know templates declarations can specify more than class types? In fact, according to the cppreference, we can use:

std::nullptr_t (since C++11)
integral type
lvalue reference type (to object or to function)
pointer type (to object or to function)
pointer to member type (to member object or to member function)
enumeration type

Let’s play around with an integer type inside a template:

#include <iostream>

template <int N = 10>
void foo() {
    std::cout << N << "\n";
}

int main() {
    foo<1>();
    foo<2>();
    foo();
}

When compiled and run:

$ g++ int.cpp
$ ./a.out
1
2
10

One can see how this might be useful to flexibly declare static array sizes at compile time (to avoid variable-length arrays, whose ugliness is described in last week’s LWN).

Template metaprogramming

Now that we’ve covered some template basics, let’s play around with template metaprogramming. Succinctly, template metaprogramming is the ability to do fancy compile time things. For example, a library developer can optimize a templated routine for certain template parameter types. One way to do this is through std::enable_if. In folly, the futures library overloads onError(F&& func) to return a different return type based on the template parameter type.

We will do something far less complicated. We will play with std::decay, a C++ standard library function that “decays” types. For example, the type Foo& and Foo should, in many cases, be considered the same. However, a C++ compiler cannot safely assume that is always the case, especially for functions like std::is_same.

In the spirit of flexibility, the C++ committee provides a mechanism to “decay” types into “base” types. Hopefully that makes more sense than the official documentation:

Applies lvalue-to-rvalue, array-to-pointer, and function-to-pointer implicit conversions to the type T, removes cv-qualifiers, and defines the resulting type as the member typedef type.

#include <iostream>
#include <type_traits>

struct A {};
struct B : A {};

int main() {
    std::cout << std::boolalpha
        << std::is_same<A, B>::value << '\n'
        << std::is_same<A, A>::value << '\n'
        << std::is_same<A, std::decay<A*>::type>::value << '\n'
        << std::is_same<A, std::decay<A&>::type>::value << '\n';
}

While seemingly complicated, this snippet is quite simple. We perform 4 checks on lines 9-12: if

types A and B are equivalent (one should hope not)
types A and A are equivalent (one should hope so)
if A* decays into A
if A& decays into A

When compiled and run:

$ g++ decay.cpp
$ ./a.out
false
true
false
true

Hopefully checks 1 and 2 are not surprising. Checks 3 and 4 deserve a brief explanation:

Check 3 shows that A* does not decay into A. This makes sense because at runtime, one cannot perform the same operations on A* as on A. For instance, if A were a class, to call a method on type A, one does A.foo(). One cannot do the same with A*. A* needs to be dereferenced first.

Check 4 shows that A& decays into A. This makes sense because at runtime, most non-metaprogramming operations work the same.

GDB hacking part 2

2020-07-09T10:34:59-07:00

GDB hacking part 2 Home

GDB hacking part 2

One of the issues I’ve had with GDB is reliably building the code. The project apparently checks in automake-1.15 artifacts and my host has automake-1.16, so that means any code changes that trigger automake regeneration causes failures. Arch linux doesn’t package auotmake-1.15 like some other distros so I’ve had to get clever.

My solution was to wrap the build in a docker container to effectively document the build. The thinking goes that I only need to get the build to work once and then it’ll work forever.

The way it works is I have an x.py:

$ ./x.py help
usage: x [-h] [-s SOURCE_DIR] [-b BUILD_DIR] {conf,build,run,shell,test,help} ...

positional arguments:
  {conf,build,run,shell,test,help}
                        subcommands
    conf                configure and build gdb container image
    build               build gdb
    run                 run gdb
    shell               open shell
    test                run `make check tests`
    help                print help

optional arguments:
  -h, --help            show this help message and exit
  -s SOURCE_DIR, --source-dir SOURCE_DIR
                        source code directory (default: ~/dev/gdb)
  -b BUILD_DIR, --build-dir BUILD_DIR
                        build directory (default: /tmp/gdb-build)

Running something like ./x.py build effectively expands to:

podman run                         \
  -v=/home/daniel/dev/gdb:/gdb/src \
  -v=/tmp/gdb-build:/gdb/build     \
  localhost/gdb-builder            \
  make -C build -j4

To those not familiar with docker, the command bind mounts the source and build directories to directories inside the container and then it runs make -C build -j4. Note I’ve had to use podman/crun instead of the usual docker/runc because I run cgroup2 on my host (I do some cgroup2 related development) and docker doesn’t support cgroup2 yet.

The gdb-builder image is built by the (simplified for post) Containerfile:

FROM ubuntu

RUN apt-get update
RUN apt-get install -y \
  automake-1.15 \
  bash \
  bison \
  build-essential \
  curl \
  dejagnu \
  flex \
  g++ \
  libncurses-dev \
  libreadline-dev \
  texinfo \
  xsltproc \
  zlib1g-dev

WORKDIR /gdb

COPY scripts/configure.sh configure.sh
RUN chmod 755 configure.sh

Final thoughts

I’ve never really used docker before and I thought this infrastructure would be a good way to play around. I’m happy I did this because docker ended up being a fairly ergonomic way to document the build. Every time I need something new, it’s easy to add another subcommand to save the command line invocation. The command line invocation is certain to work b/c it’s all done inside a reproducible container.

Code

The full script repository is available here.

Understanding btrfs internals

2020-09-12T18:00:01-07:00

Understanding btrfs internals Home

Understanding btrfs internals

This is the first of a multipart series that explains the basics of btrfs’s on-disk format.

At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls.

Example code is available here.

Background

Before we begin, it might be helpful to go over some filesystem basics. First, there’s block devices. Block devices are a level of abstraction over physical (or virtual) hardware. You can think of a block device as a linear series of bytes, of which you can read or write at any offset or size. Filesystems are typically built on top of block devices and offers structure and other very useful functionality for storing your data. Such functionality might include files, directories, data checksums, striping over multiple block devices, compression, etc. To support these features, filesystems need to store metadata on the block device in addition to user data.

Second, what you typically think of when I say “filesystem” is probably a POSIX filesystem. A POSIX filesystem has all the unix-y things we’ve come to love: files, “.” and “..” directories, a filesystem tree, as well as the standard APIs (eg read(), write(), lseek(), etc). If the APIs are implemented as system calls (they usually are), the kernel has to be filesystem aware. For linux, most filesystems are either compiled into the kernel or loaded as kernel modules.

Third, a filesystem image can be thought of as the contents of the entire block device a filesystem is in charge of. It’s usually not a great idea to manually modify a filesystem image. However, it’s totally safe to read from a mounted or unmounted one. “Safe” here means you probably won’t corrupt any data but you might get inconsistent data if you read from a mounted image (b/c the kernel could be making changes).

Notes

A couple notes to keep in mind while reading:

The code samples here may not compile – certain necessary boilerplate may be omitted in the interest of legibility
There’s a lot of uninteresting details I’ll skip over (as well as interesting details, but unfortunately I can’t cover it all)

Creating a btrfs image

Instead of a “real” block device with a real physical drive behind it, we’ll use a fake block device (a loopback) to keep things simple (or even more complicated, if you know how a loopback device works under the hood). To create a loopback btrfs image and mount it, run:

$ truncate -s 1G image

$ mkfs.btrfs image
btrfs-progs v5.7
See http://btrfs.wiki.kernel.org for more information.

Label:              (null)
UUID:               a32cd5e8-2729-4281-b41b-153ea353ffd3
Node size:          16384
Sector size:        4096
Filesystem size:    1.00GiB
Block group profiles:
  Data:             single            8.00MiB
  Metadata:         DUP              51.19MiB
  System:           DUP               8.00MiB
SSD detected:       no
Incompat features:  extref, skinny-metadata
Runtime features:
Checksum:           crc32c
Number of devices:  1
Devices:
   ID        SIZE  PATH
    1     1.00GiB  image

$ sudo mkdir /mnt/btrfs

$ sudo mount image /mnt/btrfs

$ findmnt /mnt/btrfs
TARGET     SOURCE     FSTYPE OPTIONS
/mnt/btrfs /dev/loop0 btrfs  rw,relatime,ssd,space_cache,subvolid=5,subvol=/

Note that image is a regular file with some bytes in it. If you choose to leave the image mounted, remember to run sync after modifying anything in the filesystem so that the changes are persisted to “disk”, or in our case, our image file.

Parsing the superblock

The superblock is the starting point of any filesystem. It’s a structure of predefined size written to a predefined location inside the filesystem image. The key property is that it has all the information necessary to bootstrap and initialize filesystem data structures.

First we’ll define the on-disk superblock structure:

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsSuperblock {
    pub csum: [u8; BTRFS_CSUM_SIZE],
    pub fsid: [u8; BTRFS_FSID_SIZE],
    /// Physical address of this block
    pub bytenr: u64,
    pub flags: u64,
    pub magic: [u8; 0x8],
    pub generation: u64,
    /// Logical address of the root tree root
    pub root: u64,
    /// Logical address of the chunk tree root
    pub chunk_root: u64,
    /// Logical address of the log tree root
    pub log_root: u64,
    pub log_root_transid: u64,
    pub total_bytes: u64,
    pub bytes_used: u64,
    pub root_dir_objectid: u64,
    pub num_devices: u64,
    pub sector_size: u32,
    pub node_size: u32,
    /// Unused and must be equal to `nodesize`
    pub leafsize: u32,
    pub stripesize: u32,
    pub sys_chunk_array_size: u32,
    pub chunk_root_generation: u64,
    pub compat_flags: u64,
    pub compat_ro_flags: u64,
    pub incompat_flags: u64,
    pub csum_type: u16,
    pub root_level: u8,
    pub chunk_root_level: u8,
    pub log_root_level: u8,
    pub dev_item: BtrfsDevItem,
    pub label: [u8; BTRFS_LABEL_SIZE],
    pub cache_generation: u64,
    pub uuid_tree_generation: u64,
    pub metadata_uuid: [u8; BTRFS_FSID_SIZE],
    /// Future expansion
    pub _reserved: [u64; 28],
    pub sys_chunk_array: [u8; BTRFS_SYSTEM_CHUNK_ARRAY_SIZE],
    pub root_backups: [BtrfsRootBackup; 4],
}

To parse the superblock, we write the following code:

const BTRFS_SUPERBLOCK_OFFSET: u64 = 0x10_000;

The first btrfs superblock (of possibly 3) starts at offset 0x10000.

const BTRFS_SUPERBLOCK_MAGIC: [u8; 8] = *b"_BHRfS_M";

Most superblocks have a “magic” value embedded inside so that a filesystem implementation has a way to easily identify that the image it’s been told to process is a format it can understand.

fn parse_superblock(file: &File) -> Result<BtrfsSuperblock> {
    let mut superblock: BtrfsSuperblock = unsafe { std::mem::zeroed() };
    let superblock_size = std::mem::size_of::<BtrfsSuperblock>();
    let slice;
    unsafe {
        slice = slice::from_raw_parts_mut(&mut superblock as *mut _ as *mut u8, superblock_size);
    }
    file.read_exact_at(slice, BTRFS_SUPERBLOCK_OFFSET)?;
    if superblock.magic != BTRFS_SUPERBLOCK_MAGIC {
        bail!("superblock magic is wrong");
    }

    Ok(superblock)
}

This function takes our image file as a borrowed File and returns a BtrfsSuperblock. Then we do a bit of unsafe rust to read from the right offset sizeof(BtrfsSuperblock) bytes. If, after reading, the magic doesn’t match, we bail with an error. The superblock technically contains a csum (short for checksum) value and we could (and probably should) check that the checksum for the superblock matches, but we’re lazy and verifying the magic value is probably good enough.

Now that we have the superblock, we can start bootstrapping the rest of the data structures. More in the next post.

Setting up a barebones website

2020-09-12T18:06:13-07:00

Setting up a barebones website Home

Setting up a barebones website

I had a very specific workflow in mind when I set out to host a website. First, I had no desire to be locked into a specific platform. I’ve experimented with Github pages but it’s pretty locked into Github’s ecosystem.

Second, publication and hosting had to be as cross platform as possible. I wanted to be able to write and publish from anywhere.

Lastly, I wanted writing to be as hassle-free and minimally invasive as possible. This meant I did not want to figure out how Jekyll or Hugo or whatever- have-you works. I understand that those tools exist for specific reasons, but all I wanted was a simple setup.

In other words, I wanted something that would be simple, straightforward, and portable.

I came up with the following workflow:

I write a post or make a change to my website. I do this using Pandoc flavored markdown.
I push the commit to my site’s Github repo. Note: I could use any git hosting service here, including my own.
Then on my server a systemd timer eventually fires. It pulls down the most recent commits and compiles all the Markdown files to HTML. It then updates my webserver’s hosting directory.
Some unfortunate soul visits my website and my webserver serves them this junk.

Take a peek at all the backend configuration here.

Notes

Yes, I hard code links to other pages on my site. It might be a bad idea.
Lots of inspiration was drawn from Dan Luu‘s website’.

On writing unmaintainable code

2020-09-12T18:09:55-07:00

On writing unmaintainable code Home

On writing unmaintainable code

The internet’s beaten to death the concept of legacy code, unmaintainable code, bad code, and insert-word-here-code. It’s safe to say nobody really likes dealing with code that doesn’t make much sense. But how about the other side of the coin?

Consider this: you’re tasked with writing inherently tricky code. You know this code will be nigh unreadable next week, but you also know someone has to write it. How do you manage the future pain of dealing with this code?

Last week I gave that question some thought and I now believe there are two options in such a scenario.

Write a lot of comments to help the next guy that comes along to understand your genius (or lack thereof).
Explain what this snippet of code does with clear {pre,post}-conditions so it can simply be ripped out and replaced next time someone needs it changed.

Option 1 is generally good advice. Comments enhance readability after all. But consider this snippet:

/* Replace a subsection of a buffer with a replacement string */
void
Scrubber::scrub_buffer(char *buffer, Scrub *scrub) const
{
  char *buffer_ptr;
  int num_matched;
  int match_len;
  int replacement_len;
  int buffer_len = strlen(buffer);

  buffer_ptr      = buffer;
  match_len       = scrub->ovector[1] - scrub->ovector[0];
  replacement_len = scrub->replacement.size();

  if (replacement_len <= match_len) {
    buffer_ptr += scrub->ovector[0];
    memcpy(buffer_ptr, scrub->replacement.ptr(), replacement_len);
    buffer_ptr += replacement_len;
    memmove(buffer_ptr, buffer + scrub->ovector[1], buffer_len - scrub->ovector[1] + 1);
    ink_assert(buffer[buffer_len - (match_len - replacement_len)] == '\0');
  } else {
    int n_slide = buffer_len - scrub->ovector[0] - replacement_len;
    if (n_slide < 0) {
      replacement_len += n_slide;
    } else {
      buffer_ptr += scrub->ovector[0] + replacement_len;
      memmove(buffer_ptr, buffer + scrub->ovector[1], n_slide);
    }
    buffer_ptr = buffer + scrub->ovector[0];
    memcpy(buffer_ptr, scrub->replacement.ptr(), replacement_len);
    buffer[buffer_len] = '\0';
  }
}

Notice all the C-style pointer fiddling. Reasoning through the code itself would take less time than reading a sufficiently detailed enough comment for each line to elucidate each statement. Thus, writing sufficient comments would theoretically add complexity to the codebase.

Now consider option 2. To continue the previous code example, take a look at this illustration:

/*
 * When scrubbing the buffer in place, there are 2 scenarios we need to consider:
 *
 *   1) The replacement text length is shorter or equal to the text we want to scrub away
 *   2) The replacement text is longer than the text we want to scrub away
 *
 * In case 1, we simply "slide" everything left a bit. Our final buffer should
 * look like this (where XXXX is the replacement text):
 *
 *                                new_end  orig_end
 *                                    V      V
 *   -----------------------------------------
 *   |ORIGINAL TEXT|XXXX|ORIGINAL TEXT|      |
 *   -----------------------------------------
 *
 * In case 2, since the final buffer would be longer than the original allocated buffer,
 * we need to truncate everything that would have run over the original end of the buffer.
 * The final buffer should look like this:
 *
 *                                         new_end
 *                                        orig_end
 *                                           V
 *   -----------------------------------------
 *   |ORIGINAL TEXT|XXXXXXXXXXXXXXXXXXX|ORIGI|NAL TEXT
 *   -----------------------------------------
 *
 */

To quote an old english idiom, “A picture is worth a thousand words”. In this case, our picture is actually some ASCII art. Now, the next person who needs to change this function has two options of their own:

Understand and modify the existing code with knowledge of the scope of the code.
Rip out all the old code and write something (hopefully) better.

Waiting on process exit

2020-09-12T18:25:06-07:00

Waiting on process exit Home

Waiting on process exit

How does a process wait for another process to exit? Simple question, right?

I’ve been recently working on bpftrace and I had to find an answer for this problem. Ideally, the solution would not do any polling. First, let’s try the most naive solution: waitpid(2):

waitpid(2)

#include <iostream>
#include <cstdlib>
#include <thread>

#include <sys/epoll.h>
#include <sys/eventfd.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <stdio.h>
#include <unistd.h>

#define MAX_EVENTS 10

void waiter(int pid, int efd) {
  int wstatus;
  if (waitpid(pid, &wstatus, 0) < 0) {
    perror("waitpid");
    return;
  }

  int one = 1;
  if (write(efd, &one, sizeof(one)) < 0) {
    perror("write");
    return;
  }

  return;
}

int main(int argc, const char** argv) {
  int pid;

  if (argc < 2) {
    std::cerr << "usage: ./waiter <pid>" << std::endl;
    return 1;
  }

  pid = std::atoi(argv[1]);

  // create eventfd in semaphore mode
  int efd = eventfd(0, EFD_CLOEXEC | EFD_SEMAPHORE);
  if (efd < 0) {
    perror("eventfd");
    return 1;
  }

  // set up epoll
  struct epoll_event ev, events[MAX_EVENTS];
  int epollfd = epoll_create1(EPOLL_CLOEXEC);
  if (epollfd < 0) {
    perror("epoll_create1");
    return 1;
  }
  ev.events = EPOLLIN;
  ev.data.fd = efd;
  if (epoll_ctl(epollfd, EPOLL_CTL_ADD, efd, &ev) < 0) {
    perror("epoll_ctl");
    return 1;
  }

  auto t = std::thread([&]() {
    waiter(pid, efd);
  });

  while (1) {
    int nfds = epoll_wait(epollfd, events, MAX_EVENTS, -1);
    if (nfds == -1) {
      perror("epoll_wait");
      return 1;
    }

    for (int i = 0; i < nfds; ++i) {
      if (events[i].data.fd == efd) {
        std::cerr << pid << " has exited" << std::endl;
        break;
      }
    }
  }

  t.join();
  close(efd);
  close(epollfd);
}

In short, our first program creates a thread, passes an eventfd handle to it, and writes to the eventfd handle once the thread exits waitpid(2). What happens if we run it?

In one window:

$ python3
>>> import os
>>> os.getpid()
1573

In another:

$ g++ waitpid-waitpid.cpp -lpthread
$ ./a.out 1573
waitpid: No child processes
^C

Unfortunately, waitpid(2) only works on child processes.

Let’s try a different strategy.

epoll(2) on /proc/pid

#include <iostream>
#include <cstdlib>

#include <fcntl.h>
#include <sys/epoll.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <stdio.h>
#include <unistd.h>

#define MAX_EVENTS 10

int main(int argc, const char** argv) {
  int pid;
  char buf[1024];

  if (argc < 2) {
    std::cerr << "usage: ./waiter <pid>" << std::endl;
    return 1;
  }

  pid = std::atoi(argv[1]);

  if (snprintf(&buf[0], sizeof(buf), "/proc/%d/status", pid) < 0) {
    std::cerr << "snprintf failed" << std::endl;
    return 1;
  }

  std::cerr << "trying to open=" << buf << std::endl;

  int pidfd = open(&buf[0], 0);
  if (pidfd < 0) {
    perror("open");
    return 1;
  }

  // set up epoll
  struct epoll_event ev, events[MAX_EVENTS];
  int epollfd = epoll_create1(EPOLL_CLOEXEC);
  if (epollfd < 0) {
    perror("epoll_create1");
    return 1;
  }

  ev.events = EPOLLERR | EPOLLHUP;  // wait for procfs entry to disappear
  ev.data.fd = pidfd;
  if (epoll_ctl(epollfd, EPOLL_CTL_ADD, pidfd, &ev) < 0) {
    perror("epoll_ctl");
    return 1;
  }

  while (1) {
    int nfds = epoll_wait(epollfd, events, MAX_EVENTS, -1);
    if (nfds == -1) {
      perror("epoll_wait");
      return 1;
    }

    bool exited = false;
    for (int i = 0; i < nfds; ++i) {
      if (events[i].data.fd == pidfd) {
        std::cerr << pid << " has exited" << std::endl;
        exited = true;
      }
    }

    if (exited) {
      break;
    }
  }

  close(pidfd);
  close(epollfd);
}

Hey, a clever idea! Let’s poll on /proc/pid/status and wait for an EPOLLHUP event. Let’s see if it works.

$ ./a.out 1573
trying to open=/proc/1573/status
epoll_ctl: Operation not permitted

It turns out that epoll does not support pseudo-fs kernel interfaces.

An interesting side note, according to proc(5):

/proc/[pid]/mounts (since Linux 2.4.19)
...
Since kernel version 2.6.15, this file is pollable: after
opening the file for reading, a change in this file (i.e., a
filesystem mount or unmount) causes select(2) to mark the file
descriptor as having an exceptional condition, and poll(2) and
epoll_wait(2) mark the file as having a priority event (POLL‐
PRI)

Let’s make this change:

--- waitpid_epollhup.cpp        2019-01-16 20:16:29.078024749 -0800
+++ waitpid_epollhup2.cpp       2019-01-16 20:17:25.766842080 -0800
@@ -21,7 +21,7 @@

   pid = std::atoi(argv[1]);

-  if (snprintf(&buf[0], sizeof(buf), "/proc/%d/status", pid) < 0) {
+  if (snprintf(&buf[0], sizeof(buf), "/proc/%d/mounts", pid) < 0) {
     std::cerr << "snprintf failed" << std::endl;
     return 1;
   }
@@ -42,7 +42,7 @@
     return 1;
   }

-  ev.events = EPOLLERR | EPOLLHUP;  // wait for procfs entry to disappear
+  ev.events = EPOLLERR | EPOLLHUP | EPOLLPRI;  // wait for procfs entry to disappear
   ev.data.fd = pidfd;
   if (epoll_ctl(epollfd, EPOLL_CTL_ADD, pidfd, &ev) < 0) {
     perror("epoll_ctl");

and see if it works. Run:

$ ./a.out 9001
trying to open=/proc/9001/mounts

and then kill the python process. Unfortunately (hard to show in text), it does not work. a.out hangs.

Final attempt: polling procfs

Even though I said I didn’t want to poll, we might still be able to get away with polling if we do it infrequently enough. Consider:

#include <iostream>
#include <stdexcept>

#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdio.h>

bool is_pid_alive(int pid) {
  char buf[1024];
  int ret = snprintf(&buf[0], sizeof(buf), "/proc/%d/status", pid);
  if (ret < 0) {
    throw std::runtime_error("failed to snprintf");
  }

  int fd = open(&buf[0], 0);
  if (fd < 0 && errno == ENOENT) {
    return false;
  }
  close(fd);

  return true;
}

int main(int argc, const char** argv) {
  if (argc < 2) {
    std::cerr << "usage: ./poll <pid>" << std::endl;
    return 1;
  }

  int pid = std::atoi(argv[1]);

  while (1) {
    if (!is_pid_alive(pid)) {
      std::cerr << pid << " has died" << std::endl;
      break;
    }

    sleep(1);
  }
}

When run with the same python setup:

$ ./a.out 11643
11643 has died

I guess we’ll have to live with this.

Unattempted solutions

There were a few ideas I knew about but didn’t try for various reasons:

inotify
- This doesn’t work for the same reason as epolling on procfs didn’t: inotify requires changes through a userspace filesystem.
netlink
- There does exist a netlink interface for task stats. However, creating a netlink socket and monitoring it takes quite a bit of boilerplate. For bpftrace’s use case, it was far cleaner and less bug prone to simply poll.
bpf
- We could have really nested the turtles here and created another bpf program to watch for process exit. There’s no real reason I didn’t take this route other than it was going to take a lot of code.
ptrace with PTRACE_SEIZE
- It wasn’t clear to me this would be overhead-free on the target process. Perhaps in the future I can run some tests.

Understanding btrfs internals part 2

2020-09-14T09:48:11-07:00

Understanding btrfs internals part 2 Home

Understanding btrfs internals part 2

This is the second of a multipart series that explains the basics of btrfs’s on-disk format.

At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls.

Example code is available here.

Background

btrfs stores everything (with the exception of the superblocks) in B-trees. There’s a number of different trees – each with their own purpose. All the trees are linked together with a root of roots: a root node that contains references to the roots of each tree. As a result, not much data needs to be placed at fixed offsets. This property allows users to perform in-place converstions from ext2/3/4 or reiserfs to btrfs.

btrfs also supports multiple devices, meaning data can be striped or duplicated in various RAID configurations. This information is stored in two central places: the chunk tree and the device tree. The chunk tree maps logical offsets to one or more physical offsets. The device tree does the inverse. The two trees allow btrfs to grow and shrink without unmounting, as chunks can be located and moved on the fly.

Almost all of the on-disk structures work with logical offsets, including the BtrfsSuperblock::root field. In order to get access to the root of roots, we must first bootstrap the chunk tree so that we can translate the logical offset to a physical one. We can then use the physical offset to read data from disk.

Bootstrapping the chunk tree

First, let’s define the necessary structures:

pub const BTRFS_CHUNK_ITEM_KEY: u8 = 228;

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsKey {
    pub objectid: u64,
    pub ty: u8,
    pub offset: u64,
}

BtrfsKey is a fundamental type. All data items in the on-disk b-trees are keyed by a BtrfsKey. A node has one or more data items. The data items in each node are ordered by their key in ascending order. This allows the tree walking algorithms to do efficient binary searches. I’ll go deeper into this in the next post. Note that the all on-disk structures are stored little-endian. In a “real” implementation, you’d see more little-endian to host-order helpers. However, since your author is writing this on an x86 machine, we ignore translating helpers to keep things simple.

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsStripe {
    pub devid: u64,
    pub offset: u64,
    pub dev_uuid: [u8; BTRFS_UUID_SIZE],
}

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsChunk {
    /// size of this chunk in bytes
    pub length: u64,
    /// objectid of the root referencing this chunk
    pub owner: u64,
    pub stripe_len: u64,
    pub ty: u64,
    /// optimal io alignment for this chunk
    pub io_align: u32,
    /// optimal io width for this chunk
    pub io_width: u32,
    /// minimal io size for this chunk
    pub sector_size: u32,
    pub num_stripes: u16,
    /// sub stripes only matter for raid10
    pub sub_stripes: u16,
    pub stripe: BtrfsStripe,
    // additional stripes go here
}

A BtrfsStripe describes a single stripe. One or more may be present depending on the configured RAID level. If more than one is present, the additional BtrfsStripes are laid out “after” the associated BtrfsChunk (see below).

BtrfsChunk is simply the data stored on-disk to describe a chunk. BtrfsSuperblock::sys_chunk_array stores BtrfsKeys and BtrfsChunks “head to feet”. In other words, with two stripes per chunk:

Writing the code

To bootstrap the chunk tree, we write the following code:

fn bootstrap_chunk_tree(superblock: &BtrfsSuperblock) -> Result<ChunkTreeCache> {
    let array_size = superblock.sys_chunk_array_size as usize;
    let mut offset: usize = 0;

We’ll be walking the length of sys_chunk_array, so we’ll need to know where in the array we’re at and when to stop.

    let mut chunk_tree_cache = ChunkTreeCache::default();

The ChunkTreeCache is a brain-dead implementation of a non-overlapping interval tree. We’ll omit the implementation here for brevity but you can view it in full at the link at the top of this page. Suffice to say, the ChunkTreeCache stores (logical offset start, length) -> physical offset start mappings and does a little math to help us translate a logical offset to physical offset, especially if the logical offset is in the middle of a chunk.

    while offset < array_size {
        let key_size = std::mem::size_of::<BtrfsKey>();
        if offset + key_size > array_size as usize {
            bail!("short key read");
        }

        let key_slice = &superblock.sys_chunk_array[offset..];
        let key = unsafe { &*(key_slice.as_ptr() as *const BtrfsKey) };
        if key.ty != BTRFS_CHUNK_ITEM_KEY {
            bail!(
                "unknown item type={} in sys_array at offset={}",
                key.ty,
                offset
            );
        }
        offset += key_size;

First, we start a loop that continues until the entire array is processed.

Next, we look for the current BtrfsKey to process. We do a little unsafe rust to get a &BtrfsKey from the raw sys_chunk_array buffer. There should only be BTRFS_CHUNK_ITEM_KEYs in sys_chunk_array so report an error if we see something unexpected.

        if offset + std::mem::size_of::<BtrfsChunk>() > array_size {
            bail!("short chunk item read");
        }

        let chunk_slice = &superblock.sys_chunk_array[offset..];
        let chunk = unsafe { &*(chunk_slice.as_ptr() as *const BtrfsChunk) };
        if chunk.num_stripes == 0 {
            bail!("num_stripes cannot be 0");
        }

Now that we’ve pulled out a BtrfsKey, we try and pull out the associated BtrfsChunk as well. Each BtrfsChunk stores information about one or more stripes, depending on the RAID level. We have to have at least one stripe otherwise it means there’s no backing device. We do this check just for sanity.

        let num_stripes = chunk.num_stripes; // copy to prevent unaligned access
        if num_stripes != 1 {
            println!(
                "warning: {} stripes detected but only processing 1",
                num_stripes
            );
        }

To keep things simple, we’ll only process 1 stripe, as stripes should have identical content. Warn the user just to be nice.

        let logical = key.offset;
        if chunk_tree_cache.offset(logical).is_none() {
            chunk_tree_cache.insert(
                ChunkTreeKey {
                    start: logical,
                    size: chunk.length,
                },
                ChunkTreeValue {
                    offset: chunk.stripe.offset,
                },
            );
        }

Now we add an entry into the ChunkTreeCache if the interval is not already present. Note that BtrfsKey::offset contains the logical start of the chunk. You’ll see this pattern repeated often throughout the btrfs codebase. The only field that has a fixed meaning between different trees and data items is BtrfsKey::ty. BtrfsKey::objectid and BtrfsOffset::offset are opaque and may represent different things from data item to data item.

        let chunk_item_size = std::mem::size_of::<BtrfsChunk>()
            + (std::mem::size_of::<BtrfsStripe>() * (chunk.num_stripes as usize - 1));
        if offset + chunk_item_size > array_size {
            bail!("short chunk item + stripe read");
        }
        offset += chunk_item_size;
    }

    Ok(chunk_tree_cache)
}

Despite only processing one stripe, we need to be careful to skip over the entire chunk item. As mentioned earlier, this is because each additional stripe (beyond the first) is appended to the end of BtrfsChunks.

And finally, we return.

Having bootstrapped the chunk tree, the next thing we need to do is process the rest of the chunk tree. More in the next post.

Understanding btrfs internals part 3

2020-09-27T16:23:36-07:00

Understanding btrfs internals part 3 Home

Understanding btrfs internals part 3

This is the third of a multipart series that explains the basics of btrfs’s on-disk format.

At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls.

Example code is available here.

Background

As explained in part 2, btrfs stores nearly everything on-disk in B-trees. And as promised, I’ll now describe the B-tree data format. First, an example tree:

Each node in a btrfs B-tree is prefixed with a header. The header records the node’s “level”. Level 0 means the node is a leaf node and stores a payload. Level > 0 means the node is an internal node and stores pointers to children nodes. The header also stores the number of “items” the node contains where an “item” is either a pointer to child node if level > 0, else, information on where to find the payload in the node. Recall that each item in a node is sorted by the associated BtrfsKey which allows for efficient binary searches. There’s also some other data but it’s not too important to us.

In our example, root and node 0 contain BtrfsKeyPtrs because they’re not leaf nodes. leaf 0 and leaf 1 contain BtrfsItems because they are leaf nodes.

Reading the chunk tree

Now that we understand how trees are laid out on disk, let’s process the rest of the chunk tree.

First, let’s define the necessary structures:

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsHeader {
    pub csum: [u8; BTRFS_CSUM_SIZE],
    pub fsid: [u8; BTRFS_FSID_SIZE],
    pub bytenr: u64,
    pub flags: u64,
    pub chunk_tree_uuid: [u8; BTRFS_UUID_SIZE],
    pub generation: u64,
    pub owner: u64,
    pub nritems: u32,
    pub level: u8,
}

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsKeyPtr {
    pub key: BtrfsKey,
    pub blockptr: u64,
    pub generation: u64,
}

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsItem {
    pub key: BtrfsKey,
    pub offset: u32,
    pub size: u32,
}

Note that BtrfsItem::offset is the offset from the end of the associated BtrfsHeader that we can find the payload for the BtrfsItem.

Although not strictly necessary, we also define BtrfsNode and BtrfsLeaf as the following:

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsNode {
    pub header: BtrfsHeader,
    // `BtrfsKeyPtr`s begin here
}

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsLeaf {
    pub header: BtrfsHeader,
    // `BtrfsItem`s begin here
}

We don’t need these structure definitions because all it tells us is that every node in the on-disk B-tree starts with BtrfsHeader. After parsing the header and reading BtrfsHeader::level, we can infer what follows the header.

Writing the code

To walk any tree, we need to start at the root node. The superblock contains the logical offset the chunk tree root lives at. To read it:

fn read_chunk_tree_root(
    file: &File,
    chunk_root_logical: u64,
    cache: &ChunkTreeCache,
) -> Result<Vec<u8>> {
    let size = cache
        .mapping_kv(chunk_root_logical)
        .ok_or_else(|| anyhow!("Chunk tree root not bootstrapped"))?
        .0
        .size;
    let physical = cache
        .offset(chunk_root_logical)
        .ok_or_else(|| anyhow!("Chunk tree root not bootstrapped"))?;

    let mut root = vec![0; size as usize];
    file.read_exact_at(&mut root, physical)?;

    Ok(root)
}

where chunk_root_logical is BtrfsSuperblock::chunk_root.

Walking the actual tree looks like a traditional recursive tree-walking algorithm:

fn read_chunk_tree(
    file: &File,
    root: &[u8],
    chunk_tree_cache: &mut ChunkTreeCache,
    superblock: &BtrfsSuperblock,
) -> Result<()> {
    let header = tree::parse_btrfs_header(root)
      .expect("failed to parse chunk root header");

tree::parse_btrfs_header is a simple helper function that extracts the BtrfsHeader out of root and returns a reference to the header.

    // Level 0 is leaf node, !0 is internal node
    if header.level == 0 {
        let items = tree::parse_btrfs_leaf(root)?;

If we’re at level 0, we know we’re looking at a leaf node. So we use tree::parse_btrfs_leaf to extract the BtrfsItems.

        for item in items {
            if item.key.ty != BTRFS_CHUNK_ITEM_KEY {
                continue;
            }

We skip anything that isn’t a chunk item. The chunk tree also contains BTRFS_DEV_ITEM_KEYs which help map physical offsets to logical offsets. However, we only need chunk items for our purpose so we skip everything else.

            let chunk = unsafe {
                // `item.offset` is offset from data portion of `BtrfsLeaf` where
                // associated `BtrfsChunk` starts
                &*(root
                    .as_ptr()
                    .add(std::mem::size_of::<BtrfsHeader>() + item.offset as usize)
                    as *const BtrfsChunk)
            };

As mentioned earlier, BtrfsItem::offset is the offset from the end of the BtrfsHeader. The above code does the proper math to pull out the BtrfsChunk associated with the current item.

            chunk_tree_cache.insert(
                ChunkTreeKey {
                    start: item.key.offset,
                    size: chunk.length,
                },
                ChunkTreeValue {
                    offset: chunk.stripe.offset,
                },
            );
        }

Finally, we add the chunk entry into our chunk tree cache.

    } else {
        let ptrs = tree::parse_btrfs_node(root)?;
        for ptr in ptrs {
            let physical = chunk_tree_cache
                .offset(ptr.blockptr)
                .ok_or_else(|| anyhow!("Chunk tree node not mapped"))?;
            let mut node = vec![0; superblock.node_size as usize];
            file.read_exact_at(&mut node, physical)?;
            read_chunk_tree(file, &node, chunk_tree_cache, superblock)?;
        }
    }

If we see level != 0, we know we’re looking at an internal node. So we use the tree::parse_btrfs_node helper to parse an internal node. Once we have the BtrfsKeyPtrs, we read the node the key points to and recursively call read_chunk_tree.

    Ok(())
}

If we haven’t errored out by the end, it means we successfully walked the chunk tree.

Now that we’ve loaded the entire chunk tree into our cache, we can move onto walking the trees that contain the information we actually care about. In the next post, we’ll extract the filesystem tree root from the root tree root.

Understanding btrfs internals part 5

2020-11-15T18:21:50-08:00

Understanding btrfs internals part 5 Home

Understanding btrfs internals part 5

This is the fifth and final part of a multipart series that explains the basics of btrfs’s on-disk format.

At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls.

Example code is available here.

Background

At the end of Part 4, we had access to the filesystem tree root. All that’s left for us to do now is walk the filesystem tree and make sense of the data it contains.

By now, you should be fairly familiar with the high level B-tree algorithms we’re using. You should also be familiar with how btrfs structures its metadata. The hard part is over! This final part is not very complicated – at this point it’s an exercise in reading the documentation.

Filesystem tree item types

There are quite a few item types stored in the FS tree. However, we only care about two:

BtrfsDirItem represents an entry in a directory. The name of the directory entry directly follows the structure. We’ll enumerate all the BtrfsDirItems in the filesystem, grab their names, and the compute the absolute path leading up to the directory entry.

BtrfsInodeRef is a helper structure that helps link inode numbers to BtrfsDirItems. It also contains information on the parent of the inode. We’ll use this information to locate the parents for every regular file we find. The name of the inode BtrfsInodeRef refers to directly follows the structure.

BtrfsDirItem and BtrfsInodeRef are defined as follows:

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsDirItem {
    pub location: BtrfsKey,
    pub transid: u64,
    pub data_len: u16,
    pub name_len: u16,
    pub ty: u8,
}

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsInodeRef {
    pub index: u64,
    pub name_len: u16,
}

Writing the code

fn walk_fs_tree(
    file: &File,
    superblock: &BtrfsSuperblock,
    node: &[u8],
    root_fs_node: &[u8],
    cache: &ChunkTreeCache,
) -> Result<()> {
    let header = tree::parse_btrfs_header(node)?;

This should be fairly familiar to you by now. Always parse the header of a node first.

    // Leaf node
    if header.level == 0 {
        let items = tree::parse_btrfs_leaf(node)?;
        for item in items {
            if item.key.ty != BTRFS_DIR_ITEM_KEY {
                continue;
            }

            let dir_item = unsafe {
                &*(node
                    .as_ptr()
                    .add(std::mem::size_of::<BtrfsHeader>() + item.offset as usize)
                    as *const BtrfsDirItem)
            };

            if dir_item.ty != BTRFS_FT_REG_FILE {
                continue;
            }

This too should be familiar. If we’re at a leaf node, start processing the items in the node.

For our use case, we only want BtrfsDirItems that represent regular files. Skip everything else.

            let name_slice = unsafe {
                std::slice::from_raw_parts(
                    (dir_item as *const BtrfsDirItem as *const u8)
                        .add(std::mem::size_of::<BtrfsDirItem>()),
                    dir_item.name_len.into(),
                )
            };
            let name = std::str::from_utf8(name_slice)?;

Extract the name of the directory entry. For example, if this BtrfsDirItem represented /home/daniel/dev/readme.txt, name would contain readme.txt.

Now that we have the filename, we must compute the absolute path leading up to it.

            // Capacity 1 so we don't panic the first `String::insert`
            let mut path_prefix = String::with_capacity(1);
            // `item.key.objectid` is parent inode number
            let mut current_inode_nr = item.key.objectid;

Here’s where things get a little trickier. If you haven’t already noticed, btrfs tends to store whatever it wants inside a BtrfsKey. The only thing that’s really nailed down is BtrfsKey::ty. Aside from the type field, the meaning of the BtrfsKey::objectid and BtrfsKey::offset fields completely depend on the item type. In other words, just because the name is offset doesn’t necessarily mean it’s actually the offset to anything.

Here, item.key.objectid actually means the inode number of the current directory item’s parent.

            loop {

Start a loop that’ll end when we’re done looking up the absolute path of this current directory entry.

                let (current_key, _current_inode, current_inode_payload) =
                    get_inode_ref(current_inode_nr, file, superblock, root_fs_node, cache)?
                        .ok_or_else(|| {
                            anyhow!("Failed to find inode_ref for inode={}",
                                current_inode_nr)
                        })?;
                unsafe { assert_eq!(current_key.objectid, current_inode_nr) };

Look up the BtrfsInodeRef for the current_inode_nr. When the loop beings, it holds the immediate parent to the directory entry. As the loop iterates, it’ll go to the parent’s parent, the parent’s parent’s parent, etc. until we reach the root of the filesystem.

get_inode_ref() returns a tuple of:

BtrfsKey associated with the BtrfsInodeRef
the BtrfsInodeRef struct itself
the payload after the BtrfsInodeRef

We omit get_inode_ref()s implementation for brevity. It’s essentially the same code as this function except it searches for BtrfsInodeRefs.

                if current_key.offset == current_inode_nr {
                    path_prefix.insert(0, '/');
                    break;
                }

Check if we’ve reached the root of the filesystem. For BtrfsInodeRefs, the BtrfsKey::offset field holds _current_inode’s parent’s inode number. If the parent inode # and the current inode # match, it means we’re at the root of the filesyste.

If we’re at the root, insert a / to path_prefix to root the absolute path and exit the loop.

                path_prefix.insert_str(
                    0,
                    &format!("{}/", std::str::from_utf8(&current_inode_payload)?),
                );

If we’re not yet at the root, we interpret BtrfsInodeRef’s payload as a string containing the name of the inode. We tack that onto the front of path_prefix.

                current_inode_nr = current_key.offset;
            }

The last thing we do in the loop is set current_inode_nr to its parent’s inode number. This ensure we keep moving closer to the root of the filesystem.

            println!("filename={}{}", path_prefix, name);
        }

Finally, after the loop exits, we combine path_prefix and name to get the absolute path of the regular file we’re processing. We print the result to the terminal.

    } else {
        let ptrs = tree::parse_btrfs_node(node)?;
        for ptr in ptrs {
            let physical = cache
                .offset(ptr.blockptr)
                .ok_or_else(|| anyhow!("fs tree node not mapped"))?;
            let mut node = vec![0; superblock.node_size as usize];
            file.read_exact_at(&mut node, physical)?;
            walk_fs_tree(file, superblock, &node, root_fs_node, cache)?;
        }
    }

This really ought to bore you now. If we’re at an internal node, recursively process each of the child nodes.

    Ok(())
}

And finally, if we’ve reached the end of the function, we’re done.

Conclusion

Congrats on making it this far in the series! Creating and writing up btrfs-walk was extraordinarly educational for me and hopefully for you as well.

Note that we’ve just barely scratched the surface of all the complexity btrfs contains. But with a solid understanding of the core data structures, you should feel comfortable diving deeper into btrfs internals on your own.

If you’re still interested in btrfs, take a look at btrfs-fuzz. It’s an unsupervised coverage-guided btrfs fuzzer that I’m creating using what I’ve learned from btrfs-walk.

Understanding btrfs internals part 4

2020-11-15T18:27:00-08:00

Understanding btrfs internals part 4 Home

Understanding btrfs internals part 4

This is the fourth of a multipart series that explains the basics of btrfs’s on-disk format.

At the end of this series, we’ll have a program that can print out the absolute path of every regular file in an unmounted btrfs filesystem image without external libraries or ioctl(2) calls.

Example code is available here.

Background

Part 3 went into detail on how to walk an on-disk B-tree given the logical address of a root node. However, not all tree roots are stored in the superblock. To access any tree that isn’t the chunk tree or the log tree, we must locate the relevant root using the root tree root.

The root tree root contains references to all the other trees:

The superblock contains the logical address for the root tree root node. The root tree root is a leaf node that contains BtrfsRootItems as payloads. The BtrfsRootItems contain references to other tree roots. Note that the solid lines are direct references insofar as the reference is a BtrfsKeyPtr. The dashed line is a reference through other means.

Note that the root tree may also contain other item types. They’re not particularly interesting for our tool so we’ll ignore them.

Reading the root tree root

First, we read the root tree root. This should be fairly straightforward by now:

fn read_root_tree_root(
    file: &File,
    root_tree_root_logical: u64,
    cache: &ChunkTreeCache,
) -> Result<Vec<u8>> {
    let size = cache
        .mapping_kv(root_tree_root_logical)
        .ok_or_else(|| anyhow!("Root tree root logical addr not mapped"))?
        .0
        .size;
    let physical = cache
        .offset(root_tree_root_logical)
        .ok_or_else(|| anyhow!("Root tree root logical addr not mapped"))?;

    let mut root = vec![0; size as usize];
    file.read_exact_at(&mut root, physical)?;

    Ok(root)
}

where root_tree_root_logical is available in BtrfsSuperblock::root.

Now that we’ve got the root tree root, it’s time to extract the filesystem tree root from it.

Reading the filesystem tree root

First, let’s define BtrfsRootItem:

#[repr(C, packed)]
#[derive(Copy, Clone)]
pub struct BtrfsRootItem {
    pub inode: BtrfsInodeItem,
    pub generation: u64,
    pub root_dirid: u64,
    pub bytenr: u64,
    pub byte_limit: u64,
    pub bytes_used: u64,
    pub last_snapshot: u64,
    pub flags: u64,
    pub refs: u32,
    pub drop_progress: BtrfsKey,
    pub drop_level: u8,
    pub level: u8,
    pub generation_v2: u64,
    pub uuid: [u8; BTRFS_UUID_SIZE],
    pub parent_uuid: [u8; BTRFS_UUID_SIZE],
    pub received_uuid: [u8; BTRFS_UUID_SIZE],
    /// updated when an inode changes
    pub ctransid: u64,
    /// trans when created
    pub otransid: u64,
    /// trans when sent. non-zero for received subvol
    pub stransid: u64,
    /// trans when received. non-zero for received subvol
    pub rtransid: u64,
    pub ctime: BtrfsTimespec,
    pub otime: BtrfsTimespec,
    pub stime: BtrfsTimespec,
    pub rtime: BtrfsTimespec,
    pub reserved: [u64; 8],
}

To walk the root tree root, we can use the same algorithm as we used when we walked the chunk tree root. Note that this is the benefit of using the same B-tree data structure to store all btrfs data and metadata (reusable algorithms).

fn read_fs_tree_root(
    file: &File,
    superblock: &BtrfsSuperblock,
    root_tree_root: &[u8],
    cache: &ChunkTreeCache,
) -> Result<Vec<u8>> {
    let header =
        tree::parse_btrfs_header(root_tree_root)
            .expect("failed to parse root tree root header");

    if header.level != 0 {
        bail!("Root tree root is not a leaf node");
    }

Extract the header and see if we’re at a leaf node. The root tree root must be a leaf node so bail if it’s not.

    let items = tree::parse_btrfs_leaf(root_tree_root)?;
    for item in items.iter().rev() {
        if item.key.objectid != BTRFS_FS_TREE_OBJECTID
              || item.key.ty != BTRFS_ROOT_ITEM_KEY {
            continue;
        }

Now, find the first (and only) BtrfsItem in the node that’s the filesystem tree root.

        let root_item = unsafe {
            &*(root_tree_root
                .as_ptr()
                .add(std::mem::size_of::<BtrfsHeader>() + item.offset as usize)
                as *const BtrfsRootItem)
        };

Once we have the &BtrfsItem, we want to extract the payload. BtrfsItem::offset is the offset from the end of the BtrfsHeader. The above code does the math to pull out the BtrfsRootItem associated with the current BtrfsItem.

        let physical = cache
            .offset(root_item.bytenr)
            .ok_or_else(|| anyhow!("fs tree root not mapped"))?;
        let mut node = vec![0; superblock.node_size as usize];
        file.read_exact_at(&mut node, physical)?;

        return Ok(node);
    }

Now read the filesystem tree root.

    bail!("Failed to find root tree item for fs tree root");
}

Finally, if the loop has finished and we haven’t returned yet, we return an error saying we couldn’t find the filesystem tree root. This should never happen for a proper btrfs image.

We’re almost done! We have the filesystem tree root. All that’s left is to walk the filesystem tree and make sense of the data we care about. In our case, we want the absolute path of every regular file. In the next post, we’ll finish off our project.

libbpf-rs: eBPF for the Rust ecosystem

2021-01-04T17:24:15-08:00

libbpf-rs: eBPF for the Rust ecosystem Home

libbpf-rs: eBPF for the Rust ecosystem

Summary

libbpf-rs provides safe Rust bindings to libbpf as well as a cargo plugin to help you develop “Compile Once - Run Everywhere” eBPF applications. Use libbpf-rs over BCC to produce smaller, more efficient, and more predictable applications.

Background

eBPF (BPF) programs have traditionally been written using BPF compiler collection (BCC). The application would then call into BCC using the C++ or Python bindings. BCC has an “on-the-fly” model, meaning BPF programs are compiled during runtime on production hosts. While this works, there are drawbacks:

Every production machine needs kernel headers installed. These can get out of sync and can be missing internal headers.
LLVM/clang is heavyweight in both binary size and resource consumption. This is because BCC needs to compile the BPF program (written in C) down to BPF bytecode the kernel can understand.
End results can vary with kernel version and configuration. The BPF program can and will change depending on the host it’s running on.

This diagram illustrates BCC’s compilation and runtime model:

Compile Once - Run Everywhere

Compile Once - Run Everywhere (CO-RE) is a development effort solves the above issues by enabling ahead of time BPF program compilation. A summary of developments include:

BPF type format (BTF). You can think of BTF as a very light weight alternative to DWARF; light enough that you can write a parser in an afternoon. When the kernel is compiled with CONFIG_DEBUG_INFO_BTF=y, the final kernel image contains BTF entries for all internal data structures. Type definitions can then be extracted from BTF, obviating the kernel header dependency.
Compiler (clang) support for relocatable field accesses. Consider accessing field bar from struct Foo. In BCC’s model, BCC doesn’t need to worry about struct Foo’s structure layout changing. That’s because BCC compiles the program against the installed headers on every target host. This becomes an issue for CO-RE BPF progs. CO-RE BPF progs must have their field accesses relocated so that they reflect the target host’s structure layouts.
BPF loader (libbpf) support for BPF program fixups. libbpf must “custom-tailor” the BPF program for the target host it’s running on.

More in-depth documentation as well as user facing APIs is available here.

This diagram illustrates the CO-RE compilation and runtime model:

Enter Rust

Why Rust? Rust’s appeal (to systems programmers like myself) is its emphasis on safety without compromising performance or expressiveness. As such, there are powerful facilities for library designers to build abstractions that resist misuse. This, coupled with the kernel’s guarantee that a verified BPF program (modulo bugs) can never crash, hang or interfere with the kernel negatively makes Rust + BPF a very attractive combination.

libbpf-rs provides safe (w.r.t. Rust’s unsafe keyword) bindings to libbpf. On top of that, libbpf-rs is designed such that if your code can compile, you’re almost certainly using the library correctly. Much of that guarantee comes from liberal use of newtype and builder APIs.

libbpf-cargo is a cargo plugin that integrates with cargo workflows Rust programmers are already familiar with.

Together, libbpf-rs and libbpf-cargo provide CO-RE support in the Rust ecosystem.

Rust skeleton

If you want to skip ahead and browse the unabridged example, look here.

BPF skeletons started out as an alternative interface to libbpf. Its goal was to simplify and reduce boilerplate when developing BPF applications. It ended up being so successful that it’s now the recommended interface. Naturally, libbpf-rs supports Rust BPF skeletons.

Given BPF object files (eg myobj.bpf.o), cargo libbpf gen can generate Rust skeletons for your object files. Consider runqslower: a prototypical BPF application. runqslower shows high latency scheduling times between tasks being ready to run and them running on CPU after that. Below is an abridged copy of runqslower.bpf.c (the BPF program bits):

const volatile __u64 min_us = 0;
const volatile pid_t targ_pid = 0;
const volatile pid_t targ_tgid = 0;

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 10240);
    __type(key, u32);
    __type(value, u64);
} start SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
    __uint(key_size, sizeof(u32));
    __uint(value_size, sizeof(u32));
} events SEC(".maps");

SEC("tp_btf/sched_wakeup")
int handle__sched_wakeup(u64 *ctx);

SEC("tp_btf/sched_wakeup_new")
int handle__sched_wakeup_new(u64 *ctx);

SEC("tp_btf/sched_switch")
int handle__sched_switch(u64 *ctx);

To compile the object:

$ cd $RUST_PROJECT_ROOT

$ # Program sources must be placed in `src/bpf`
$ find . -name runqslower.bpf.c
./src/bpf/runqslower.bpf.c

$ # If you haven't already
$ cargo install libbpf-cargo

$ cargo libbpf build

To generate the skeleton:

$ cargo libbpf gen

$ # Generated skeleton files are placed in `src/bpf`
$ find . -name runqslower.skel.rs
./src/bpf/runqslower.skel.rs
$ find . -name mod.rs
./src/bpf/mod.rs

The following is example code that drives the skeleton:

mod bpf;
use bpf::*;

fn main() -> Result<()> {
    // Open skeleton
    let mut skel_builder = RunqslowerSkelBuilder::default();
    let mut open_skel = skel_builder.open()?;

    // Write arguments into prog
    open_skel.rodata().min_us = 42;  // The answer to everything
    open_skel.rodata().targ_pid = 1;
    open_skel.rodata().targ_tgid = 1;

    // Load skeleton into kernel
    let mut skel = open_skel.load()?;

    // Attach progs to attach points
    skel.attach()?;

    // Set up ring buffer processing
    let perf = libbpf_rs::PerfBufferBuilder::new(skel.maps().events())
        .sample_cb(handle_event)
        .lost_cb(handle_lost_events)
        .build()?;

    // Process entries from ring buffer
    loop {
        perf.poll(core::time::Duration::from_millis(100))?;
    }
}

fn handle_event(cpu: i32, data: &[u8]) {
    unimplemented!();
}

fn handle_lost_events(cpu: i32, count: u64) {
    unimplemented!();
}

Note there’s no mention of the runqslower.bpf.o file anywhere. That’s because the contents of the object file have been embedded into the skeleton file (using include_bytes!() macro). As a result, you don’t have to worry about shipping BPF object files along with your application binaries.

Also note how open_skel.rodata() and skel.maps() is infallible. Generated skeletons have infallible accessors for all maps, programs, and global variables. This reduces the number of errors your application can encounter at runtime.

Finally, note how we can set and get values for global data. cargo libbpf gen can read the BTF (that clang generates for the object file) and generate appropriate Rust definitions. These structures are then mmap()ed to the kernel at runtime. Global data is the most convenient way to communicate with BPF programs from userspace. Simply read and write values like usual. Note that OpenRunqslowerSkel::rodata() returns a mutable reference to the read-only data and RunqslowerSkel::rodata() returns an immutable reference. That’s because constant values can only be set before the program is loaded into the kernel. Safety at work!

To build the application:

$ cargo libbpf make
...
$ sudo ./target/debug/runqslower
...

Conclusion

If you have questions (you most likely do) about libbpf-rs, please consult the official documentation:

libbpf-rs docs.rs
libbpf-cargo docs.rs and cargo libbpf --help
runqslower docs

If you have more questions / concerns after that, please reach out to me in any way possible.

C++ patterns: static registration

2021-02-19T11:42:43-08:00

C++ patterns: static registration Home

C++ patterns: static registration

Plugin architectures are useful: it’s hard to predict future needs in the present. Thus, it’s often useful to punt as much business logic as possible to the future. But how do we create a robust and scalable plugin architecture?

A naive approach is a large if-block. Suppose you have a configuration file where the user specifies which plugin he or she wants by name:

config.toml

plugin=MyPlugin

The easiest way to support selecting plugins could look something like:

if (plugin_name == "default") {
    return std::make_unique<Default>();
} else if (plugin_name == "myplugin") {
    return std::make_unique<MyPlugin>();
} else {
    return nullptr;
}

You could imagine this code exists in the “core” codebase. However, this is not scalable. Every time a developer authors a plugin (or renames an existing) plugin, the core implementation needs to change. It would be far better if we could make the addition, change, or removal of plugins generic.

The static registration pattern accomplishes this. We take advantage of the fact that static variables are initialized before main() is reached. Furthermore, static variables may call other static methods. [citation?] Consider a plugin “registry” implemented like so:

Registry.h

#define REGISTER_PLUGIN(plugin_name, create_func) \
    bool plugin_name ## _entry = PluginRegistry<Plugin>::add(#plugin_name, (create_func))

template <typename T>
class PluginRegistry {
  public:
    typedef std::function<T*()> FactoryFunction;
    typedef std::unordered_map<std::string, FactoryFunction> FactoryMap;

    static bool add(const std::string& name, FactoryFunction fac) {
      auto map = getFactoryMap();
      if (map.find(name) != map.end()) {
        return false;
      }

      getFactoryMap()[name] = fac;
      return true;
    }

    static T* create(const std::string& name) {
      auto map = getFactoryMap();
      if (map.find(name) == map.end()) {
        return nullptr;
      }

      return map[name]();
    }

  private:
    // Use Meyer's singleton to prevent SIOF
    static FactoryMap& getFactoryMap() {
      static FactoryMap map;
      return map;
    }
};

Notice how PluginRegistry is completely generic. It can hold factory methods for any type. On line 1, we define a macro that specializes PluginRegistry for a class Plugin, which we’ll pretend is trivial.

In this manner, when a developer authors a plugin, registration is trivial.

RealPlugin.h

class RealPlugin : public Plugin {
  public:
    void doWork() override {
      std::cout << "I\'m doing real work!\n";
    };

    static Plugin* create() {
      return new RealPlugin();
    }
};

REGISTER_PLUGIN(RealPlugin, RealPlugin::create);

And in Main.cpp, we can instantiate RealPlugin like this:

p = PluginRegistry<Plugin>::create("RealPlugin");
p->doWork();

And so, our nasty if block turns into:

return std::unique_ptr<Plugin>(PluginRegistry<Plugin>::create(plugin_name));

Note on linking

Particularly astute readers might wonder: will the linker garbage collect my static plugin variables (ie bool *_entry = ...)? The answer is maybe, depending on how you’re building your program. Because the “core” program does not “touch” any symbols in plugin translation units, some linkers assume that it is safe to garbage collect the plugin TU. In most cases, this is safe and ok and a worthy optimization. However in our case, this is unwanted.

The solution is to pass to the linker a “-lwhole-achive” [citation?] flag.

Switching to Caddy

2021-03-20T15:10:49-07:00

Switching to Caddy Home

Switching to Caddy

I decided to upgrade my ancient Fedora 25 VPS to Fedora 33 this weekend. I couldn’t get the in-place upgrade to work across so many releases so I just blew away the entire server and started fresh. While typing away, I remembered I had heard some good stuff about Caddy. I was also getting annoyed manually refreshing my Let’s Encrypt certs every three months for lighttpd. I figured why not give Caddy a try?

Turns out caddy is a joy compared to lighttpd. Compare my old lighttpd notes to the new caddy config:

lighttpd:

Caddyfile:

dxuuu.xyz

root * /var/www/dxuuu.xyz
file_server

And caddy automatically renews https certs!

One final note: I originally chose lighttpd out of resource concerns b/c I’m running this site on a $5/mo DigitalOcean droplet. A brief inspection reveals caddy is doing just fine:

[dxu@shiva ~]$ systemctl status caddy
● caddy.service - Caddy web server
     Loaded: loaded (/usr/lib/systemd/system/caddy.service; enabled; vendor preset: disabled)
     Active: active (running) since Sat 2021-03-20 21:53:36 UTC; 12min ago
       Docs: https://caddyserver.com/docs/
    Process: 6395 ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile (code=exited, status=>
   Main PID: 6401 (caddy)
      Tasks: 7 (limit: 1125)
     Memory: 29.1M
        CPU: 227ms
     CGroup: /system.slice/caddy.service
             └─6401 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Debug

2021-05-19T21:25:18-07:00

Debug Home

Debug

This page is used for debugging.

DWARF stack walks and fast symbolization for BPF

2021-10-12T18:05:57-07:00

DWARF stack walks and fast symbolization for BPF Home

DWARF stack walks and fast symbolization for BPF

10/12/21 update: I have no immediate plans to work on this b/c I’ve recently switched jobs. If you’re really dying to see this implemented maybe we can work something out. Just email me.

Background

In this document we are proposing solutions for two separate but related problems: stack walking and symbolization. We’ll first go over stack walking and then symbolization.

Stack walking

A call stack is a stack function names. Whenever a function is called, that function name is (conceptually) pushed onto the stack. When that function returns, that function name is popped. Stalk walking is the act of retrieving a program call stack at a certain point of execution. This is a fairly complicated topic and heavily depends on the programming language and runtime (if applicable). For the purpose of this document, we’ll stick to native languages like C/C++/Rust.

There are a number of ways to walk stacks. For a long time, frame pointer based stack walks were the best way to do it. We won’t go into detail here, but frame pointers basically form a chain on the stack such that you can always find the caller’s frame given the current frame. Unfortunately, for various reasons, frame pointers started to fall out of favor as compilers started performing more complex codegen optimizations.

Thus, DWARF-based stack walks were born. Compilers will now (sometimes) generate unwind information in .eh_frame or .debug_frame ELF sections. By parsing this information, you can walk the stack.

Unfortunately, Linus is not a great lover of DWARF, so there is not and probably will not be in-kernel DWARF support. This is why bpf_get_stackid() and bpf_get_stack() will often return gibberish if frame pointers are not built into the userspace application. Thus, a different solution is needed.

Symbolization

Before we move onto solutions, we still need to discuss symbolization. Conceptually, symbolization is very simple. Computers deal with addresses when calling functions. Assuming we sucessfully obtained a call stack as outlined in the previous section, we still need to convert the list of addresses into human readable function names. This process is called symbolization.

A symbolizer takes as input an address along with a process ID or executable and returns the function name (and maybe line number) as output. There are plenty of symbolizers to choose from includig llvm-symbolizer and addr2line. llvm-symbolizer suffers from huge memory consumption issues and addr2line is fairly slow. With modern tools and libraries (e.g. Rust), it should be possible to construct a fast, efficient, and portable symbolizing library that covers all known edge cases. Whether we can execute on this remains to be seen.

In practice, this is a tricky problem. You must deal with inlined functions, VDSO symbols, deleted-but-still-running binaries, debug links, split debug info, and more.

Proposed design

The proposed design for the BPF ecosystem is two-fold:

A libstackwalker library that can DWARF-walk a chunk of stack
A libsymbolize library that can symbolize addresses

For (1), given that we can’t perform the DWARF walk in kernelspace, we must perform it in userspace. BPF programs will copy out a chunk of the stack (the exact size can be adaptively sized, as we should be able to know in userspace if more stack is required). libstackwalker will then perform the stack walk on that buffer of memory. There will need to be some level of coordination as to whether an executable has frame pointers and can have its stack walked in kernelspace or does not have frame pointers and must be libstackwalkered.

For (2), we will implement libsymbolize (which is another conceptually simple task). On top of that, we will add to bpftrace a dlopen()-based symbolization framework where users can plug and play different symbolizers. This is important because it decouples bpftrace from language/runtime changes. For example, if you were to implement a python symbolizer today, you’d need to reach into the guts of the python runtime (b/c there’s no standard interface) that can change at any time. By decoupling bpftrace from the symbolizer, we can allow python symbolization matched to the python runtime. The decoupling also enables remote symbolization for more controlled environments.

Status

This project is currently in the planning phase. Status updates will be available here as progress is made.

Ahead-Of-Time compiled bpftrace programs

2021-10-17T15:14:42-07:00

Ahead-Of-Time compiled bpftrace programs Home

Ahead-Of-Time compiled bpftrace programs

This page serves as a design document for bpftrace AOT compilation support. Design is currently a work-in-progress and will be (somewhat) regularly updated.

Background

AOT is motivated by the following technical concerns:
- Shipping/running LLVM onto prod hosts is resource-heavy (binary size, memory consumption)
- Time to first trace is slow for dynamic bpftrace b/c we have to do all the AST passes AND setup runtime resources, where for AOT bpftrace we can just do runtime resource setup
- Dynamically compiled bpftrace scripts cannot be easily signed

Overall design

Ship a fully executable runtime shim with bpftrace
When compiling a AOT bpftrace program:
- Build the metadata
- Build the bytecode
- Make a copy of runtime shim and store metadata + bytecode into a special ELF section (this is the final executable)
When the shim runs, it knows to look inside itself for the metadata + bytecode and start execution

Current architecture

Proposed architecture

AST passes:

Simplified architecture (some unchanged parts omitted):

Key:

Green -> added to all codepaths
Orange -> added to AOT compile codepath

AOT execution:

Unsolved problems

Create fully static aot-rt binary
Generate CO-RE field access instructions
- Will need to generate some kind of access identifier (eg. 1:0:3:4)
Runtime still use bcc (eg. symbolization) and bcc links LLVM
Punt all loading to libbpf to gain access to CO-RE features?
- Map creation, map FD fixup, extern symbol resolution
- Reason is that some new bpf helpers (bpf_per_cpu_ptr()) require BTF ID to be in immediate operand of certain instructions, and we may not want to redo/maintain that stuff in bpftrace
- Another reason is that in the future we may want to link other BPF programs in (like DWARF-based stack walker)
- Reasons not to completely switch over to libbpf include:
  - Large amount of bpftrace code modifications
  - Loss of ability to mock out maps / unit testing
  - Extra overhead constructing ELF files for libbpf to work with
  - If there’s ever a bug in libbpf or we need a new feature, we are beholden to distros shipping an updated libbpf. We cannot vendor either as it’s not going to fly with the distros.

Notes

Can save metadata into special ELF section; fortunately we don’t need to worry about compatability as an AOT executable is hermetic
Must ship a stubbed (no bytecode) AOT executable that knows to look inside itself for bytecode
- Should be simple enough with cmake
Will need to relocate pseudo-map-FDs at runtime to FDs of created maps (see BPF_PSEUDO_MAP_FD in libbpf)
Create RequiredResources abstraction
- That:
  - Describes all the runtime resources that need to be setup before a script is run
  - Can initialize all the resources given a BPFtrace &
    - Or should RequiredResources contain live map FDs too?
  - Can be serialized/deserialized
- Can just be a public field: BPFtrace::resources
  - So it can be easily mocked out for tests
  - So semantic analyser can add resources

Future goals

Re-enable all features block by portability analyser
User can select features to enable in codegen
- eg. “tell codegen that the target host has XXX feature”
Emitted bytecode takes advantage of CO-RE to be more compatible on other hosts
Support -k[k] error helper flags (runtime state currently constructed in codegen)

Towards a functional prototype

Eliminate hard-coded map FDs in bytecode
- PR #1861
Add AST pass to check for non-portable features
- PR #1871
Encapsulate runtime resource requirements
- PR #1883
Create serialization/deserialization routines for RequiredResources
- PR #1893
Create runtime shim
- PR #1950

Slow march to feature completeness

Big ticket

Shim injection / shim extraction
Fully static shim
Re-enable portability analyser disabled features

Test suite

Fix AOT build/tests with older BCC versions
- PR #2010
Completely remove dependency on LLVM
- PR #2031
Reuse as much of existing runtime tests as possible for AOT
- PR #2035
Print out attached_probes message
- PR #2037

Rewriting clipd

2021-11-28T20:40:50-08:00

Rewriting clipd Home

Rewriting clipd

clipd is a distributed clipboard I wrote back in 2017. Distributed in this context means that a server somewhere runs the clipd server and accepts requests from clients to either:

Store new data into the clipboard, or
Grab the current data in the clipboard

This solved an annoying problem I had back then where I had too many computers (still do, actually) and had difficulty sharing links between them. This was not a very complicated piece of software (I think it took maybe two weekends). Nonetheless, it still bitrot over the years and recently stopped working for reasons I was too lazy to debug. I did, however, take it as a chance to rewrite the code from Python to Rust, along with a bunch of other improvements along the way.

It took about 6-8 hours to do the full rewrite and after having done so, I thought it would be interesting to document the different design choices I’ve made. Put differently, why current-me is better at programming than old-me four years ago.

Language and framework

We’ll ignore the client in this section because the client is very simple in both implementations: open a connection, write some bytes, read some bytes, print. The server is more interesting to discuss.

Old server

The old server was written in Python and used socketserver.TCPServer with socketserver.ThreadingMixIn, meaning each connection was handled in a separate thread.

New server

For the new server, I used async Rust with tokio and tokio::TcpListener on a single thread to drive the event loop.

Commentary

Python as a language was an OK choice. Using socketserver is fine as well, thought I might have chosen asyncio if I had to do things over in Python. I’m not sure what the state of asyncio was back then.

I quite like Rust these days. Rust is pretty easy to write and reason about, plus the memory and type safety is extremely appealing for a project I have no intention of ever writing tests for. The extra efficiency from being a compiled language is nice as well. Using an async framework seems like a good choice since clipd is entire I/O bound.

Protocol

clipd was and still is TCP-based protocol. TCP remains an excellent choice because I really don’t want to deal with an unreliable connection. On top of TCP, we have our own framed protocol. Framing is obviously necessary because TCP is a streaming protocol, not a message based one. And delimiters seems troublesome given we support arbitrary payloads.

Old protocol

`clipd` communicates over TCP using ASCII encoded data (please don't hate me,
utf-8 people). The wire format is as follows:

+---+------+-------+
|LEN|HEADER|PAYLOAD|
+---+------+-------+

where:

LEN is the total number of characters in the message (LEN excluded).
HEADER can be either "PUSH" or "PULL" for requests and "OK" or "ERR" for responses
PAYLOAD is either empty, contains the clipboard payload, or the error message

New protocol

clipd protocol (request and response):

0       8      16
+-------+-------------------------+
| magic | type | optional payload |
+-------+-------------------------+

The payload, if present:

0     64
+-----+---------+
| len | payload |
+-----+---------+

where:

magic is a clipd-specific magic value (integer)
type is the type of request (integer)
len is length of payload
payload is opaque series of bytes

Commentary

Thinking about the old protocol now, I have a couple immediate thoughts:

Using ASCII is plain bad because UTF-8 is clearly superior
LEN is a strange field because any wire protocol usually has a fixed set of fields in the front and length is only needed if there is a variable sized field. Although not wrong, the placement is strange
The text-based HEADER field is extremely strange, as it’s extra bytes and could be replaced by an integer

One thing I learned from staring at the output from the old clipd server is that there are a lot of scanners on the internet that will try probing (sending bytes to) any open port on your server. The magic in the new protocol is there to quickly drop invalid requests. The type in the new protocol is now an integer to be more efficient. Similarly, the length field is now only included when necessary.

Protocol parsing

Protocol parsing is, well, parsing the bytes that come over the wire into the protocol we have designed in the previous section. Although this needs to happen in both the server and the client, the process is similar enough in both binaries to describe together.

Old process

The raw bytes are both constructed and processed inline with the application level logic. Meaning something like the following:

req = str(len(HDR_PULL)) + HDR_PULL
req = bytes(req, 'ascii')
resp = _sock_send_recv(...)

hdr, payload = _parse_resp(resp)
if (hdr != HDR_OK):
  raise ClipdException(payload)
return payload

New process

The new code is structured to have a protocol “library” that hides the details of how the requests and responses are sent over the wire. This “library” is used in both the client and server. The library more or less looks like:

pub enum RequestFrame {
    /// A request to push bytes to the server
    Push(Vec<u8>),
    /// A request to pull the currently stored bytes on the server
    Pull,
}

impl RequestFrame {
    pub async fn from_socket(socket: &mut TcpStream) -> Result<RequestFrame>;
    pub fn to_bytes(&self) -> Vec<u8>;
}

ResponseFrame has an symmetric API except with different enum variants.

Commentary

The new way is obviously superior looking back after the fact. Mixing protocol level details with the application logic leads to more complexity (not necessarily in this simple application, but the principle still applies) as well as duplicated logic in both client and server. It’s a clear win to separate concerns and share code between the client and server.

Operation in production

“Operation” and “production” in this context means “how it’s run” and “on my server”, respectively.

Old way

I ran the clipd server binary inside a session of tmux.

New way

I run clipd as a systemd system service.

Commentary

The old way was simpler (not by much) but easier to mess up. For example if the server was restarted I would have to remember to run clipd again.

The new way is clearly better b/c systemd is designed to manage system daemons (among other things). I suspect I didn’t know systemd well enough back then to know to make a systemd service.

Final commentary

After the rewrite clipd feels much snappier and seems to work well again. I found the process of exploring my old code interesting enough that I spent even more time to write a post. So in a way I suppose I’m doubly glad I finally got around to it.

Pull request review (prr)

2022-05-08T18:15:25-07:00

Pull request review (prr) Home

Pull request review (prr)

Last month I developed a new tool that brings mailing-list style code reviews to github. The tool was born out of frustration with github’s point and click web UI. Since I do a lot of code review, the tabbing between my text editor and my browser became a constant source of friction.

Ideally, I wanted to stay in my text editor so that my code writing and code reviewing workflows could be unified. I didn’t find any such tool on the internet so I built my own.

Demo

Consider the following pull request:

To download the PR for review:

$ prr get danobi/prr-test-repo/2
/home/dxu/dev/review/danobi/prr-test-repo/2.prr

When you open the review file, you’ll see:

> diff --git a/ch4.txt b/ch4.txt
> index 41e1932..8741175 100644
> --- a/ch4.txt
> +++ b/ch4.txt
> @@ -16,6 +16,10 @@ CHAPTER 4. TACTICAL DISPOSITIONS
>
>  8. To see victory only when it is within the ken of the common herd is not the acme of excellence.
>
> +HERE IS SOME CONTENT I'M TRYING TO ADD
> +LALALALLALALALALAL
> +LALAL
> +
>  9. Neither is it the acme of excellence if you fight and conquer and the whole Empire says, "Well done!"
>
>  10. To lift an autumn hair is no sign of great strength; to see the sun and moon is no sign of sharp sight; to hear the noise of thunder is no sign of a quick ear.

Now suppose you want to have the author remove the extraneous edits and reject the PR because of it. Simply mark up the review file as follows:

@prr reject

> diff --git a/ch4.txt b/ch4.txt
> index 41e1932..8741175 100644
> --- a/ch4.txt
> +++ b/ch4.txt
> @@ -16,6 +16,10 @@ CHAPTER 4. TACTICAL DISPOSITIONS
>
>  8. To see victory only when it is within the ken of the common herd is not the acme of excellence.
>

> +HERE IS SOME CONTENT I'M TRYING TO ADD
> +LALALALLALALALALAL
> +LALAL
> +

Remove this junk

>  9. Neither is it the acme of excellence if you fight and conquer and the whole Empire says, "Well done!"
>
>  10. To lift an autumn hair is no sign of great strength; to see the sun and moon is no sign of sharp sight; to hear the noise of thunder is no sign of a quick ear.

Two things to note:

We can use prr directives by “tagging” @prr. Just like how you might use a github bot.
We added a “spanned” comment on the hunk by adding a newline before the start of the span.

To submit the review back to github, run:

$ prr submit danobi/prr-test-repo/2

You’ll now see something like this on github:

(Note that you don’t see the “Changes requested” in this example because github doesn’t let you request changes on your own PR. It should work on a real PR.)

Looking forward

The above demo shows off most of the features I’ve currently implemented. There’s still a few things I want to add but they’re not too pressing as I’m currently using prr every day just fine.

If you wanna try out prr or request new features, feel free to file an issue or open a PR at https://github.com/danobi/prr .

BPF and go pprof

2022-09-10T14:49:39-07:00

BPF and go pprof Home

BPF and go pprof

Earlier this week I was made aware of what appeared to be an infinite loop in the in-kernel BPF verifier. Golang stacks were pulled, flamegraphs were captured (shoutout to profiler.firefox.com), and by all appearances it seemed like there was indeed an infinite loop.

I spent some time studying the verifier source and nothing really stood out to me as infinite loopable. Plus, the flamegraph showed a rather equal distribution of callees so the verifier being stuck in a single place seemed unlikely.

By some random chance I thought it would be useful to see how often the verifier was being called. I answered that question by running the following bpftrace script for a few seconds:

$ sudo ./bpftrace -e 't:syscalls:sys_enter_bpf { @ = count(); }'
Attaching 1 probe...
^C

@: 932

So clearly this was not an infinite loop but rather bpf(BPF_PROG_LOAD, ...) being called in a tight loop. That by itself seemed rather odd so I took to the cilium/ebpf source. A quick dig turned up:

// BPF wraps SYS_BPF.
//
// Any pointers contained in attr must use the Pointer type from this package.
func BPF(cmd Cmd, attr unsafe.Pointer, size uintptr) (uintptr, error) {
    for {
        r1, _, errNo := unix.Syscall(unix.SYS_BPF, uintptr(cmd), uintptr(attr), size)
        runtime.KeepAlive(attr)

        // As of ~4.20 the verifier can be interrupted by a signal,
        // and returns EAGAIN in that case.
        if errNo == unix.EAGAIN && cmd == BPF_PROG_LOAD {
            continue
        }

        var err error
        if errNo != 0 {
            err = wrappedErrno{errNo}
        }

        return r1, err
    }
}

Ah ha, so the looping condition is -EAGAIN. I quickly verified that by running:

$ sudo ./bpftrace -e 't:syscalls:sys_exit_bpf { @[args->ret] = count(); }'
Attaching 1 probe...
^C

@[-11]: 543

-EAGAIN is -11, so great. That’s confirmation.

To answer why the verifier would ever return -EAGAIN, I grepped through kernel/bpf/verifier.c. It turns out there’s precisely one occurence of -EAGAIN in the file (which contains all verifier logic):

static int do_check(struct bpf_verifier_env *env)
{
        /* [...] */
        for (;;) {
                struct bpf_insn *insn;

                /* [...] */
                insn = &insns[env->insn_idx];
                class = BPF_CLASS(insn->code);

                /* [...] */
                if (signal_pending(current))
                        return -EAGAIN;

                /* Main logic follows */

So the only reason the verifier would return -EAGAIN is if there is a pending signal waiting for the caller.

In general this would be pretty difficult to debug b/c any code modification could potentially disturb the bad state. Fortunately, we can turn to bpftrace:

$ sudo ./bpftrace -e 't:syscalls:sys_exit_bpf { @[args->ret] = count(); } t:signal:signal_deliver / pid == $1 / { @sigs[args->sig] = count() }' $(pidof binary-under-investigation)
Attaching 2 probes...
^C

@[-11]: 876
@sigs[27]: 1136

Here lies the smoking gun: signal 27 is SIGPROF. From here, two things were realized:

Go’s built in CPU profiling uses SIGPROF to sample
We recently turned on continuous pprof based profiling

In other words, what was happening was:

The verifier was taking time for verification to complete
While the verifier was working, pprof was sending SIGPROF at high enough frequency to prevent verifier from completing its work
cilium/ebpf kept retrying bpf(BPF_PROG_LOAD, ..) because it received -EAGAINs
All the above caused our process to spin on prog verification

Conclusion

This was a really nice bug b/c it’s simple to grok and totally non-obvious.

My only takeaway is that I’m not sure I like go’s signal based profiling mechanism. It intereres quite heavily with long-running syscalls like bpf(BPF_PROG_LOAD, ...). I would much rather prefer using PMU-based profilers like perf.

File capabilities and /proc/<pid>

2022-10-21T07:43:11-07:00

File capabilities and /proc/<pid> Home

File capabilities and /proc/<pid>

I’ve spent the past few evenings staring at a rather interesting kernel “bug”. I put “bug” in quotes b/c, spoiler alert, it’s not a bug.

It all started with an integration test failing due to a -EACCES return code. Normally I’d assume it’s a bug with the code under test, but we also happen to cover the same codepath in unit tests. So a failure when the code was run in a separate binary was something worth investigating.

The details of the test and test infrastructure aren’t important b/c I was able to come up with a short reproducer:

#include <fcntl.h>
#include <stdio.h>

int main() {
  int fd = open("/proc/self/auxv", O_RDONLY);

  if (fd < 0) {
    perror("open");
    return 1;
  }

  printf("ok\n");
  return 0;
}

$ gcc main.c
$ ./a.out                                             # (1)
ok
$ sudo setcap "cap_net_admin,cap_sys_admin+p" a.out   # (2)
$ ./a.out                                             # (3)
open: Permission denied

Basically the above program tries to open /proc/self/auxv and reports on whether it was successful. Pretty simple, and that’s all we do in step (1).

Things get more interesting when we get to step (2). For the unititiated, in step (2) we set a couple file capabilities on the executable. File capabilities (filecaps) are basically the same thing as setuid, except filecaps operate on capabilities which are much more granular than setuid’s “root or not”.

More specifically, we add the capabilities to the permitted set. This is an important distinction b/c it means the executable, when run as an unprivileged process, starts out with no extra capabilities. However, the process may grant itself any of the permitted capabilities (ie. “elevate”). Note how the reproducer does not elevate to any capabilities. In other words, our reproducer is capability-dumb.

Finally, in step (3) we run the reproducer just like in step (1). However unlike in step (1), we get -EACCES. And to be explicit, this is really strange b/c usually programs have extra powers with more capabilities, not less. Furthermore, the process doesn’t even use extra capabilities at runtime.

The stage is set for an interesting investigation: a trivial reproducer and a wildly unexpected result.

Odd ownership

I spent some time trying to trace kernel execution with everyone’s favorite tracer (bpftrace!). Unfortunately, a lot of the interesting logic was either in the body of a long function or in an inlined function call so I had to fall back to printk() debugging.

What followed was a lot of edit-compile-run loops (shoutout to virtme for making testing kernels so easy). I won’t bore you with the details, so suffice to say after a few hours I discovered that /proc/self/auxv was owned by root:root. And since /proc/self/auxv has 0400 permissions, our process running under daniel:daniel fails the UNIX permission check in acl_permission_check().

Christian Brauner later confirms this discovery on the lists.

So -EACCES is explained. But this begs a new question: why is /proc/self/auxv owned by root?

Process dumpability

I’ve looked at procfs internals before so I had an idea where to start looking. It took another few hours and a mixture of printk() and tracing, but I discovered where the procfs inodes get their owners calculated: task_dump_owner().

Around the same time (within minutes actually), Omar Sandoval kindly pointed out on twitter to look at process dumpability.

Process dumpability appears to govern whether or not processes can have their cores dumped. This seems reasonable – you may not want a privileged process’s memory to be dumped to disk if it contains sensitive information.

With the above hint, the search is narrowed down to the following block:

void task_dump_owner(struct task_struct *task, umode_t mode,
                     kuid_t *ruid, kgid_t *rgid)
{
        [...]

        if (mode != (S_IFDIR|S_IRUGO|S_IXUGO)) {
                struct mm_struct *mm;
                task_lock(task);
                mm = task->mm;
                /* Make non-dumpable tasks owned by some root */
                if (mm) {
                        if (get_dumpable(mm) != SUID_DUMP_USER) {
                                struct user_namespace *user_ns = mm->user_ns;

                                uid = make_kuid(user_ns, 0);
                                if (!uid_valid(uid))
                                        uid = GLOBAL_ROOT_UID;

                                gid = make_kgid(user_ns, 0);
                                if (!gid_valid(gid))
                                        gid = GLOBAL_ROOT_GID;
                        }
                } else {
                        uid = GLOBAL_ROOT_UID;
                        gid = GLOBAL_ROOT_GID;
                }
                task_unlock(task);
        }
        *ruid = uid;
        *rgid = gid;
}

Line 1853 checks if the inode is a globally accessible directory. /proc/pid/auxv is a regular file so we enter the body.

Line 1859 then checks for process dumpability. If it’s not globally dumpable (globally dumpable meaning anyone can read the process’s core dumps), the procfs inode is set to be owned by root:root. This decision is returned via ruid and rgid out-params on lines 1876-1877.

File capabilities

Now that we suspect process dumpability is involved, all that’s left to do is figure out who or what is setting process dumpability. To that end, we run the following bpftrace script against the reproducer:

$ cat b.bt
kfunc:set_dumpable
/ comm == "a.out" /
{
  printf("value=%d\n", args->value);
  print(kstack);
}

$ sudo bpftrace ./b.bt
Attaching 1 probe...
value=2

        bpf_prog_f61e3e4e5dc2ac1c_set_dumpable+306
        bpf_get_stackid_raw_tp+119
        bpf_prog_f61e3e4e5dc2ac1c_set_dumpable+306
        bpf_trampoline_6442491445_0+71
        set_dumpable+9
        commit_creds+111
        begin_new_exec+1653
        load_elf_binary+1665
        bprm_execve+639
        do_execveat_common.isra.0+429
        __x64_sys_execve+54
        do_syscall_64+92
        entry_SYSCALL_64_after_hwframe+99

^C

Here we see the smoking gun: __x64_sys_execve+54. This means that dumpability is being set at process creation time. This smells badly of file capability shenanigans. However for completeness, we’ll look at commit_creds() which directly calls set_dumpable():

int commit_creds(struct cred *new)
{
        [...]

        /* dumpability changes */
        if (!uid_eq(old->euid, new->euid) ||
            !gid_eq(old->egid, new->egid) ||
            !uid_eq(old->fsuid, new->fsuid) ||
            !gid_eq(old->fsgid, new->fsgid) ||
            !cred_cap_issubset(old, new)) {
                if (task->mm)
                        set_dumpable(task->mm, suid_dumpable);
                task->pdeath_signal = 0;
                /*
                 * If a task drops privileges and becomes nondumpable,
                 * the dumpability change must become visible before
                 * the credential change; otherwise, a __ptrace_may_access()
                 * racing with this change may be able to attach to a task it
                 * shouldn't be able to attach to (as if the task had dropped
                 * privileges without becoming nondumpable).
                 * Pairs with a read barrier in __ptrace_may_access().
                 */
                smp_wmb();
        }

        [...]
}

Line 475 is what we’ve been after. cred_cap_issubset() is checking is if the old capabilities are a subset of the new capabilities. If so, it means the process has gained capabilities during an execve(2) and process dumpability is adjusted. Again, this checks out b/c the new process may (or already) have elevated capabilities and could contain sensitive information going forward.

Conclusion

So we have our answer: our unprivileged cannot read /proc/self/auxv because it has filecaps assigned. And b/c filecaps can potentially make a process contain sensitive information, the kernel makes its /proc/<pid> inodes owned by root:root.

Although nothing terribly actionable resulted from this investivation, I nonetheless found this to be an interesting case study in emergent properties. Each new discovery made sense in isolation. But despite each step making sense, the final result remains unexpected. And b/c each step makes sense in isolation, I cannot think of a good way to change the final result without making some piece of the puzzle not make sense.

I suppose that’s why documentation is a thing.

And by the way, this behavior is actually documented in proc(2) man page under PR_SET_DUMPABLE.

3D modeling with OpenSCAD

2023-02-13T08:51:39-08:00

3D modeling with OpenSCAD Home

3D modeling with OpenSCAD

The Boring Background

A few months ago I got tired of commuting to the local hacker space so I bought a cheap 3D printer to use at home. Like any good neophyte, I printed quite a few precooked models before getting the itch to design some of my own.

The biggest hurdle was that I couldn’t find good modeling software to run on Linux. I tried FreeCAD and Blender but found them hard to approach. I eventually stumbled upon OpenSCAD which I ended up absolutely loving. So hopefully this is the first of at least a few posts where I dive into some of the techniques I’ve learned.

Spice rack

There’s obviously a lot of useful things you can make with a 3D printer. For my first project, I decided on a spice rack for two reasons:

My spice cabinet is a total mess and it take forever to find anything.
It’s a simple geometry, meaning I could focus most of my attention on learning the tooling.

Here’s the final design for reference:

A quick primer

I’m not really an expert at 3D modeling so I won’t say much about the theory. But based on my limited research, I found OpenSCAD’s marriage between declarative scripting and constructive solid geometry (CSG) to be a wonderfully elegant combination.

The premise behind CSG is that there are a limited set of 2D and 3D primitives such as squares, circles, spheres, cylinders, cubes, etc. which can be combined with with boolean operators to yield complex geometries. For example, with the following code, you can take the union of a cube and a sphere to yield a cube-sphere fusion:

union() {
    cube([5, 5, 5], center=true);
    translate([5, 0, 0])
        sphere(r=3);
}

Expressions are not limited to the top level – they can be arbitrarily nested. Consider this following example:

module ring() {
    difference() {
        circle(r=10);
        circle(r=5);
    }
}

linear_extrude(15) {
    union() {
        translate([15, 0, 0]) ring();
        ring();
    }
}

Before (left) and after (right) extrusion

The basic idea is we first create two rings, one next to the other. We then take the two rings and join them with a union. Finally, take the union and extrude along Z axis to create the binocular shape on the right above.

The other new technique here is a module. A module, taken in the context of C/C++ like languages, is essentially a macro. We don’t use any parameters in the module here, but we could have.

Knowing these basics, we move onto the rack design.

Rack design

To start, we create the backboard:

translate([(225/2), 25, 0])
    cube([225, 5, 15], center=true);

We translate the backboard a bit so that it’s length is easier to eyeball on the X axis and so it’s arms are more easily positioned on the Y axis. Note that OpenSCAD is unitless but my slicer (Cura) interprets the units by default as millimeters.

Next we create the arms:

inner_radius = 20;
outer_radius = inner_radius + 3;

module arm() {
    difference() {
        intersection() {
            circle(outer_radius);
            translate([0, 7.8, 0])
                square([500, 41], center=true);
        }
        circle(inner_radius);
    }
}

for (i=[0:3]) {
    translate([i*56.5 + 28, 2, 0])
        #arm();
}

The arms follow a similar design to the examples above. We make each arm by intersecting two circles and slicing off the end so the spices can fit. And instead of making one, we make four by using a for loop.

One practical tidbit here is we lexically prefix each arm with # to give it a different render color. This is useful for quickly eyeballing changes to your model. In this case, we’ll be using it to illustrate the changes from the previous code snippet.

Once we have the 2D arms, all that’s left is to extrude them to the appropriate height:

module arm() {
    #linear_extrude(15, center=true) {
        // Previous contents of `arm()`
    }
}

At this point we have a functional model. In fact, this is what I have printed and installed in my cabinets. However, there’s still one small improvement we can make: the tips of the arms. The tips are slightly sharp and could use some rounding out. To achieve this, we’ll use a Minkowski Sum.

Minkowski sums are hard to explain through text but surprisingly easy to understand with a visualization. In the previously link, the animation demonstrates that the sum of two nodes involves running the child node around the edges of the parent node and adding the shape that the child node’s motion creates to the parent node.

In our case, we’ll run a small circle around the edges of the arm while also subtracting an additional unit off the inside and outside to keep the final dimensions roughly the same. The end result is the sharp edges are rounded out but the smooth parts remain unchanged.

In code, this looks like:

minkowski_radius = 1;

module arm() {
    linear_extrude(15, center=true) {
        minkowski() {
            difference() {
                intersection() {
                    circle(outer_radius-minkowski_radius);
                    translate([0, 7.8, 0])
                        square([500, 41-minkowski_radius], center=true);
                }
                circle(inner_radius+minkowski_radius);
            }
            circle(minkowski_radius);
        }
    }
}

With this we arrive at the model displayed in the beginning of this post.

For completeness, the following is the physical result. As I mentioned earlier, the physical model doesn’t yet have the rounded edges (the 3M adhesive strips are surprisingly expensive!). So you’ll have to use your imagination a little here.

Conclusion

Obviously this is not a complicated geometry. In fact, a (real) engineer friend of mine was timed creating a similar model using professional tooling and it took him about 90 seconds. Despite that, I still think this was a really enjoyable process. I particularly love how the entire language fits on a small cheat sheet – something unimaginable for industrial grade tools. So I suppose my preference for elegance over practicality is why he’s a real engineer and I’m not.

Generating kernel patchsets

2023-02-27T13:29:18-08:00

Generating kernel patchsets Home

Generating kernel patchsets

Background

The Linux kernel famously has an email workflow. Ignoring whether or not it’s a good thing, it is still a fact that it can be tricky to follow the process.

One problem I’ve had for years is wrangling git-format-patch and git-send-email to do the right thing. Any time I’ve wanted to send a patchset, I’ve had to search through my shell history to help remember the right incantations.

Obviously this gets old fast and becomes a constant source of friction.

Problem

Automation is clearly the answer. A wrapper script over git is always an option but can become overcomplicated over time. I’d rather try to integrate with git than hide it.

The above is not a new idea; there’ve been plenty of previous writing on this. Basically what the two linked approaches suggest is to hook git-send-email to automatically generate To and Cc headers using scripts/get_maintainer.pl.

Unfortunately there are multiple drawbacks of the above approaches:

They do not correctly generate To and Cc headers for cover letters
It is painful to realize the generation is incorrect halfway through sending a large patchset. Half the emails are already sent – you cannot take them back.

After some experimenting, I offer a slightly improved solution.

Solution

The first improvement I’m offering is to generate the headers during git-format-patch time. This allows you to eyeball the results before sending. This also gives you a way to incrementally add more emails.

To that end, I ended up using Git aliases to “create” my own git command: git patchset. The alias is defined as follows:

[alias]
        patchset = "!linux-patchset.sh"

In other words, all it does is run a script. Note that all arguments after git patchset are forwarded to the script as positional arguments.

linux-patchset.sh is more or less the following:

ROOT=$(git rev-parse --show-toplevel)
OUTDIR="${ROOT}/outgoing/$(git rev-parse --abbrev-ref HEAD)"

git format-patch -o "$OUTDIR" -s --to "" --cc "" "$@"
LINUX_ROOT="$ROOT" linux-address-patchset.py "$OUTDIR"

Basically what this script does is first git-format-patch the patchset and place the output into linux/outgoing. The output location is not that important – I just prefer it there.

Then it runs linux-address-patchset.py over the formatted patches. The logic in linux-address-patchset.py is fairly mechanical and might be subject to change over time. But the basic idea is for each patch file:

If the patch file is a cover letter, iterate through the rest of the series and collect all the mailing list addresses for the To and Cc headers
For non-cover letter patches, collect maintainer addresses for the To header
Similarly for Cc, collect reviewer and mailing list addresses
Once all the addresses are collected and deduped, append addresses to the appropriate header

The full scripts are available here.

Conclusion

All together, this how my latest patchset was generated:

$ git patchset -v2 --subject-prefix "PATCH bpf-next" bpf-next/master
~/linux/outgoing/ip_check_defrag-v2/v2-0000-cover-letter.patch
~/linux/outgoing/ip_check_defrag-v2/v2-0001-ip-frags-Return-actual-error-codes-from-ip_check_.patch
~/linux/outgoing/ip_check_defrag-v2/v2-0002-bpf-verifier-Support-KF_CHANGES_PKT-flag.patch
~/linux/outgoing/ip_check_defrag-v2/v2-0003-bpf-net-frags-Add-bpf_ip_check_defrag-kfunc.patch
~/linux/outgoing/ip_check_defrag-v2/v2-0004-net-ipv6-Factor-ipv6_frag_rcv-to-take-netns-and-u.patch
~/linux/outgoing/ip_check_defrag-v2/v2-0005-bpf-net-ipv6-Add-bpf_ipv6_frag_rcv-kfunc.patch
~/linux/outgoing/ip_check_defrag-v2/v2-0006-bpf-selftests-Support-not-connecting-client-socke.patch
~/linux/outgoing/ip_check_defrag-v2/v2-0007-bpf-selftests-Support-custom-type-and-proto-for-c.patch
~/linux/outgoing/ip_check_defrag-v2/v2-0008-bpf-selftests-Add-defrag-selftests.patch

And if you take a peek at the first two patches:

$ head -n7 outgoing/ip_check_defrag-v2/v2-0000-cover-letter.patch
From 99ddd1e2b35f6133c1f49a0245340e1a8aaaf32f Mon Sep 17 00:00:00 2001
Message-Id: <cover.1677526810.git.dxu@dxuuu.xyz>
From: Daniel Xu <dxu@dxuuu.xyz>
Date: Mon, 27 Feb 2023 12:40:10 -0700
Subject: [PATCH bpf-next v2 0/8] Support defragmenting IPv(4|6) packets in BPF
To: bpf@vger.kernel.org,linux-kselftest@vger.kernel.org,netdev@vger.kernel.org,linux-doc@vger.kernel.org,linux-kernel@vger.kernel.org
Cc: bpf@vger.kernel.org

$ head -n11 outgoing/ip_check_defrag-v2/v2-0001-ip-frags-Return-actual-error-codes-from-ip_check_.patch
From bf4afe3484836972f94c1b7738845ba69d7008f5 Mon Sep 17 00:00:00 2001
Message-Id: <bf4afe3484836972f94c1b7738845ba69d7008f5.1677526810.git.dxu@dxuuu.xyz>
In-Reply-To: <cover.1677526810.git.dxu@dxuuu.xyz>
References: <cover.1677526810.git.dxu@dxuuu.xyz>
From: Daniel Xu <dxu@dxuuu.xyz>
Date: Tue, 6 Dec 2022 17:47:16 -0700
Subject: [PATCH bpf-next v2 1/8] ip: frags: Return actual error codes from
 ip_check_defrag()
To: kuba@kernel.org,edumazet@google.com,willemdebruijn.kernel@gmail.com,davem@davemloft.net,pabeni@redhat.com,dsahern@kernel.org
Cc: netdev@vger.kernel.org,linux-kernel@vger.kernel.org,bpf@vger.kernel.org

Happy hacking!

Visualizing below using Grafana

2023-03-04T10:35:11-08:00

Visualizing below using Grafana Home

Visualizing below using Grafana

The fancy way to describe below is “a time traveling resource monitor for modern Linux systems”. The less fancy description is that below is pretty much like top or htop but can persist historical data to disk. It comes with quite a few more neat features but I’ll defer the full explanation and demo to the previous link.

Despite having a nice TUI client to replay data, it is sometimes useful to be able to plot a graph to identify patterns. This is the main weakness (open source) below has today. Since the current $DAYJOB now has a vested interest in making below work for our production use cases, I decided to tackle the visualization weakness.

Design

I considered a few design ideas:

Build custom visualization into a web page below would serve
Have below serve Grafana’s UI
Teach Grafana to talk to below using a data source plugin
Ingest below’s data into Prometheus and have Grafana talk to Prometheus

Options (1) and (2) were invalidated after a day of research. They were too much work and don’t fully leverage all the amazing work that went into Grafana. Option (3) is valid, but seemed like more of a hassle to maintain than (4). Option (4) is ultimately what I ended up going with b/c it fully leverages Grafana’s power without getting off the beaten path.

OpenMetrics

Luckily for us Prometheus has fairly good backfill support. However the only format it understands is OpenMetrics. OpenMetrics has a fairly straightforward schema (the overly rigorous spec notwithstanding). For example consider the following “exposition”:

# TYPE my_gauge gauge
# HELP my_gauge Current value of the gauge
my_gauge 123 1677953705
# TYPE my_counter counter
# HELP my_counter Monotonically increasing counter
my_counter 1677953705
# EOF

The above exposition describes two metrics: my_gauge and my_counter, where a gauge provides a point-in-time value that can go up and down and a counter is a monotonically increasing counter.

The # TYPE line describes the name and type of the metric.

The # HELP line provides a human readable description of the metric.

The sample has three parts: the name of the metric, the value, and the unix timestamp. The timestamp is optional. For backfilling we will need to always provide a timestamp.

The # EOF line terminates the exposition.

Ingestion

From here it’s clear that below, Prometheus, and Grafana all think about data in the same way. All that’s left is data conversion. Since below has good structured output support, it’s a fairly straightforwad but somewhat tedious exercise to munge below’s data into a text file.

I won’t bore you with the details so here’s the link to the code instead.

Once the OpenMetrics text file is generated, we can ingest the data into Prometheus using the following Prometheus CLI subcommand:

$ promtool tsdb create-blocks-from openmetrics ./path/to/input ./path/to/prom/data

Putting it all together

Here’s what the final user workflow looks like given some below-snapshot data at /tmp/snapshot-data:

$ git clone https://github.com/danobi/below-grafana; cd below-grafana
$ docker compose up -d
$ ./import.py /tmp/snapshot-data --begin "4h ago" --end "2h ago"

The user then visits http://localhost:3000 and can start Grafana.

Conclusion

Hopefully this is useful to people. I know I’ll be using it heavily in the future.

Full code is available here.

Not so perfect forwarding

2023-04-20T08:14:14-07:00

Not so perfect forwarding Home

Not so perfect forwarding

Consider the following C++ structs:

struct Foo {
  char b;
  int a;
} __attribute__((packed));

struct Bar {
  Bar(int a) {}
};

Now consider the following code:

#include <memory>

int main() {
  Foo f;
  auto p = std::make_unique<Bar>(f.a);
}

It should compile right? Wrong:

/tmp/cppsh-NLnGU.cpp: In function ‘int main()’:
/tmp/cppsh-NLnGU.cpp:14:36: error: cannot bind packed field ‘f.Foo::a’ to ‘int&’
   14 |   auto p = std::make_unique<Bar>(f.a);
      |

What gives? I can take it on faith that references to packed fields are banned – in general unaligned memory access is bad. But Bar::Bar takes an int by value, not by reference.

The answer lies in std::make_unique. std::make_unique uses perfect forwarding, a way to forward function arguments “without changing its lvalue or rvalue characteristics” such as const-ness and rvalue-ness. Its function signature looks like this:

template< class T, class... Args >
unique_ptr<T> make_unique( Args&&... args );

In order to forward the perfectly forward the arguments without extra copies, it tries to take a reference, leaving the final (and required) copy to Bar::Bar. Usually this is good, but in this specific case it breaks our code.

To workaround this issue, we could do:

int main() {
  Foo f;
  int a = f.a;
  auto p = std::make_unique<Bar>(a);
}

Addendum

Scott Meyers actually lays out a few more perfect forwarding failure cases. Packed members are not covered, but bitfields are probably close enough semantically.

vmtest: Run your tests in virtual machines

2023-05-16T16:38:45-07:00

vmtest: Run your tests in virtual machines Home

vmtest: Run your tests in virtual machines

Motivation

Systems software that depend on specific kernel or host properties often contains logic to gate functionality based on what features are available at runtime. This necessarily begs the question: how do you test platform specific logic? Mocks can help, but they come with their own set of tradeoffs. At the end of the day, there is no substitute for actually running your code.

eBPF (BPF) is somewhat of an extreme example because of how fast development occurs in that ecosystem. Most, if not all, BPF features are baked into the kernel, meaning that any non-trivial BPF-powered application heavily depends on kernel version. Since I do quite a bit of BPF development, the rest of this post is as a result, BPF focused. Despite that, keep in mind that all proposed techniques/solutions are quite general and can be applied to other domains.

Background

Paradoxicaly enough, despite BPF being my motivation for vmtest, BPF has excellent support for testing through BPF_PROG_RUN.

BPF programs are normally attached to a hook and only executed when when the hook is reached by the kernel. BPF_PROG_RUN, on the other hand, allows userspace to run BPF programs in “freestanding” mode. More precisely, it allows userspace to both provide the input context, run the program on demand, and collect the program’s return value (among a few other things).

Contrast this with something like iptables where it takes considerably more setup (network namespaces, fake network devices, traffic generators, etc.) to get automated tests. And even after all of that, it’s more of an integration test than a true unit test.

Basically given BPF_PROG_RUN, all that’s left to solve is how to control the kernel your tests run on top of.

Goals

The primary goal is to create infrastructure that runs code on target kernels. Along the way we may solve other problems or enable other use cases, but our primary goal will help inform our design decisions.

In addition, the following are design ideas (in decreasing order of importance) that will help inform our decisions:

Make it reusable: vmtest should be as generic as possible and adoptable across projects in different domains.
Make it work in CI: vmtest should be able to run in Github Actions. Supporting other providers would be even better.
Make it simple: defaults should “just work”. Any configurables should be unavoidable to expose.

Solution

Our solution is two-fold: vmtest and vmtest-action.

vmtest is the workhorse binary and contains all the logic to orchestrate qemu on the root host and qemu-guest-agent inside the guest VM. Among other things, it maps the running userspace into the guest VM as its rootfs. The end result is that your host’s userspace “just runs”, except the kernel it’s running under is different. The binary also supports a handful of nice-to-have things such as a standard unix interface, a terminal user interface, and support for multiple targets through a configuration file.

vmtest-action is a Github Actions wrapper (ie. a singular Github Action) that downloads/installs vmtest as well as its runtime dependencies. The action exposes a subset of the features vmtest supports but at the same time adds a few conveniences (such as URL asset downloads) – effectively tailoring vmtest to CI usage.

The two part solution helps separate concerns but also allows for novel uses cases such iterating on kernel changes using the vmtest binary.

Multi-target example

Consider the following vmtest.toml configuration:

[[target]]
name = "check if kernel is 6.2"
kernel = "./bzImage-v6.2"
command = "/bin/bash -c 'uname -r | grep -e ^6.2.0$'"

[[target]]
name = "check if kernel is 6.1"
kernel = "./bzImage-v6.2"
command = "/bin/bash -c 'uname -r; uname -r | grep -e ^6.1.0$'"

The above has two targets: the first target checks if the running kernel is v6.2 and the second checks if the running kernel is v6.1. Since they’re both going to run under a v6.2 kernel, only the first target should succeed. Carefully note here that we do not define a rootfs or run install scripts to get access to bash and coreutils – the entire running userspace on your host is shared into the VM.

This is what a run looks like:

Note that our second target fails as expected.

CI example

Now pretend you have a bog standard Github Actions (GHA) workflow. It has fairly standard steps – for example installing dependencies, build configuration, and executing a test runner. This workflow implicitly uses the kernel the GHA runner’s kernel. Perhaps it was new enough for a while but now it’s too old and you need to find a way to run your tests against a matrix of newer kernels.

vmtest-action would satisfy your needs with the following diff:

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 4b1d68f..d62fed9 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -7,6 +7,16 @@ on:
 
 jobs:
   build:
+    strategy:
+      matrix:
+        kernel:
+          - name: v6.0
+            url: https://github.com/danobi/vmtest/releases/download/test_assets/bzImage-v6.0-empty
+          - name: v6.1
+            url: https://github.com/danobi/vmtest/releases/download/test_assets/bzImage-v6.1-empty
+          - name: v6.2
+            url: https://github.com/danobi/vmtest/releases/download/test_assets/bzImage-v6.2-empty
+
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v3
@@ -21,5 +31,8 @@ jobs:
         ninja -C build
 
     - name: Run
-      run: |
-        ./build/vmtest-action-demo
+      uses: danobi/vmtest-action@master
+      with:
+        name: ${{ matrix.kernel.name }}
+        kernel_url: ${{ matrix.kernel.url }}
+        command: ${{ github.workspace}}/build/vmtest-action-demo

Note how the dependency install, configure, and build steps are being run outside the VM. This is desirable b/c anything running inside the VM is liable to be slower in the event emulation is necessary.

Implementation

The first big idea is that vmtest tries to orchestrate everything through QEMU’s programmable interfaces, namely the QEMU machine protocol (QMP) for orchestrating QEMU and qemu-guest-agent (which also uses QMP under the hood) for running things inside the guest VM. Both interfaces use a unix domain socket for transport.

Using QEMU’s programmable interfaces is probably the only novel contribution we make to this problem space. The other ideas (even the name) are shamelessly borrowed from other folk’s earlier explorations. This contribution, however, makes developing and extending vmtest a much saner and pleasant experience over things like triple-escaping shell commands. qemu-guest-agent alone gives us a clean out-of-band mechanism (so not requiring networking or SSH) to run commands inside the guest.

The second big idea is that we use 9p filesystems to share the host filesystem inside the guest. This is useful so that vmtest targets can import/export data in bulk without having to specify what to copy. In a kernel target, vmtest exports two volumes: /mnt/vmtest and the root filesystem. The first volume is the directory rooted by vmtest.toml mounted read/write. The latter export effectively gives the guest VM the same userspace environment as the host, except we mount it read-only so the guest cannot do too much damage to the host.

Conclusion

vmtest is only just getting started. By the time it goes into maintenance mode, there should be no excuse to be missing VM-based testing from your development workflows.

The case for a community maintained BPF book

2023-05-22T11:50:28-07:00

The case for a community maintained BPF book Home

The case for a community maintained BPF book

It’s widely accepted that it’s hard to get started with eBPF (BPF). It’s an extremely fast moving ecosystem and the state-of-the-art today is not the same as it was the last year. For new and perhaps more casual developers who cannot afford to follow along closely, this is a problem. How do we expect them to become productive while also avoiding potential stumbling blocks?

You have to understand C. You have to understand the kernel. You have to understand the verifier and what it tries to tell you. There is a lot to understand and no one good place to learn it all. Reading the kernel selftests is not a good answer for someone who is not involved with kernel development (consider how you’d explain making and running a change). Sure, there are some BPF books these days. But they’re out of date as soon as the ink is dry.

There is, however, precedence for solving this problem. Take a look at the Rust ecosystem of community maintained books. They’re all version controlled and contributions are made through pull requests. A large, gaping hole exists in the BPF ecosystem for such a community maintained book. Having an open book allows us all to share best practices as well as learned experience and keep it up to date over time. It’d be in a digestible format – something someone new can read cover to cover and feel up to speed.

Yeah, it’s a lot of work. But I’m pretty sure it’d be worth it.

Sudo and signal propagation

2023-05-27T10:11:23-07:00

Sudo and signal propagation Home

Sudo and signal propagation

This week I spent some time debugging a curious case where signals were not being delivered to a sudo’ed process.

Consider this shell script:

#!/bin/bash

sudo timeout 5 dd if=/dev/urandom of=/dev/null &
sudo_pid=$!
sudo kill $sudo_pid

if ps -p $sudo_pid &>/dev/null; then
  echo Still running
else
  echo Dead
fi

What should be printed? If you guessed “Dead”, then you would be correct:

$ ./sudo_kill.sh
Dead

However, when run in CI, I was seeing the opposite. The following is run from a recreated CI environment (through docker):

docker@cd71223430f4$ ./sudo_kill.sh
Still running

Signal propagation

Before going deeper, we need a little background on how sudo handles signals.

According to the man page:

When the command is run as a child of the sudo process, sudo will relay signals it receives to the command. The SIGINT and SIGQUIT signals are only relayed when the command is being run in a new pty or when the signal was sent by a user process, not the kernel. This prevents the command from receiving SIGINT twice each time the user enters control-C. Some signals, such as SIGSTOP and SIGKILL, cannot be caught and thus will not be relayed to the command.

This makes sense. Signals need to be relayed by sudo to the process it’s managing. Otherwise, scripts would need to calculate the child PID or send signals to the appropriate process group (a rather arcane concept). It is much simpler to run a backgrounded sudo instance and send signals to $! (as we did in the test script).

However, there are some drawbacks to this approach. The documentation goes on to say:

As a special case, sudo will not relay signals that were sent by the command it is running. This prevents the command from accidentally killing itself. On some systems, the reboot(8) utility sends SIGTERM to all non-system processes other than itself before rebooting the system. This prevents sudo from relaying the SIGTERM signal it received back to reboot(8), which might then exit before the system was actually rebooted, leaving it in a half-dead state similar to single user mode.

While this note is not directly related to our curious issue (b/c we are not getting dd to send signals), it is important to note there are special cases carved out.

Changes in sudo

Keeping in mind the aforementioned signal propagation corner cases, the obvious suspect is a change in sudo. From comparing the local version to the CI version (1.9.13p2 vs 1.8.21p2), we discover this commit which landed in 1.9.13p1.

The old behavior (before the change) was to:

Only forward user-generated signals not sent by a process in the command’s own process group. […]

This explains the discrepancy we saw earlier. bash runs all the commands in the script in the same process group, so kill’s signals were being ignored by sudo b/c kill is in the same process group as dd.

In my opinion this is surprising and unexpected, if only because this behavior is undocumented. That being said, I did manage to find a zero upvote stack overflow answer for a setsid workaround. But I think that just proves my point.

Fortunately, as of November 2022, the new behavior is to:

[…] forward signals from a process in the same pgrp if the pgrp leader is not either sudo or the command itself. […]

which fixes the script use case and obviates the need for a setsid workaround.

Complexity

I had remarked a few months ago I was surprised the sudo project has over 12,000 commits. Now I am no longer surprised. Behind a rather simple interface (from the common use case perspective) lies a mountain of complexity. That being said, I’m still quite surprised it took over 40 years for this surprising and undocumented behavior to be fixed.

Kernel string overcopying

2023-06-09T19:45:52-07:00

Kernel string overcopying Home

Kernel string overcopying

While on a long roadtrip the past weekend, I somehow remembered one of the fun bugs I fixed in the kernel a couple years ago.

A bug was reported to me about bpftrace: somehow there were duplicate map entries for the same string key. So something like:

# bpftrace -e '...  END { print(@) }'
^C

@[asdf]: 1
@[asdf]: 2

At first glance, this seemed like a bpftrace bug. I remember being confused for a few hours until finally narrowing down the issue to the kernel. It turns out the kernel had an optimized strncpy() routine where instead of copying a single byte at a time and stopping when it saw a zero (NUL), it was doing word-sized strides and stopping whenever the word contained a NUL. This also meant that there could be extra bytes copied after the NUL. This actually works perfectly fine for C use cases, but BPF is a slightly different environment and it turns out it does matter.

BPF maps

BPF hash maps store arbitrarily sized keys and values. In other words, they’re completely string agnostic. To simulate a string keyed map, bpftrace creates the map with key size 64 bytes (configurable with BPFTRACE_STRLEN environment variable) and just shoves bytes inside. Since the kernel map implementation is string-oblivious, any hashing or comparison doesn’t stop at a NUL byte – it keeps going for all 64 bytes.

Herein lies the issue. The BPF map implementation calls into strncpy_from_user() which can overcopy. To C string functions (eg. strncmp(), strnlen()), this is all kosher. However for a string-oblivious hash table, semantically equivalent strings can occupy different hash table entries.

The fix

The fix was rather simple conceptually but took several iterations to reach. Starting from 6fa6d28051e9 ("lib/strncpy_from_user.c: Mask out bytes after NUL terminator."), the kernel’s strncpy_from_user() masks out trailing bytes after the NUL terminator.

And since map keys should always start out zeroed, the above fix solves the issue.

bpfilter is forever

2023-07-29T19:19:19-07:00

bpfilter is forever Home

bpfilter is forever

This week it was brought to my attention that bpfilter might be delaying our kernel boot sequence. The initial thought was that bpfilter’s usermode upcalls were stalled for some reason and caused boot time stalls.

While it is true that module initialization for built-in modules (ie. CONFIG_FOO=y) is serialized and that in theory it is possible for the boot to be stalled if a module was slow, it turned out not to be the case for bpfilter, as we had CONFIG_BPFILTER_UMH=m which actually causes bpfilter.ko to be built and loaded separately.

So end of story, at least for the important side of the investigation.

Mystery reloads

While debugging the above, I unloaded and loaded bpfilter.ko a lot. After one modprobe -r, I noticed that bpfilter was reloaded without my involvement. It looked like this:

$ sudo modprobe -r bpfilter
$ lsmod | grep bpf
$ lsmod | grep bpf
$ lsmod | grep bpf
$ lsmod | grep bpf
$ lsmod | grep bpf
$ lsmod | grep bpf
$ lsmod | grep bpf
lsmod | grep bpf
bpfilter               24576  0

No good investigation is without a rabbit hole, so naturally I fixated on this.

The first thing I checked was modules-load.d and modprobe.d. A thorough grep of the relevant directories for choice keywords like “bpf” or “bpfilter” turned up nothing. That was all I had for top-down approaches so I moved to bottom-up.

Luckily we have bpftrace

I had poked around the kernel module subsystem a few weeks back for an unrelated task so I knew about the entry and exit points: init_module(2), finit_module(2), and delete_module(2).

Knowing that, it’s a simple matter of writing a one-liner to monitor those system calls:

$ sudo bpftrace -e 't:syscalls:sys_enter_*module { printf("%s: comm=%s\n", probe, comm) }' &
$ sudo modprobe -r bpfilter
tracepoint:syscalls:sys_enter_delete_module: comm=modprobe
tracepoint:syscalls:sys_enter_finit_module: comm=modprobe

So something is calling modprobe. I was halfway through writing a wrapper script to stash the parent’s comm in a file when I remembered that bpfilter will call request_module() in certain cases.

For the uninitiated, request_module() is a kernel function that requests userspace load a particular kernel module. Internally, it performs an upcall to userspace’s modprobe(8) – exactly the sort of thing we might have seen above.

Indeed, net/ipv4/bpfilter/sockopt.c confirms this:

static int bpfilter_mbox_request(struct sock *sk, int optname, sockptr_t optval,
                                 unsigned int optlen, bool is_set)
{
        int err;
        mutex_lock(&bpfilter_ops.lock);
        if (!bpfilter_ops.sockopt) {
                mutex_unlock(&bpfilter_ops.lock);
                request_module("bpfilter");
                mutex_lock(&bpfilter_ops.lock);

                if (!bpfilter_ops.sockopt) {
                        err = -ENOPROTOOPT;
                        goto out;
                }
        }

        [...]
}

bpfilter_mbox_request() is called from both the getsockopt(2) and setsockopt(2) codepaths. Since getsockopt() seemed like a more common codepath, I further traced there:

$ sudo bpftrace -e 'k:bpfilter_ip_get_sockopt { printf("comm=%s\n kstack=\n%s\n", comm, kstack) }'
Attaching 1 probe...
comm=iptables
kstack=

        bpfilter_ip_get_sockopt+1
        raw_getsockopt+52
        sock_common_getsockopt+26
        __sys_getsockopt+181
        __x64_sys_getsockopt+36
        do_syscall_64+87
        entry_SYSCALL_64_after_hwframe+6

In this case, I didn’t even have to unload the module! It turns out iptables was regularly calling getsockopt(2).

Now the question is why iptables is talking to bpfilter. bpfilter literally has no functionality, so it does not make sense for iptables to use it. To answer this, we must look further up the call stack to where getsockopt() is dispatched to bpfilter_ip_get_sockopt() – in net/ipv4/ip_sockglue.c:

int ip_getsockopt(struct sock *sk, int level,
                  int optname, char __user *optval, int __user *optlen)
{
        int err;

        err = do_ip_getsockopt(sk, level, optname,
                               USER_SOCKPTR(optval), USER_SOCKPTR(optlen));

#if IS_ENABLED(CONFIG_BPFILTER_UMH)
        if (optname >= BPFILTER_IPT_SO_GET_INFO &&
            optname < BPFILTER_IPT_GET_MAX)
                err = bpfilter_ip_get_sockopt(sk, optname, optval, optlen);
#endif
#ifdef CONFIG_NETFILTER
        /* we need to exclude all possible ENOPROTOOPTs except default case */
        if (err == -ENOPROTOOPT && optname != IP_PKTOPTIONS &&
                        !ip_mroute_opt(optname)) {
                int len;

                if (get_user(len, optlen))
                        return -EFAULT;

                err = nf_getsockopt(sk, PF_INET, optname, optval, &len);
                if (err >= 0)
                        err = put_user(len, optlen);
                return err;
        }
#endif
        return err;
}

Fortunately the code here is pretty simple: optname must be in the range [BPFFILTER_IPT_SO_GET_INFO, BPFILTER_IPT_GET_MAX) for our fateful function to be called. Another quick trace gives us there raw values:

$ sudo bpftrace -e 'k:bpfilter_ip_get_sockopt { printf("comm=%s\n optname=%d\n ustack=%s\n kstack=%s\n", comm, arg1, ustack, kstack) }'
Attaching 1 probe...
comm=iptables
 optname=64
 ustack=
        0x7fa0acd7783a

 kstack=
        bpfilter_ip_get_sockopt+1
        raw_getsockopt+52
        sock_common_getsockopt+26
        __sys_getsockopt+181
        __x64_sys_getsockopt+36
        do_syscall_64+87
        entry_SYSCALL_64_after_hwframe+68

From here, it was basic due dilligence to check the values of BPFFILTER_IPT_SO_GET_INFO and BPFILTER_IPT_GET_MAX to see if:

The range contains 64
If the range is shared with any iptables sockopts

Some quick grepping gives us the (abbreviated) definitions:

#define IPT_BASE_CTL            64
#define IPT_SO_GET_INFO         (IPT_BASE_CTL)

enum {
        BPFILTER_IPT_SO_GET_INFO = 64,
        [...]
        BPFILTER_IPT_GET_MAX,
};

So there you have it. bpfilter deliberately overlaps the iptables sockopt optname ranges. Probably so iptables(8) (the CLI) can be transparently used with the bpfilter backend. Unfortunately as a result of this design, if CONFIG_BPFILTER=y, then bpfilter_ip_get_sockopt() is unconditionally called when iptables getsockopt(2)/setsockopt(2) is used. And this triggers our module reloads. Unfortunate, understandable, and relatively harmless.

The obvious question

Astute readers will have already wondered: why do you build kernels with CONFIG_BPFILTER=y if bpfilter does not currently contain any functionality?

Well, that’s a question for Ubuntu:

$ vagrant init ubuntu/jammy64
$ vagrant up
[...]
$ vagrant ssh
vagrant@ubuntu-jammy:~$ uname -r
5.15.0-76-generic
vagrant@ubuntu-jammy:~$ grep CONFIG_BPFILTER /boot/config-$(uname -r)
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m

Revamping bpftrace’s CI with Nix

2023-09-04T22:07:59-07:00

Revamping bpftrace’s CI with Nix Home

Revamping bpftrace’s CI with Nix

I’ve been rewriting the bpftrace CI on-and-off the past few months. Now that it’s nearly done, I thought it would be nice to sum up some of the changes.

Previous state

The previous iteration of the bpftrace CI relied heavily on Github Actions and docker. Specifically, we had three major workflows: CI, embedded, and code scanning.

The CI workflow was the main workhorse – it built and tested bpftrace across various combinations of LLVM version, distro base, and compiler suite (ie GCC and clang).

The embedded workflow was fairly similar to the CI workflow except that it built bpftrace in semi-static configurations. In other words, the bpftrace binary under test would be statically linked with the exception of dynamic linking against glibc and/or llvm/clang. If the tests passed, it would then push the semi-static binary up to quay.io for end user consumption.

The code scanning job has seen various iterations over the years (from LGTM to CodeQL) but the basic premise has been the same: to scan for known code smells or bugs.

Musings

I was very happy with Github Actions (GHA). The integration with Github is obviously great and the free usage for open source projects is killer. So I had very little desire to rip that out.

The docker bits were a different story. Over time, we grew somewhat of a maze of build scripts that our infra as well as our Dockerfiles would call into. To make matters worse, almost all configuration was propagated through environment variables, making adhoc analysis quite difficult. To top it off, it was not very user friendly to run the CI configuration locally.

While I’ll grant it would’ve been possible to refactor the docker setup so that it was more user friendly, it still had the unavoidable drawback that binaries built inside the container could not be easily run on the root host. This adds friction to the development experience.

The final pain point we had was the embedded builds. The embedded builds were awesome for our users – a single binary that could run on almost any linux system. That had nice properties like being able to curl it onto a host you’re debugging without worrying if the distro had packaged the latest version of bpftrace yet. The drawback was that it was nearly impossible to maintain. Dale Hamel did tremendously valuable work years ago adding it, but unfortunately time has shown it was nearly impossible for the maintainers to keep up. Static C++ binaries are notoriously difficult to build. LLVM is notoriously difficult to integrate and keep up with. Combine both together and you have a serious problem. Since most of the maintainers are volunteers, we simply could not do it.

Enter Nix

I had not planned on Nix fixing all these problems. Originally it was an experiment for me to learn some new tech. But as I kept experimenting, I realized Nix had a lot of awesome properties. I won’t go into how I arrived at the solutions – it was an uninteresting combination of curiosity and long walks. Rather, I’ll lay out how it all fits together.

At the center of it all is the bpftrace flake. A flake is a Nix feature that enables deterministic configuration and builds of software. Inside our flake.nix, we define various bpftrace configurations. The main variable we change is the LLVM version, as it’s the most component bpftrace needs to integrate with. We also define pinned versions of bcc and libbpf as they are the other core bpftrace dependencies.

The flake immediately solves the developer experience problem. A new bpftrace developer can simply install nix and run nix build to get a fully working bpftrace binary. There are also other convenient features like developer shells that allow for more incremental builds and easy debugging. The nice thing is that all binaries built under Nix can run on the root host (not that there was ever a non-root host involved).

Next is CI. To make running CI configurations easier locally, I wrote a dependency-free python script that mirrored all the previously existing CI features. The most salient bit of the python script (and the part that took the longest to figure out) is the code that runs commands inside the Nix “environment”:

def shell(
    cmd: List[str],
    as_root: bool = False,
    cwd: Optional[Path] = None,
    env: Optional[Dict[str, str]] = None,
):
    """
    Runs the specified command in the proper nix development
    environment.

    Note that output is sent to our inherited stderr/stdout and
    that any errors immediately raise an exception.
    """
    [..]

With ci.py in place, all the workflow does is:

    steps:
    - uses: actions/checkout@v2
    - uses: DeterminateSystems/nix-installer-action@v4
    - uses: DeterminateSystems/magic-nix-cache-action@v2
    - name: Load kernel modules
      # nf_tables and xfs are necessary for testing kernel modules BTF support
      run: |
        sudo modprobe nf_tables
        sudo modprobe xfs
    - name: Build and test
      env: ${{matrix.env}}
      run: ./.github/include/ci.py

And to run a basic CI configuration locally:

$ NIX_TARGET=.#bpftrace-llvm10  ./.github/include/ci.py

Full reproduction instructions are documented here.

Appimages

The embedded builds required a bit more lateral thinking. Semi-static builds were clearly not tenable, so they were out. But we still need to give users a replacement. At the end of the day, we can’t screw our users just to make maintenance easier.

Fortunately, I had learned about AppImages recently. This is a particularly well written motivation on the topic if you’re not already familiar. The make matters even better, there already exists Nix support for appimage builds in the form of nix-appimage.

Nix and appimages are a well-made match. Basically what a nix appimage does is bundle a binary and all its runtime dependencies inside a squashfs image. It then prepends to the image a statically linked entrypoint binary that handles mounting the concatenated squashfs image through squashfuse. Once the image is mounted, control is transferred to the apprun binary which bind mounts the Nix store in a mount namespace at the places the final binary expects and then runs the final binary.

To drive the elegance home, consider the commit adding support to bpftrace. With support merged, the following is all you need to locally build a fully static bpftrace binary:

$ nix build .#appimage
$ ldd ./result
        not a dynamic executable
$ sudo ./result -e 'BEGIN { print("static!"); exit() }'
Attaching 1 probe...
static!

With this, we can delete the embedded workflow and start shipping appimages instead.

CodeQL

Last but not least is the code scanning. The previous code scanning workflow installed all the requisite build dependencies on the root host and then ran the standard cmake build. That was fine, but it was a separate maintenance path.

This commit brings Nix to the workflow. Note how we can leverage the bpftrace flake while at the same time making things simpler.

Conclusion

Like I mentioned earlier, while my Nix journey started off as an experiment, it eventually proved to be useful beyond my wildest imagination. With one new small file we now control developer experience, continuous integration, and binary artifacts. All while greatly reducing maintenance burden.

AppImage explosions

2023-09-13T06:40:45-07:00

AppImage explosions Home

AppImage explosions

In a previous post, I detailed how bpftrace’s CI was reworked to use Nix. As part of that effort, the project-distributed semi-static binary was replaced with an appimage. In this post I dive into an issue I had to debug.

The problem

The first thing I did after building the appimage was try to run it. ./result --help worked as expected. sudo bpftrace --info (a quick and dirty way to validate end to end functionality) seemed to work as well. But shortly after, while trying to open a new terminal, I was getting unable to allocate pty: No such device errors. This resulted in a bit of head scratching. I don’t remember how I figured it out, but eventually I noticed the mount table was all messed up.

To be more specific, here is what was happening:

[vagrant@fedora vagrant]$ mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=997040k,nr_inodes=249260,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=404844k,nr_inodes=819200,mode=755,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,seclabel,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime,seclabel)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
/dev/sda1 on / type ext4 (rw,relatime,seclabel)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,nosuid,noexec,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14870)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime,seclabel)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime,seclabel)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,nr_inodes=409600,inode64)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=202420k,nr_inodes=50605,mode=700,uid=1000,gid=1000,inode64)

[vagrant@fedora vagrant]$ sudo ./result --info &>/dev/null

[vagrant@fedora vagrant]$ mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=997040k,nr_inodes=249260,mode=755,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,size=404844k,nr_inodes=819200,mode=755,inode64)
/dev/sda1 on / type ext4 (rw,relatime,seclabel)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,seclabel,nr_inodes=409600,inode64)

Obviously this is not ok. bpftrace tries very hard not to mess up the host. After all, one of it its selling points is that you can safely use it on a production host.

What’s an AppImage anyways?

To the user, an appimage is a binary that looks and feels like a statically linked binary. In fact, you’d have a hard time discovering a binary is an appimage at all.

At its core, an appimage is a squashfs image that contains an application and all of its dependencies. Prepended to the binary is a statically linked runtime that:

Knows where the squashfs image starts
Mounts the squashfs image via squashfuse
Transfers control to (yet another statically linked) entrypoint binary inside the squashfs mount

In the case of nix-appimage, this entrypoint binary creates a new user and mount namespace. The user namespace is necessary to perform privileged operations like mounting. If the appimage is being run as root, the user namespace may be omitted.

Inside the new mount namespace, the entrypoint binary then:

Mounts a tmpfs at the “mountroot”
Bind mounts everything in the hosts’s root directory into $mountroot
Bind mounts the embedded nix store to $mountroot/nix/store
Chroot to the mountroot
Transfers control to the (possibly dynamically linked) application in the Nix store

In this way, the application’s dynamic linker can “just find” the shared libraries without any additional rpath manipulation.

As you can see, appimages integrate quite elegantly with Nix — it’s fairly trivial with Nix to query and embed a derivation and all its dependencies into a squashfs image.

Everyone loves printf

Back to the problem. The good news is that both the appimage runtime and entrypoint binary amount to about 1000 lines of fairly straightforward C. So it is easy to both debug and maintain if the need arises.

Since the mount logic was getting fancy, I figured it was a good place to start debugging. After a few choice placements of system("cat /proc/self/mountinfo | column -t")s, I got the following dump:

/            /                                                                                 shared:1
/            /sys                                                                              shared:2
/            /sys/kernel/security                                                              shared:3
/            /sys/fs/cgroup                                                                    shared:4
/            /sys/fs/pstore                                                                    shared:5
/            /sys/fs/bpf                                                                       shared:6
/            /sys/fs/selinux                                                                   shared:7
/            /sys/kernel/tracing                                                               shared:16
/            /sys/kernel/debug                                                                 shared:18
/            /sys/kernel/config                                                                shared:19
/            /sys/fs/fuse/connections                                                          shared:20
/            /dev                                                                              shared:8
/            /dev/shm                                                                          shared:9
/            /dev/pts                                                                          shared:10
/            /dev/hugepages                                                                    shared:14
/            /dev/mqueue                                                                       shared:15
/            /run                                                                              shared:11
/            /run/user/1000                                                                    shared:210
/            /proc                                                                             shared:12
/            /proc/sys/fs/binfmt_misc                                                          shared:13
/            /tmp                                                                              shared:17
/            /tmp/.FLCABm                                                                      shared:255
/            /tmp/.FLCABm/mountroot                                                            shared:350
/            /tmp/.FLCABm/mountroot/proc                                                       shared:12
/            /tmp/.FLCABm/mountroot/proc/sys/fs/binfmt_misc                                    shared:13
/usr/bin     /tmp/.FLCABm/mountroot/bin                                                        shared:1
/            /tmp/.FLCABm/mountroot/run                                                        shared:11
/            /tmp/.FLCABm/mountroot/run/user/1000                                              shared:210
/root        /tmp/.FLCABm/mountroot/root                                                       shared:1
/usr/lib64   /tmp/.FLCABm/mountroot/lib64                                                      shared:1
/media       /tmp/.FLCABm/mountroot/media                                                      shared:1
/home        /tmp/.FLCABm/mountroot/home                                                       shared:1
/vagrant     /tmp/.FLCABm/mountroot/vagrant                                                    shared:1
/opt         /tmp/.FLCABm/mountroot/opt                                                        shared:1
/mnt         /tmp/.FLCABm/mountroot/mnt                                                        shared:1
/lost+found  /tmp/.FLCABm/mountroot/lost+found                                                 shared:1
/srv         /tmp/.FLCABm/mountroot/srv                                                        shared:1
/            /tmp/.FLCABm/mountroot/sys                                                        shared:2
/            /tmp/.FLCABm/mountroot/sys/kernel/security                                        shared:3
/            /tmp/.FLCABm/mountroot/sys/fs/cgroup                                              shared:4
/            /tmp/.FLCABm/mountroot/sys/fs/pstore                                              shared:5
/            /tmp/.FLCABm/mountroot/sys/fs/bpf                                                 shared:6
/            /tmp/.FLCABm/mountroot/sys/fs/selinux                                             shared:7
/            /tmp/.FLCABm/mountroot/sys/kernel/tracing                                         shared:16
/            /tmp/.FLCABm/mountroot/sys/kernel/debug                                           shared:18
/            /tmp/.FLCABm/mountroot/sys/kernel/config                                          shared:19
/            /tmp/.FLCABm/mountroot/sys/fs/fuse/connections                                    shared:20
/usr         /tmp/.FLCABm/mountroot/usr                                                        shared:1
/            /tmp/.FLCABm/mountroot/tmp                                                        shared:17
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm                                    shared:255
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot                          shared:350
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/proc                     shared:12
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/proc/sys/fs/binfmt_misc  shared:13
/usr/bin     /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/bin                      shared:1
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/run                      shared:11
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/run/user/1000            shared:210
/root        /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/root                     shared:1
/usr/lib64   /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/lib64                    shared:1
/media       /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/media                    shared:1
/home        /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/home                     shared:1
/vagrant     /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/vagrant                  shared:1
/opt         /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/opt                      shared:1
/mnt         /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/mnt                      shared:1
/lost+found  /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/lost+found               shared:1
/srv         /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/srv                      shared:1
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys                      shared:2
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys/kernel/security      shared:3
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys/fs/cgroup            shared:4
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys/fs/pstore            shared:5
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys/fs/bpf               shared:6
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys/fs/selinux           shared:7
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys/kernel/tracing       shared:16
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys/kernel/debug         shared:18
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys/kernel/config        shared:19
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sys/fs/fuse/connections  shared:20
/usr         /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/usr                      shared:1
/            /tmp/.FLCABm/mountroot/dev                                                        shared:8
/            /tmp/.FLCABm/mountroot/dev/shm                                                    shared:9
/            /tmp/.FLCABm/mountroot/dev/pts                                                    shared:10
/            /tmp/.FLCABm/mountroot/dev/hugepages                                              shared:14
/            /tmp/.FLCABm/mountroot/dev/mqueue                                                 shared:15
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/dev                      shared:8
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/dev/shm                  shared:9
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/dev/pts                  shared:10
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/dev/hugepages            shared:14
/            /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/dev/mqueue               shared:15
/var         /tmp/.FLCABm/mountroot/var                                                        shared:1
/var         /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/var                      shared:1
/usr/sbin    /tmp/.FLCABm/mountroot/sbin                                                       shared:1
/usr/sbin    /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/sbin                     shared:1
/etc         /tmp/.FLCABm/mountroot/etc                                                        shared:1
/etc         /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/etc                      shared:1
/boot        /tmp/.FLCABm/mountroot/boot                                                       shared:1
/boot        /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/boot                     shared:1
/usr/lib     /tmp/.FLCABm/mountroot/lib                                                        shared:1
/usr/lib     /tmp/.FLCABm/mountroot/tmp/.mount_resultFLCABm/mountroot/lib                      shared:1

(Note that I’ve trimmed some of the columns to make the output more legible.)

There are two problems to notice here.

First is that we are sharing quite a few mounts with the root namespace in peer group 1 (see shared:1 at the end of the mount entries).

Second is the “mount explosion” problem, as described in mount_namespaces(7). By virtue of iterating through the root namespace’s /, the entrypoint binary is recursively binding the root namespace’s mounts into the new mount namespace over and over again.

Problem 1

According to mount_namespaces(7):

This mount shares events with members of a peer group. mount(2) and umount(2) events immediately under this mount will propagate to the other mounts that are members of the peer group. Propagation here means that the same mount(2) or umount(2) will automatically occur under all of the other mounts in the peer group. Conversely, mount(2) and umount(2) events that take place under peer mounts will propagate to this mount.

In other words, if a mount M is shared in a peer group, any events that occur under M are propagated to all peers. This explains the host’s mounts were getting messed up after the appimage exits. When the appimage exits, its FUSE mount is cleaned up. This leads to all the mounts under the FUSE being unmounted. And since these lower mounts are shared, the umount event is propagated to the root namespace. Note that this would only occur for appimages run as root — an unprivileged user namespace would not have sufficient permissions to umount mounts from the root namespace.

As for why the system doesn’t immediately crash, presumably mounts which have any of their files still opened are not automatically umounted.

The fix is to rebind the mountroot as MS_PRIVATE. This prevents events from under the mountroot from propagating out.

Problem 2

The second problem is the “mount explosion”. Granted, this is not a critically important problem. It is, however, kinda ugly and probably wastes resources. Fortunately the fix is simple: use MS_UNBINDABLE to prevent recursive mount explosion problems.

But what about the previous MS_PRIVATE fix? The good news is that the VFS developers have thought of that already! MS_UNBINDABLE is simply a more powerful MS_PRIVATE. Put differently: MS_UNBINDABLE is strictly additive.

Two birds with one stone, as they say.

Conclusion

Appimages are quite an elegant mechanism in my opinion. They attempt solve a difficult problem — how we ship increasingly complex software. And they do it with relative simplicity w.r.t. lines of code. While this bug was a little tricky to debug, it didn’t take much time at all b/c I could hold all the code in my head.

Truthiness in C

2023-09-19T00:58:03-07:00

Truthiness in C Home

Truthiness in C

This week someone mentioned that C supports casting from int to bool. That naturally triggered my curiosity – what does the generated code look like?

First I think it’s important to point out many casts in C are “free”. “Free” in the sense that the compiler changes its internal understanding of an expression. But in actual assembly nothing really changes – registers don’t have types. (NB: architecture dependent)

For example, consider:

char a(int x)
{
    return (char)x;
}

int b(char x)
{
    return (int)x;
}

char *c(int *x)
{
    return (char *)x;
}

The generated code looks like:

a(int):
        mov     eax, edi
        ret
b(char):
        movsx   eax, dil
        ret
c(int*):
        mov     rax, rdi
        ret

Clearly we are just moving values between the argument register and the return register. So the cast is “free” (ignoring the sign extension in b).

But what about casting from int to bool?

bool d(int x)
{
    return (bool)x;
}

Well, the compiler gives us:

d(int):
        test    edi, edi
        setne   al
        ret

We see that it generates code to first test eax and then to setne al. test is used to set the status flags. setne is used to set the lowest 8 bit subregister in rax. rax is used to return integer values from a function in the System V ABI.

Note that the top 56 bits in rax are not zeroed – they contain junk. This is fine b/c the compiler will only make callers check the lowest bit of a register for boolean operations. This is why changing the compiler’s “understanding” (ie the cast) is necessary.

So, two otherwise extra instructions. Not too bad for how useful it is.

See the full code here.

Errata

Thanks thxg on HN for pointing out that I was incorrectly compiling with -m32 but talking about x86-64.

And also to AshamedCaptain for pointing out the x86 centered assumptions.

Packet drop reasons

2023-09-19T02:17:36-07:00

Packet drop reasons Home

Packet drop reasons

Figuring out why the kernel is dropping packets can be challenging. Fortunately, newer kernel releases contain dropreason infrastructure that can make the problem more tractable.

We can easily tap into dropreason by using bpftrace. Consider this one-liner:

$ sudo bpftrace -e 'tracepoint:skb:kfree_skb { printf("%-18s %d\n", comm, args->reason); }'
Attaching 1 probe...
irq/160-iwlwifi    50
irq/160-iwlwifi    50
irq/160-iwlwifi    50
irq/170-iwlwifi    6
irq/170-iwlwifi    2
irq/160-iwlwifi    50
syncthing          6
syncthing          6
syncthing          6
irq/170-iwlwifi    6
irq/170-iwlwifi    2
irq/160-iwlwifi    50
irq/160-iwlwifi    50

^C

So we see some output. But what does args->reason correspond to? Find out by querying the tracepoint arguments:

$ sudo bpftrace -lv 'tracepoint:skb:kfree_skb'
tracepoint:skb:kfree_skb
    void * skbaddr
    void * location
    unsigned short protocol
    enum skb_drop_reason reason

Now that we know it’s an enum skb_drop_reason, query the values of the enum with:

$ sudo bpftrace -lv 'enum skb_drop_reason'
enum skb_drop_reason {
        SKB_NOT_DROPPED_YET = 0,
        SKB_CONSUMED = 1,
        SKB_DROP_REASON_NOT_SPECIFIED = 2,
        SKB_DROP_REASON_NO_SOCKET = 3,
        SKB_DROP_REASON_PKT_TOO_SMALL = 4,
        SKB_DROP_REASON_TCP_CSUM = 5,
        SKB_DROP_REASON_SOCKET_FILTER = 6,
        SKB_DROP_REASON_UDP_CSUM = 7,
        SKB_DROP_REASON_NETFILTER_DROP = 8,
        SKB_DROP_REASON_OTHERHOST = 9,
        SKB_DROP_REASON_IP_CSUM = 10,
        SKB_DROP_REASON_IP_INHDR = 11,
        SKB_DROP_REASON_IP_RPFILTER = 12,
        SKB_DROP_REASON_UNICAST_IN_L2_MULTICAST = 13,
        [...]
};

Now suppose we are curious about the SKB_DROP_REASON_SOCKET_FILTER (6) drops. Let’s see where in the kernel this is happening:

$ sudo bpftrace -e 't:skb:kfree_skb / args->reason == SKB_DROP_REASON_SOCKET_FILTER / { print(kstack) }'
Attaching 1 probe...

        kfree_skb_reason+143
        kfree_skb_reason+143
        rawv6_rcv+792
        raw6_local_deliver+269
        ip6_protocol_deliver_rcu+134
        ip6_input_finish+67
        ip6_mc_input+238
        ip6_sublist_rcv_finish+130
        ip6_sublist_rcv+552
        ipv6_list_rcv+319
        __netif_receive_skb_list_core+501
        netif_receive_skb_list_internal+465
        napi_complete_done+114
        iwl_pcie_napi_poll_msix+173
        __napi_poll+40
        net_rx_action+676
        __do_softirq+209
        do_softirq.part.0+95
        __local_bh_enable_ip+104
        iwl_pcie_irq_rx_msix_handler+205
        irq_thread_fn+32
        irq_thread+251
        kthread+229
        ret_from_fork+41

This gives us a call stack in the kernel where the drop occurs. Now the problem is within reach – we know where to start reading code.

Kernel IPIs

2023-11-24T19:53:36-08:00

Kernel IPIs Home

Kernel IPIs

An inter-processor interrupt (IPI) is a way for one processor to signal to another processor that some work needs to be done. I’m not sure why I got curious about this today. But since I dug into it, I thought I’d write it down.

Interface

Since it looks like there’s quite a few IPI vectors, I narrowed down the scope by only looking at a fairly general user: smp_call_function_single(). It helps that it has an intuitive API:

/*
 * smp_call_function_single - Run a function on a specific CPU
 * @func: The function to run. This must be fast and non-blocking.
 * @info: An arbitrary pointer to pass to the function.
 * @wait: If true, wait until function has completed on other CPUs.
 *
 * Returns 0 on success, else a negative status code.
 */
int smp_call_function_single(int cpu, smp_call_func_t func, void *info, int wait);

where smp_call_func_t is:

typedef void (*smp_call_func_t)(void *info);

In otherwords, this API lets you run code on another processor.

Implementation

Since I and many others run x86-64, I’m only looked at the x86 codepaths. And to spare you the code walkthough, here is the rough call chain instead:

smp_call_function_single(cpu, ..)
    generic_exec_single(cpu, ..)
        __smp_call_single_queue(cpu, ..)
            llist_add(node, &per_cpu(call_single_queue, cpu))              /* (1) */
            send_call_function_single_ipi(cpu)
                arch_send_call_function_single_ipi(cpu)
                    native_send_call_func_single_ipi(cpu)
                        __apic_send_IPI(cpu, CALL_FUNCTION_SINGLE_VECTOR)  /* (2) */

(1) is the crucial part. Basically what’s happening is that the calling CPU enqueues some data onto the target CPU’s percpu queue. That’s why when (2) returns void, this all still makes sense - the target CPU will be interrupted and can then pull functions off its queue to run.

But just to confirm that theory, let’s check the target CPU codepath.

CALL_FUNCTION_SINGLE_VECTOR, if you grep for it, is mapped to the following handler:

DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SINGLE_VECTOR,        sysvec_call_function_single);

sysvec_call_function_single()’s call chain, in turn, is:

sysvec_call_function_single()
    generic_smp_call_function_single_interrupt()
        __flush_smp_call_function_queue(..)
            llist_for_each_entry_safe(..)
                csd_do_func(func, info, ..)
                    func(info)

That’s basically it. I’m sure there’s quite a bit more complexity under the surface of this, but the high level view of the implementation is fairly straightforward.

Aside

I don’t know if it’s just me or not, but seeing and re-seeing basically the same technique for running code on other CPUs is kind of a mind bender. You see remotely appending to percpu queues quite often, for example with BPF cpumap’s struct ptr_ring data structure or with the core networking stack’s enqueue_to_backlog() as used by Receive Packet Steering (RPS):

        cpu = get_rps_cpu(skb->dev, skb, &rflow);
        if (cpu < 0)
                cpu = smp_processor_id();
        
        ret = enqueue_to_backlog(skb, cpu, &rflow->last_qtail);

In userspace you would typically use threads and queues and maybe some kind of signaling mechanism like a semaphore to keep a thread parked while waiting for data. I’d have thought in the kernel there was a wildly different technique. But at some level, IPIs and the regular userspace stuff are quite similar.

Pull request review: still files!

2023-12-10T16:43:08-08:00

Pull request review: still files! Home

Pull request review: still files!

It’s been about a year since the core prr features features went in. From my perspective, one surprising thing is that there’s now a small group of dedicated users who not only report bugs, but are also willing to implement new features! Speaking for myself, the only thing better to a maintainer than happy users are happy users who also send high quality PRs.

The following are some of the major features that myself or others have added. And as always, reviews are still files!

Review management

There is now rudimentary support for review management. To get a quick look at local state, run prr-status:

$ prr status
Handle                  Status     Review file
danobi/prr/18           NEW        /home/dxu/dev/review/danobi/prr/18.prr
danobi/prr/20           NEW        /home/dxu/dev/review/danobi/prr/20.prr
danobi/prr/33           NEW        /home/dxu/dev/review/danobi/prr/33.prr
danobi/prr-test-repo/3  SUBMITTED  /home/dxu/dev/review/danobi/prr-test-repo/3.prr
iovisor/bpftrace/2387   NEW        /home/dxu/dev/review/iovisor/bpftrace/2387.prr
iovisor/bpftrace/2392   REVIEWED   /home/dxu/dev/review/iovisor/bpftrace/2392.prr
iovisor/bpftrace/2397   NEW        /home/dxu/dev/review/iovisor/bpftrace/2397.prr
iovisor/bpftrace/2399   SUBMITTED  /home/dxu/dev/review/iovisor/bpftrace/2399.prr

On top of the existing prr-submit you can now also:

prr-apply to locally apply a review file to your working directory
prr-remove and to delete a review file

Advanced Vim integration (syntax colors)

To enable advanced Vim integration, simply add this (or simliar) to your plugin manager config:

Plugin 'danobi/prr', {'rtp': 'vim/'}

You can now control exactly what syntax colors are used down to a fairly low level. This is relatively important for viewing diffs as otherwise it’s visually difficult to pick out changes.

For example, I have this in my .vimrc:

"Automatically set up highlighting for `.prr` review files
augroup Prr
  autocmd!
  autocmd BufRead,BufNewFile *.prr set syntax=on

  "Make prr added/deleted highlighting more apparent
  autocmd BufRead,BufNewFile *.prr hi! link prrAdded Function
  autocmd BufRead,BufNewFile *.prr hi! link prrRemoved Keyword
  autocmd BufRead,BufNewFile *.prr hi! link prrFile Special

  "Make file delimiters more apparent
  autocmd BufRead,BufNewFile *.prr hi! link prrHeader Directory

  "Reduce all the noise from color
  autocmd BufRead,BufNewFile *.prr hi! link prrIndex Special
  autocmd BufRead,BufNewFile *.prr hi! link prrChunk Special
  autocmd BufRead,BufNewFile *.prr hi! link prrChunkH Special
  autocmd BufRead,BufNewFile *.prr hi! link prrTagName Special
  autocmd BufRead,BufNewFile *.prr hi! link prrResult Special
augroup END

and this is how my review files look:

Advanced Vim integration (diff-aware folds)

Folds are useful in general. But for diff review they are extremely powerful. If you install prr’s Vim “plugin” (as shown above), then this works out of the box.

For prr reviews, folds allow you to mark regions as “done” for organization. For example, when I’m done reviewing a file in a multi-file PR, I’ll close the file’s fold. You do this by putting your cursor inside the file preamble hitting za to toggle the fold:

So from this:

to this:

The same also works for hunks inside a file. For example, this shows the second hunk in src/parser.rs as folded:

Thanks @hauleth for adding better Vim integration!

File level comments

Github recently added the ability to comment on entire files. prr now also supports this functionality. A file level comment might look something like:

> diff --git a/ch3.txt b/ch3.txt

file comment

> index 159cf1c..20814b6 100644
> --- a/ch3.txt
> +++ b/ch3.txt
> @@ -8,22 +8,6 @@ CHAPTER 3. ATTACK BY STRATAGEM
>                                                                                                               >  4. The rule is, not to besiege walled cities if it can possibly be avoided. The preparation of mantlets, movable shelters, and various implements of war, will take up three whole months; and the piling up of mounds over against the walls will take three months more.
>

When submitted, the web UI shows:

Thanks @bcspragu for adding this!

Innovative –help messages

2023-12-24T14:02:24-08:00

Innovative –help messages Home

Innovative –help messages

Last week I heard about how Google Gemini was offering a very generous free tier on their API. So yesterday I wrote a small CLI application to interface with Gemini. It took about 30 minutes to get something reasonable working. I was satisfied - but something was missing.

Fortunately, I woke up with an idea to innovate on a crusty old staple of CLI applications: the help message. Most boring CLI applications would do something like this:

func main() {
        // Handle help message
        for _, arg := range os.Args {
                if arg == "-h" || arg == "--help" {
                        fmt.Fprintf(os.Stderr, "llm [-][context]..\n")
                        return
                }
        }

        [...]

Fast and reliable; who would want that? Bleh!

Try this on for size instead:

//go:embed main.go
var src string

func help() {
        // Boring, non-innovate help message
        if innovate := os.Getenv("INNOVATE"); innovate == "" {
                fmt.Fprintf(os.Stderr, "llm [-][context]..\n")
                return
        }

        // Now we're talking - it's time to innovate
        prompt := `
                Given the following go program that compiles to a binary titled
                'llm', print out an appropriate --help message.  Your response
                should be human readable text suitable to immediately print to
                terminal.  Do not use any code blocks or backticks.  Only show
                short form help text which at minimum explains the positional
                parameters.  Do not show anything like a man page.
        `
        parts := []genai.Part{
                genai.Text(prompt),
                genai.Text(src),
        }
        if err := ask(parts); err != nil {
                log.Fatalf("Failed to innovate: %v", err)
        }
}

[...]

Now you can innovate while you’re getting help:

$ INNOVATE=1 llm -h
Usage: llm [-][context]..

Queries gemini to provide conversation continuation.  If no parameters are
provided, input is read from stdin.  Parameters beginning with '-' are
considered prompts and passed directly to stdin.

Try out llm here!

errno and libc

2024-01-30T11:10:07-08:00

errno and libc Home

errno and libc

A few weeks ago I wanted to confirm if errno was a libc abstraction or a kernel feature. The glibc docs are deliberately vague on the topic, so experimentation seemed like the best course.

Using libc

static int use_wrapper(int cmd, union bpf_attr *attr, unsigned int size)
{
    long ret;

    /* Clear errno */
    errno = 0;

    ret = syscall(__NR_bpf, cmd, attr, size);

    if (ret < 0)
        printf("wrapped syscall failed, ret=%d, errno=%d\n", ret, errno);
    else
        printf("wrapped syscall succeeded\n");
}

Here we make a syscall to bpf(2). Mostly b/c I’m quite familiar with it.

Using assembly

static int use_raw(int cmd, union bpf_attr *attr, unsigned int size)
{
    long ret;

    /* Clear errno */
    errno = 0;

    __asm__(
        "movq %1, %%rax\n"        /* syscall number */
        "movq %2, %%rdi\n"        /* arg1 */
        "movq %3, %%rsi\n"        /* arg2 */
        "movq %4, %%rdx\n"        /* arg3 */
        "syscall\n"
        "movq %%rax, %0\n"

        /* retval */
        : "=r"(ret)

        /* input operands */
        : "r"((long)__NR_bpf), "r"((long)cmd), "r"((long)attr), "r"((long)size)

        /* clobbers */
        : "rax", "rdi", "rsi", "rdx"
       );

    /* Check return value */
    if (ret < 0)
        printf("raw syscall failed, ret=%d, errno=%d\n", ret, errno);
    else
        printf("raw syscall succeeded\n");
}

use_raw() does the exact same thing as use_wrapper(), except we use inline assembly. Even if you don’t know assembly or inline assembler, it should be quite clear the semantics from the comments.

Trying it out

#include <sys/syscall.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>
#include <linux/bpf.h>

[..]

int main()
{
    int cmd = BPF_PROG_LOAD;
    union bpf_attr *attr = NULL;
    unsigned int size = 0;

    use_raw(cmd, attr, size);
    use_wrapper(cmd, attr, size);
}

Finally, we put it together in main(). Consider this output:

$ gcc main.c

$ ./a.out
raw syscall failed, ret=-1, errno=0
wrapped syscall failed, ret=-1, errno=1

$ sudo ./a.out
raw syscall failed, ret=-7, errno=0
wrapped syscall failed, ret=-1, errno=7

Notice how errno is always 0 when skiping libc. Further notice how the return value of the raw syscall flipped to a positive value and stored into errno by libc.

So that settles it – errno is a libc abstraction.

Reading local variables with bpftrace

2024-04-07T10:59:39-07:00

Reading local variables with bpftrace Home

Reading local variables with bpftrace

Most tracers are designed around functions. You attach a probe to a function and then when the function is called and the probe is run, you access the function arguments and/or the return value. Function arguments are sufficient for the majority of use cases, as function bodies typically process and manipulate the arguments. But sometimes they are not enough. Sometimes we want to access the local variables defined in the function body.

This post walks through a hypothetical example of accessing local variables. More specifically, we will use bpftrace to trace bpftrace.

Async IDs

Whenever bpftrace needs userspace to take an action (such as printing a message to console), we must have kernelspace send a message to userspace. In order to coordinate actions, we assign unique IDs to all possible actions. For example, each print() or printf() will have its own unique ID. Userspace and kernelspace share this map of ID to action. Thus, when kernelspace sends up a message, it prefixes the blob with the ID so userspace can know how to interpret the blob.

Now that we know that async IDs are critical to bpftrace machinery, let’s find a way to trace it. For legacy reasons, in source it’s called printf_id. The editorialized source follows:

void perf_event_printer(void *cb_cookie, void *data, int size)
{
  std::vector<uint8_t> data_aligned;
  data_aligned.resize(size);
  memcpy(data_aligned.data(), data, size);

  int err;
  auto bpftrace = static_cast<BPFtrace *>(cb_cookie);
  auto arg_data = data_aligned.data();
  auto printf_id = *reinterpret_cast<uint64_t *>(arg_data);

  if (bpftrace->finalize_)
    return;
  if (bpftrace->exitsig_recv) {
    bpftrace->request_finalize();
    return;
  }

  if (printf_id == asyncactionint(AsyncAction::exit)) {
    bpftrace->request_finalize();
    return;
  } else if (printf_id == asyncactionint(AsyncAction::print)) {
    [...]
  } else if (printf_id == asyncactionint(AsyncAction::print_non_map)) {
    [...]
  } else if (printf_id == asyncactionint(AsyncAction::clear)) {
    [...]
  } [...]

Notice how it’d be tricky to derive the ID only from function arguments. If we tried to, we’d risk taking page faults or accessing unaligned memory.

Technique

The basic technique for accessing a local variable is to:

Find where in the function the variable is live
Find which register the variable is live in
Attach a probe at correct offset reading the correct register

The hand-wavy explanation for this (see register allocation for a more detailed discussion) is that variables, when used in an operation, must be in a register. And since there can be a lot more variables than registers, the compiler only keeps a subset of variables in registers at any given time and leaves the rest on the stack. As such, when (offset) a variable is “live” is just as important as where (register) is live.

Finding our register

I wouldn’t say I’m particularly good at reading assembly, so don’t worry if you’re not good either. Here I’ll show off a neat trick that helps cut through all the noise.

But first we must disassemble. There’s many ways to do this, but I’ll be using nm and objdump.

$ nm /usr/bin/bpftrace | grep perf_event_printer
00000000000ecb00 t _ZN8bpftrace18perf_event_printerEPvS0_i
0000000000020765 t _ZN8bpftrace18perf_event_printerEPvS0_i.cold

We’ll ignore the .cold function since I think that’s where the compiler keeps code unlikely to be executed (in order to reduce icache footprint).

$ objdump --disassemble=_ZN8bpftrace18perf_event_printerEPvS0_i /usr/bin/bpftrace

/usr/bin/bpftrace:     file format elf64-x86-64


Disassembly of section .init:

Disassembly of section .text:

00000000000ecb00 <_ZN8bpftrace18perf_event_printerEPvS0_i>:
   ecb00:       f3 0f 1e fa             endbr64
   ecb04:       41 57                   push   %r15
   ecb06:       41 56                   push   %r14
   ecb08:       41 55                   push   %r13
   ecb0a:       41 54                   push   %r12
   ecb0c:       55                      push   %rbp
   ecb0d:       53                      push   %rbx
   ecb0e:       48 81 ec 00 10 00 00    sub    $0x1000,%rsp
   ecb15:       48 83 0c 24 00          orq    $0x0,(%rsp)
   ecb1a:       48 81 ec e8 06 00 00    sub    $0x6e8,%rsp
   ecb21:       66 0f ef c0             pxor   %xmm0,%xmm0
   ecb25:       64 48 8b 04 25 28 00    mov    %fs:0x28,%rax
   ecb2c:       00 00
   ecb2e:       48 89 84 24 d8 16 00    mov    %rax,0x16d8(%rsp)
   ecb35:       00
   ecb36:       31 c0                   xor    %eax,%eax
   ecb38:       48 63 ea                movslq %edx,%rbp
   ecb3b:       49 89 ff                mov    %rdi,%r15
   ecb3e:       48 89 f3                mov    %rsi,%rbx
   ecb41:       48 c7 44 24 70 00 00    movq   $0x0,0x70(%rsp)
   ecb48:       00 00
   ecb4a:       49 89 ec                mov    %rbp,%r12
   ecb4d:       31 ff                   xor    %edi,%edi
   ecb4f:       0f 29 44 24 60          movaps %xmm0,0x60(%rsp)
   ecb54:       48 85 ed                test   %rbp,%rbp
   ecb57:       0f 85 5b 03 00 00       jne    eceb8 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x3b8>
   ecb5d:       48 89 ea                mov    %rbp,%rdx
   ecb60:       48 89 de                mov    %rbx,%rsi
   ecb63:       ff 15 47 8e 33 00       call   *0x338e47(%rip)        # 4259b0 <memcpy@GLIBC_2.14>
   ecb69:       4c 8b 6c 24 60          mov    0x60(%rsp),%r13
   ecb6e:       45 0f b6 77 28          movzbl 0x28(%r15),%r14d
   ecb73:       49 8b 45 00             mov    0x0(%r13),%rax
   ecb77:       45 84 f6                test   %r14b,%r14b
   ecb7a:       0f 85 20 03 00 00       jne    ecea0 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x3a0>
   ecb80:       8b 2d 22 ac 33 00       mov    0x33ac22(%rip),%ebp        # 4277a8 <_ZN8bpftrace8BPFtrace12exitsig_recvE>
   ecb86:       85 ed                   test   %ebp,%ebp
   ecb88:       0f 85 72 02 00 00       jne    ece00 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x300>
   ecb8e:       48 3d 30 75 00 00       cmp    $0x7530,%rax
   ecb94:       0f 84 46 05 00 00       je     ed0e0 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x5e0>
   ecb9a:       48 3d 31 75 00 00       cmp    $0x7531,%rax
   ecba0:       0f 84 f2 04 00 00       je     ed098 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x598>
   ecba6:       48 3d 37 75 00 00       cmp    $0x7537,%rax
   ecbac:       0f 84 1e 03 00 00       je     eced0 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x3d0>
   ecbb2:       48 3d 32 75 00 00       cmp    $0x7532,%rax
   ecbb8:       0f 84 82 05 00 00       je     ed140 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x640>
   ecbbe:       48 3d 33 75 00 00       cmp    $0x7533,%rax
   ecbc4:       0f 84 de 0a 00 00       je     ed6a8 <_ZN8bpftrace18perf_event_printerEPvS0_i+0xba8>
   ecbca:       48 3d 34 75 00 00       cmp    $0x7534,%rax
   ecbd0:       0f 84 0a 09 00 00       je     ed4e0 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x9e0>
   ecbd6:       48 3d 35 75 00 00       cmp    $0x7535,%rax
   ecbdc:       0f 84 fe 0c 00 00       je     ed8e0 <_ZN8bpftrace18perf_event_printerEPvS0_i+0xde0>
   ecbe2:       48 3d 36 75 00 00       cmp    $0x7536,%rax
   ecbe8:       0f 84 fb 0f 00 00       je     edbe9 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x10e9>
   ecbee:       48 3d 39 75 00 00       cmp    $0x7539,%rax
   ecbf4:       0f 85 3e 0e 00 00       jne    eda38 <_ZN8bpftrace18perf_event_printerEPvS0_i+0xf38>
   [...]

Experienced assembly programmers can probably read this and make sense of the structure. But if you’re like me and can only nod along to explanations, you can use this trick:

Basically, you’re going to look for a constant value that a variable is being compared to. This creates a mapping between source to assembly.

In our example, notice the following code:

  if (printf_id == asyncactionint(AsyncAction::exit)) {

Also notice AsyncAction::exit is a constant (30000):

enum class AsyncAction {
  printf  = 0,     // printf reserves 0-9999 for printf_ids
  syscall = 10000, // system reserves 10000-19999 for printf_ids
  cat     = 20000, // cat reserves 20000-29999 for printf_ids
  exit    = 30000,
  print,
  clear,
  zero,
  time,
  join,
  helper_error,
  print_non_map,
  strftime,
  watchpoint_attach,
  watchpoint_detach,
  skboutput,
};

Converting that to hex, we get:

>>> hex(30000)
'0x7530'

And when we check the assembly for that constant, we see:

   ecb8e:       48 3d 30 75 00 00       cmp    $0x7530,%rax
   ecb94:       0f 84 46 05 00 00       je     ed0e0 <_ZN8bpftrace18perf_event_printerEPvS0_i+0x5e0>

This tells us both when and where our printf_id is live. To get the offset, we do:

>>> 0xecb8e - 0xecb00
142

0xecb8e is the offset of the cmp instruction. And 0xecb00 is the offset of the function (see objdump output). The cmp instruction is acceptable b/c it is unconditionally run after the initial exit checks.

To get the register the variable is live in, look at the other operand in cmp: %rax.

Putting it all together

We have two probes: BEGIN and our accessor. We use the BEGIN probe as a trigger to actually run BPFtrace::perf_event_printer. This makes the example a nice one liner. The accessor probe uses all the information we gathered.

$ sudo bpftrace - <<EOF
    BEGIN { print("trigger!") }
    uprobe:/usr/bin/bpftrace:_ZN8bpftrace18perf_event_printerEPvS0_i+142 { print(reg("ax")); exit() }
EOF
Attaching 2 probes...
trigger!
30007

The script is telling us the value of our local variable is 30007. Consulting the source, we see 30007 maps to AsyncAction::print_non_map. This checks out – the BEGIN probe indeed uses a non-map print.

Big strings in bpftrace

2024-09-16T04:46:09-07:00

Big strings in bpftrace Home

Big strings in bpftrace

If you’ve ever written BPF code you’ve probably run into the limits of the BPF programming model. bpftrace, a high level tracing language built on BPF, does its best to hide these limits – usually through language design but sometimes through clever code generation. Even so, some fundamental issues still leak through.

One such example is with the string type.

BPF memory

The BPF programming model makes an implicit distinction between kernel memory, user memory, and BPF memory. BPF memory is memory the verifier knows is safe to access. This includes the BPF stack, map allocated memory, etc. In order to work with data originating from the first two categories, a BPF program first needs to pull it into BPF memory through bpf_probe_read_[user|kernel][_str]() helper functions. Sometimes the line is blurry. With BPF Type Format (BTF) the verifier can type check and ensure the memory being read is safely paged-in and in-bounds. For example, if the verifier can prove you have a pointer to valid kernel memory, the program can directly use the pointer in string functions without the aforementioned helpers.

To paper over such differences and expose a unified interface, bpftrace users are required to call str() on C-strings to get a handle to a string they can interact with. For example, assuming foo is a kernel function declared as void foo(char *s), the following bpftrace program counts the occurrences of every string passed to foo and stores the results into map @occurences:

# bpftrace -e ‘kfunc:foo { @occurences[str(args.s)] = count() }’

BPF memory limitations

Because there is no way to efficiently allocate dynamic BPF memory (at least, until very recently) in BPF program context, all strings need to be read onto the stack. However, the verifier imposes both a 512 byte stack limit as well as a prohibition on variable sized stack allocations. The implication being that all strings need to be pre-sized at compile time - a classic right-sizing problem.

As a result, bpftrace now has the now infamous BPFTRACE_MAX_STRLEN configuration. This knob defaulted to 64 bytes and was tunable up to around 200 bytes, at which point both LLVM and the verifier would start complaining. In the above example, any string over 63 bytes (excluding NUL) would be summarily truncated. Any experienced programmer can tell you 63 bytes (or even 200) is not nearly enough for real world strings.

Even so, this has been the status quo for some number of years… until now.

Everything is a map

Astute readers might wonder: in the above example, why does the string need to exist on the stack at all? Can’t it just be read directly into @occurences’s storage, where there are no size limitations? The answer is perhaps one day, if the kernel adds support for it. The current CRUD interface to BPF maps requires pointers for keys and values, thus implying storage somewhere in BPF memory.

This is a good thought - is there somewhere else in BPF memory without size limitations we can read the string into? The answer, as with many questions in BPF, is to use a map.

The 512 byte stack limit is a well known limitation in BPF. While there are new developments in this area, the long-standing workaround has been to use an array map sized to a single entry as scratch space. At load time, the verifier rewrites array map lookups to regular memory accesses (without the source-level helper call). This means that performance remains good despite appearances.

So what if we created an array map with a single entry of arbitrary BPFTRACE_MAX_STRLEN? Well, it wouldn’t work as a single global, as the same program can concurrently run on different CPUs. So it would have to be a per-cpu array map, which the kernel already supports. But consider the following case:

# bpftrace -e ‘kfunc:bar { @m[str(args.s)] = str(args.s2) }’

The update to map @m is a single bpf_map_update_elem() call. Meaning both strings have to be alive at the same time. In other words, we have an overlapping lifetime problem for that single array entry. This is the key issue that has blocked progress on big strings for so long.

One entry per node

Fortunately there is a rather simple and general (as you’ll see) solution lurking in plain sight. Consider the following script:

$s = str(args.s);                           // (1)
@m[$s, str(args.s2)] = str(args.s3);        // (2), (3)
$i = 0;
while ($i < 5) {
    @m2[$i] = str(args.s4);                 // (4)
    $i++;
}

This program creates 8 strings at runtime but only 4 unique storage slots are necessary to avoid all potential lifetime conflicts. To accurately count and assign slots, a compiler need only walk the abstract syntax tree (AST) and assign each str() node an increasing index or id. As long as bpftrace is careful to load strings into their assigned slot, the whole thing will work. Note it is very simple for any compiler to walk and count nodes like this.

Once all the slots are tallied up, we can create a per-cpu array map with nr_slots entries sized to any arbitrary BPFTRACE_MAX_STRLEN size. Note this size is basically unlimited now - multiple gigabytes should work as long as you have enough memory. The downside to this approach is that we are pessimistically pre-allocating memory. Strings can be smaller than BPFTRACE_MAX_STRLEN or worse: the strings may never be created. In this case, we’ve found in practice that our users are happy to trade some wasted memory for smoother debugging sessions.

Printing big strings

While assigning storage slots to each string is a novel trick, its implementation exposed a few long standing assumptions. The biggest challenge was printing big strings. For example, consider:

# bpftrace -e ‘kfunc:foo { printf(“%s, %s\n”, str(args.s), str(args.s)) }’

In this case, the program will allocate a two string “tuple” on the stack, copy both strings into the tuple, and then copy the tuple into a ring buffer for userspace to consume. Note that the stack allocation is not strictly necessary for BPF ring buffer (as it allows reserving and committing buffer space), but for simplicity we try to keep perf and BPF ring buffer support on as similar codepaths as possible.

However, printf() (and also print()) were implemented assuming that all the arguments can fit neatly on the stack. Now that strings do not necessarily fit on the stack, we must rework the printing code.

Fortunately, we can just reuse the same per-cpu buffer trick as with big strings. Except this time there are no lifetime overlap issues so we can simply give the per-cpu array one entry sized to the largest tuple the program can create. Determining the largest tuple is also easy for a compiler - walk the AST again and keep the running max.

Conclusion

Now that the big issues (as well as some smaller ones we’ve glossed over) are solved, bpftrace plans to change the default value of BPFTRACE_MAX_STRLEN up to 1024 bytes. Assuming all goes well, scripts limited by small strings will magically start working better and new users will be none the wiser about any underlying BPF limitations.

Application blackbox

2024-12-23T16:19:45-08:00

Application blackbox Home

Application blackbox

In aviation, a “black box” is the colloquial term for the flight recorder, a device that records the recent history of a flight. This device is designed to be robust to a potential crash such that the recovered data can be later used during investigation.

blackbox applies this idea to software. The principle goal of blackbox is to enable applications to save arbitrary (but structured) data during runtime and allow that data to survive any kind of crash, even SIGKILL. In addition, the blackbox can be extracted from outside the application (live or post-mortem) without introducing any blocking in the application.

API

Let’s first start with the API. The following is the abbreviated application-side API. The full documentation is available in blackbox.h.

int init(std::size_t size = DEFAULT_SIZE) noexcept;

int write(std::string_view s) noexcept;
int write(std::int64_t i) noexcept;
int write(std::string_view key, std::string_view value) noexcept;
[..]

int dump(std::ostream &out);

For example, it could be used like this:

#include <chrono>
#include <cstring>
#include <iostream>
#include <thread>

#include "blackbox.h"

int main() {
  if (auto err = blackbox::init()) {
    std::cerr << std::strerror(-err) << std::endl;
    return 1;
  }

  blackbox::write("hello world!");
  blackbox::write(123);
  blackbox::write("key1", "val1");

  blackbox::dump(std::cout);
  std::this_thread::sleep_for(std::chrono::milliseconds(500));
}

which when run, would produce this output:

$ ./demo
hello world!
123
key1=val1

Alternatively, you could do a live external extraction:

$ ./demo >/dev/null &; ./extractor $(pidof demo)
hello world!
123
key1=val1

Features

Crash resilience

In line with the idea of a flight recorder, we want to save data in the event of a crash. However, because blackbox implements this with POSIX shared memory, we want to be conservative with system resources. To that end, blackbox registers an atexit(3) handler that unlinks the shared memory. This is a clean mechanism that gives us for free the following properties:

Blackbox deletion on normal exit (return from main() or std::exit())
Blackbox deletion on normal exit from signal handler
Blackbox preservation on unhandled signal (SIGSEGV, SIGABRT, etc.)
Blackbox preservation on uncatchable signal (SIGKILL)

The rationale behind properties 1 and 2 is that the application has the chance to dump the blackbox so we can assume it will be explicitly preserved if necessary. Property 2 is why blackbox::dump() is async-signal-safe. That is, it can be safely called from signal handlers as long as operator<< on the std::ostream is also async-signal-safe.

There are various corner cases (for example _exit(3) or std::terminate()) that skip atexit handlers. In those cases, blackbox still achieves its design goal but only fails to conserve resources. For this reason, it’s recommended to have a cron job or equivalent to reap old blackboxes.

Non-blocking extraction

Above we mentioned POSIX shared memory. However, crash resilience is not the only reason for this technical choice. Shared memory was chosen to also satisfy our non-blocking design goal by exploiting shared memory’s linux implementation.

More specifically, POSIX shared memory is backed by a tmpfs mount at /dev/shm. mmap(2)ing files in tmpfs does not acquire mmap_lock in the application. Constrast this with something like a remote process_vm_readv(2) which will take mmap_lock in the application.

Why all the effort to avoid mmap_lock? mmap_lock is a particularly thorny subject in the kernel. In particular, it’s been known to cause priority inversions during out-of-memory (OOM) scenarios, which blackbox was initially designed to help troubleshoot. For this reason, we chose shared memory as it will not induce any blocking in the application.

Data consistency

So we communicate through shared memory, fine. But what about synchronization?

Traditionally, shared memory mutexes are chosen for synchronizing shared memory. However, it has somwhat peculiar corner cases on top of being blocking for the application even if we were to get clever with multi-reader locks. Therefore, another solution was needed.

For blackbox, we chose to implement a userspace sequence lock (seqlock). In short, a seqlock does not ever block the writer. Instead, the reader will retry if a modification was detected during the read. This has all the properties we need, except that excessive writes may starve readers. We don’t expect heavy writes to the blackbox, so this caveat is fine.

The same code is used during post-mortem extraction to check if the blackbox crashed during a write. In the unlikely event that occurred, we would know the data was lost. Or if we’re OK with best effort, that the extracted data is unreliable.

See further below for more details on the userspace seqlock.

Structured data

Under the hood blackbox is backed by a ring buffer to bound memory usage. However, unlike typical ring buffers, we write Tag-Length-Value (TLV) entries into the ring. This is important for two reasons:

Arbitrary structured data can be supported
It allows for perfect forward compatability

An alternative to (1) would be to force the user to serialize all data to strings. However, this prevents structured extraction. Monitoring systems (eg. prometheus) support strongly typed data. It’s wise to be sympathetic to modern monitoring systems and allow users to leverage their full analytical power.

For (2), in the event that the extractor is newer than the application’s blackbox, we are fine as long as blackbox is careful about backwards compatability. However, equally likely is if the extractor is older than the application blackbox. In that case, TLVs allow the extractor to skip unknown entries by looking at the type and length fields.

Novel techniques

Userspace sequence lock

Seqlocks are a somewhat obscure lock originating from the linux kernel. The basic principle is that prior to entering write side critical section, a sequence counter is incremented. Immediately after exiting the write side critical section, the counter is incremented again. With this, the reader knows it was interrupted during a read if the sequence counter value is not even and not the same before and after the read. Since this is all done without traditional locks, lock-free techniques must be carefully used.

blackbox offers a rather unique addition to the landscape in that we did a somewhat thorough review of the literature before attempting an implementation, going as far as discovering a very small but long-standing bug in the kernel. We also didn’t find any userspace seqlock implementations that seemed correct, so a custom implementation was called for.

Lock-free programming is a tricky topic, so we’ll spare the details. Suffice to say, it’s about controlling the observed order of memory operations. In this case, it’s ensuring that the first sequence increment is visible before (and the second sequence increment after) the critical section.

For reference, here are the writer and reader bits. They come with comments which hopefully justify the choice of memory barriers.

Writer side critical section:

template <typename F> auto write_locked(F &&f) -> decltype(f()) {
  // Ensure only one writer is writing
  lock.lock();

  // Transition into write and prevent any stores from being reordered around
  // this increment.
  //
  // memory_order_acq_rel is chosen b/c the critial section (subsequent stores)
  // must not be reordered before the sequence is incremented.
  // memory_order_release is not sufficient as it only guarantees prior stores
  // are not reordered after the increment, and not the other way around (which
  // we need).
  blackbox->sequence.fetch_add(1, std::memory_order_relaxed);
  std::atomic_thread_fence(std::memory_order_acq_rel);

  // Write new entry
  auto retval = f();

  // Transition out of write and prevent any previous stores from being
  // reordered after this increment.
  //
  // Note we do not need acquire_release semantics like above b/c release
  // semantics are strong enough to ensure stores in the critical section will
  // be visible before the sequence increment.
  //
  // Any subsequent stores are free to move into the critical section if it
  // makes the program run faster. Any subsequent critical sections will be
  // fenced by its corresponding acquire_release increment.
  blackbox->sequence.fetch_add(1, std::memory_order_release);

  // Pair with above lock()
  lock.unlock();

  return retval;
}

Reader side critical section (abbreviated):

std::uint64_t read_seq(std::atomic_uint64_t &seq) {
  // memory_order_acquire is sufficient here b/c we only need ensure
  // subsequent loads are not reordered before the sequence load.
  //
  // The `& ~0x1` is just to ensure we retry in the event there is
  // a writer in the critical section. The sequence will be odd if
  // there is a write in progress.
  return seq.load(std::memory_order_acquire) & ~0x1ULL;
}

// Returns whether or not to retry the read critical section
bool read_retry(std::atomic_uint64_t &seq, std::uint64_t old_seq) {
  // memory_order_acq_rel is necessary here to ensure prior loads
  // are not reordered after the sequence load. memory_order_acquire
  // only guarantees subsequent loads are not reordered before the
  // sequence load, and not the other way around.
  //
  // If we see that the sequence was even _and_ unchanged before and
  // after the critical section, it means we got a consistent view
  // of the blackbox.
  std::atomic_thread_fence(std::memory_order_acq_rel);
  return seq.load(std::memory_order_relaxed) != old_seq;
}

bool work(Blackbox *bb) {
  std::uint64_t seq;

  do {
    seq = read_seq(bb->sequence);
    critial_section(bb);
  } while (read_retry(bb->sequence, seq));
}

Double mapped ring buffer

One interesting consequence of writing TLVs into the ring buffer is that things get tricky when the entry header is split between the beginning and the end of the array. When this happens, a simple pointer cast to the entry is not possible. Naively, you would copy the header to a temporary buffer sized to the largest possible header:

But we can do better. blackbox makes use of a classic (but possibly esoteric) ring buffer technique: double mapping:

With a double mapped ring buffer, it is always possible to do a linear read or write. No extra memory is required - just an extra copy of page table entries that the hardware resolves to the same physical memory. This solves the header split problem on top of being more efficient.

Consider the case when the data portion of the entry needs to be split between head and tail. The writer would need to either issue two memcpy(3)s or write a loop containing a modulo on each iteration to simulate a ring-aware memcpy. Both leave performance on the table:

Two memcpy’s makes poor use of hardware prefetching and branch prediction.
A custom memcpy would be hard-pressed to make use of architecture specific optimized memcpy instructions. The modulo in each iteration is also relative expensive.

Closing thoughts

There’s currently no plans to make the code any more reusable than it currently is. If you want to use it, please reach out and we’ll see what can be done.

Flaky tests, or: why not to ignore mysteries

2025-01-17T16:57:11-08:00

Flaky tests, or: why not to ignore mysteries Home

Flaky tests, or: why not to ignore mysteries

I spent a few weeks earlier this year tracking down a set of flaky end-to-end tests where bpftrace would occasionally cease to print output. I had gotten as far as figuring out std::cout had badbit set after a write but had run out of ideas on how to debug it. At the time, because I could not reproduce it locally, I had assumed it was an oddity with pipes and CI and given up.

Except bugs never go away. They only lay dormant.

The Return of the King Bug

This week the issue re-appeared. Except in the form of new flakiness after a seemingly harmless test reliability change. I’ll spare you the details - just tedious printf bisection through many CI runs - but I essentially picked up where I left off in March and isolated the buggy code down to:

LOG(DEBUG) << "before std::cout good=" << std::cout.good();
std::cout << fmt.format_str(arg_values) << std::endl;
LOG(DEBUG) << "after std::cout good=" << std::cout.good();

Note stderr (LOG(DEBUG)) continued to work while stdout did not. The above code would show the following:

before std::cout good=1
after std::cout good=0

I’m not sure why I didn’t think to drop down to syscall level and actually read the errno last time, but fortunately I had a fresh chance to redeem myself:

auto ret = write(STDOUT_FILENO, fmted.c_str(), fmted.size());
if (ret > 0) {
    LOG(DEBUG) << "write was ok";
} else if (ret == 0) {
    LOG(DEBUG) << "write wrote nothing?";
} else {
    LOG(DEBUG) << "write failed: errno=" << errno << ": " << strerror(errno);
}

The above revealed:

write failed: errno=9: Bad file descriptor

EBADF is interesting. According to the write(2) man page:

EBADF fd is not a valid file descriptor or is not open for writing.

To not rule out either case out of hand, I opted to lsof on bpftrace startup as well as right before the write(2). I used this chunk of code:

char buf[256];
snprintf(buf, sizeof(buf), "lsof -p %d 1>&2", getpid());
system(buf);

On startup, we see:

[..]
bpftrace 7526 root   0r  FIFO   0,14       0t0   16880 pipe
bpftrace 7526 root   1w  FIFO   0,14       0t0   28011 pipe
bpftrace 7526 root   2w  FIFO   0,14       0t0   28011 pipe
[..]

but later at the write:

[..]
bpftrace 7526 root   0r     FIFO   0,14       0t0   16880 pipe
bpftrace 7526 root   2w     FIFO   0,14       0t0   28011 pipe
[..]

This can only mean something inside the process is closing FD 1.

Clear and present data

bpftrace works with raw file descriptors quite often. So it wasn’t really practical to audit every close(2) callsite. Not to mention all the dependencies. Therefore, I opted for some runtime instrumentation by scripting GDB. I originally had bpftrace to do this, but due to all the debug logs the output started getting confusing - multiple copies of everything was getting logged.

Regardless, I switched the end-to-end test invocation to this:

RUN gdb -q -ex "set confirm off" -ex "break close if \$rdi==1" -ex "commands;backtrace;kill;end" -ex run --args {{BPFTRACE}} -e 'uprobe:/proc/{{BEFORE_PID}}/root/uprobe_test:uprobeFunction1 { printf("func %s\n", func); exit() }'

What this snippet does is set a breakpoint on close(2) but only dump backtrace if the argument value is 1.

Running that in CI reveals:

Attaching 1 probe...
__BPFTRACE_NOTIFY_PROBES_ATTACHED

Thread 1 "bpftrace" hit Breakpoint 1.2, 0x00007fffe4504910 in close ()
   from /nix/store/wn7v2vhyyyi6clcyn0s9ixvl7d4d87ic-glibc-2.40-36/lib/libc.so.6
(gdb)

The lack of a stack seemed to indicate missing debug information. However, these CI jobs configure bpftrace to be built with DWARF. Therefore, I started to suspect one of bpftrace’s dependencies.

Phantom thread

The idea that it could be one of bpftrace’s dependencies sparked an interesting question: what do all these failing tests have in common? Given that the above test case was really only using the func builtin, I checked how that feature is implemented. To my surprise, it’s implemented through userspace symbolication!

That makes the common thread userspace symbolication, as the original 4 disabled tests involve symbolizing userspace stacks. So the obvious next place to look is bcc.

Upon inspection, there’s indeed a bit of suspicious looking code in ProcSyms::ModulePath::fd_ where fd_ is declared without a default value and ModulePath() has an early return that does not set fd_. In turn, ~ModulePath() does if (fd_ > 0) close(fd_) which would be buggy if fd_ is not initialized to -1.

It’s also more clear why the above reliability change started triggering flakiness. It’s b/c before the change, uprobe_test probably exited before symbolization occurred, so we were not tickling the bad codepath in bcc. After the change the process hangs around, so symbolization does occur.

The Validator

Continuing with the GDB approach, I thought if I could turn on debug information for bcc it could give me a second opinion on the source. Through our Nix based CI, this was as easy as adding to the bcc package overlay:

cmakeFlags = old.cmakeFlags ++ [ "-DCMAKE_BUILD_TYPE=Debug" ];

Unfortunately, stacks still did not work. I’m guessing it’s b/c glibc doesn’t have frame pointers enabled. And changing that would probably take too much compute to rebuild everything. So it was a matter of luck that I started thinking about dependencies - the lack of stacks is unrelated.

So I opted to check a different way: through printf debugging the dependency. Through the power of Nix, I applied a patch to bcc:

From e5609182cf0ede7c9bb32bf54670cbfba754a6c0 Mon Sep 17 00:00:00 2001
Message-ID: <e5609182cf0ede7c9bb32bf54670cbfba754a6c0.1737135820.git.dxu@dxuuu.xyz>
From: Daniel Xu <dxu@dxuuu.xyz>
Date: Fri, 17 Jan 2025 10:43:33 -0700
Subject: [PATCH] XXX: Log ~ModulePath close()

Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
---
 src/cc/syms.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/cc/syms.h b/src/cc/syms.h
index 3dffdda7..415fe28f 100644
--- a/src/cc/syms.h
+++ b/src/cc/syms.h
@@ -16,6 +16,7 @@
 #pragma once

 #include <algorithm>
+#include <iostream>
 #include <memory>
 #include <string>
 #include <sys/types.h>
@@ -144,6 +145,7 @@ class ProcSyms : SymbolCache {
       return proc_root_path_.c_str();
     }
     ~ModulePath() {
+      std::cerr << "XXX BCC: trying to ~ModulePath() close fd_=" << fd_ << std::endl;
       if (fd_ > 0)
         close(fd_);
     }
--
2.47.1

And to the flake:

patches = old.patches ++ [
    # Need to use pkgs.fetchpatch for remote patches or pkgs.writeText for local ones
    (pkgs.writeText "my-patch" (builtins.readFile ./0001-XXX-Log-ModulePath-close.patch))
];

From this, we finally get:

XXX BCC: trying to ~ModulePath() close fd_=1

So mystery solved. To validate, I applied this patch through the same mechanism:

From 02fffe05dec4eca98da053d8f6d4741a758c184d Mon Sep 17 00:00:00 2001
Message-ID: <02fffe05dec4eca98da053d8f6d4741a758c184d.1737137000.git.dxu@dxuuu.xyz>
From: Daniel Xu <dxu@dxuuu.xyz>
Date: Fri, 17 Jan 2025 11:02:55 -0700
Subject: [PATCH] XXX: FIX

Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
---
 src/cc/syms.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/cc/syms.h b/src/cc/syms.h
index 415fe28f..b9563e87 100644
--- a/src/cc/syms.h
+++ b/src/cc/syms.h
@@ -131,7 +131,7 @@ class ProcSyms : SymbolCache {
     // process by storing a file descriptor created from openat(2) if possible
     // if openat fails, falls back to process-dependent path with /proc/.../root
    private:
-    int fd_;
+    int fd_ = -1;
     std::string proc_root_path_;
     std::string path_;

--
2.47.1

A 90 minute bake (failures usually appeared after 5 minutes) shows no failures.

Conclusions

This is a really nasty bug. Arbitrarily closing file descriptors as a library dependency is very not nice. Anything can happen. For bpftrace, this is particularly risky, as bcc could’ve been closing probe file descriptors which would lead to silent data loss.

One takeaway I have is that as a tool’s developer, we are best positioned to debug crazy mysteries like this. Our users likely have very little chance of figuring this stuff out. So when we find an alarming mystery (like truncated output!), we should be doing our best to chase it to its conclusion.

Also that making puns out of movie titles got pretty hard after 2 titles.

Nix dependency tree

2025-02-09T14:10:00-08:00

Nix dependency tree Home

Nix dependency tree

This tip is so useful to me but simultaneously so hard to find that I’m going to dedicate an entire entry to it.

How to find all the dependencies a flake output pulls in:

$ nix path-info .#bpftrace-llvm18
[..]
this derivation will be built:
  /nix/store/ada4fcj69rqqqxvdzg3f0gvwm6pmzrch-bpftrace.drv

$ nix-store --query --tree /nix/store/ada4fcj69rqqqxvdzg3f0gvwm6pmzrch-bpftrace.drv
/nix/store/ada4fcj69rqqqxvdzg3f0gvwm6pmzrch-bpftrace.drv
├───/nix/store/v6x3cs394jgqfbi0a42pam708flxaphh-default-builder.sh
├───/nix/store/s63zivn27i8qv5cqiy8r5hf48r323qwa-bash-5.2p37.drv
│   ├───/nix/store/05q48dcd4lgk4vh7wyk330gr2fr082i2-bootstrap-tools.drv
│   │   ├───/nix/store/0m4y3j4pnivlhhpr5yqdvlly86p93fwc-busybox.drv
[...]

There’s a lot of output, but the full dependency tree should be there. It’ll help you answer questions like “why is XXX package being pulled in?”.

A secondary tip is that if a dependency is directly attached to your root, it means some kind of Nix magic is pulling it in. For example, a build rule could be directly referencing a package.

Parameter-efficient fine-tuning in tinygrad

2026-01-01T10:04:04-08:00

Parameter-efficient fine-tuning in tinygrad Home

Parameter-efficient fine-tuning in tinygrad

Parameter-efficient fine-tuning (PEFT) is a family of techniques used to adapt LLMs (along with other types of large models) to specific tasks or datasets. For example, if you had a private dataset from your legal practice, you could take gpt-oss off the shelf and customize it to be really good at your specific legal specialty. Unlike full fine-tuning (FFT), with PEFT you only need to update a subset of the weights in the model, leading to better computational efficiency while requiring significantly less storage.

This holiday season, I decided to implement Low Rank Adaptation (LoRA) using tinygrad. Tinygrad because it was different enough from pytorch where I would learn more but similar enough so that any learning would transfer.

Borrowing from tinygrad

For this exercise I chose Llama 3.2 1B to build on, as I had a tinygrad inference demo to base my work off of, as well as a 1B model being quicker to iterate on. Much credit goes to tinygrad for providing this example, as it would’ve been a much larger exercise to re-implement the llama architecture by hand.

While getting the tinygrad demo to work as-is, I found a nice bug where the demo failed to account for Llama3 1B and 3B having tied embeddings, for which I fixed upstream. It was a small edge case that only triggered on the smaller models, and only if using Huggingface safetensor checkpoints. I suspect GGUF, when quantizing, denormalizes the input and output embedding tensors.

Towards full fine-tuning

After that, to create a solid starting point, I inlined all the code from tinygrad examples into a single file outside of the tinygrad tree. Might just be me, but a single file ends up being a lot simpler to hack on that having a bunch of files everywhere.

With that, I started moving the code towards being trainable. Training, at this point, implies FFT, as weight updates would apply to all layers of the original base model. It ended up being fairly straightforward and not missing that many things, namely:

Ensuring batch dimension worked, by forcing batch prefill
Making forward pass return logits, so I could calculate loss
Implementing weighted cross entropy loss
Removing the bf16 -> float16 casts to stabilize training
Adding greedy sampling

With all the above, I could successfully adapt llama to speak pig latin, following the general recipe for pig-latin documented in the Tinker quickstart:

$ uv run main.py --model ~/weights/llama3.2-1b-instruct/ --size 1B
[..]
loaded weights in 2308.98 ms, 2.47 GB loaded at 1.07 GB/s
step 0: loss 9.0799
step 1: loss 5.8170
step 2: loss 3.4434
step 3: loss 2.0425
step 4: loss 1.1673
step 5: loss 0.5927
step 6: loss 0.2497

--- Inference ---
Input: hello world
Output: ellohay orldway

Note LoRA learning rates should be 10x higher than FFT (see LoRA without Regret), hence Tinker’s 1e-4 vs my 1e-5.

LoRA in a nutshell

LoRA is a popular PEFT method. In short, it “attaches” to targetted layers in the base model and replaces it with a modified version. For each targetted layer, given base weights W with shape (X, Y), LoRA replaces the layer with:

W′ = W + γBA

where γ is a constant scaling factor, A is a matrix with shape (r, Y), B is a matrix with shape (X, r), and r is a rank, tunable to match the capacity needed for a task. During weight updates, W is frozen - only B and A are updated. There are a bunch of benefits to this technique that I won’t redescribe - the reader is encouraged to consult the referenced literature.

Implementation

The implementation is fairly straightforward, especially if you’re not writing production grade code. There are two parts: a generic LoRA layer to perform the W′ update and a mechanism to attach the LoRA to relevent base weights.

For the LoRA layer:

class LoraLinearLayer:
  def __init__(self, base_layer, rank:int, alpha:float):
    self.base_layer = base_layer
    self.out_features, self.in_features = self.base_layer.weight.shape

    self.lora_A = nn.Linear(self.in_features, rank, bias=False)
    self.lora_B = nn.Linear(rank, self.out_features, bias=False)
    self.lora_B.weight = Tensor.zeros(self.out_features, rank)
    self.scaling = alpha / rank

  def __call__(self, input:Tensor, *args, **kwargs):
    result = self.base_layer(input, *args, **kwargs)
    lora_result = result + self.lora_B(self.lora_A(input)) * self.scaling
    return lora_result

and for the attachment:

class Lora:
  LORA_LAYERS = {"feed_forward", "attention", "output"}

  def __init__(self, model, rank:int, alpha:float, linear=nn.Linear):
    linear_modules = []
    for name, module in named_modules(model):
      if any(s in name for s in self.LORA_LAYERS) and isinstance(module, linear):
        linear_modules.append((name, module))

    # Freeze base weights
    self.model = model
    for param in get_parameters(self.model):
      param.requires_grad_(False)

    for name, linear in linear_modules:
      lora = LoraLinearLayer(linear, rank, alpha)
      nested_setattr(model, name, lora)

  def __call__(self, *args, **kwargs):
    return self.model(*args, **kwargs)

Applying LoRA to the full fine-tunable model is simple:

model = build_transformer(args.model, model_size=args.size, device=device, disable_kv_cache=not args.chat)
if args.lora:
  model = Lora(model, rank=4, alpha=32)
[..]
train(model, args)

Using r = 4 with a fixed α = 25 (thus γ = 25/4 = 6.25), we get:

$ uv run main.py --model ~/weights/llama3.2-1b-instruct/ --size 1B --lora
[..]
loaded weights in 2273.75 ms, 2.47 GB loaded at 1.09 GB/s
step 0: loss 9.0882
step 1: loss 6.6885
step 2: loss 3.7176
step 3: loss 2.0421
step 4: loss 1.1277
step 5: loss 0.5100
step 6: loss 0.1654

--- Inference ---
Input: hello world
Output: ellohay orld-way

The astute reader will notice the output isn’t exactly identical to the FFT output. If I had to guess, it’d be that it’s a combination of the training corpus being too limited and the model size being too small, or more likely scenario of me messing up the implementation somewhere.

Closing thoughts

This was a pretty fun exercise where I ended up going just a bit deeper than I expected, which is always nice, especially if something kind-of works by the end.

With respect to tinygrad, I thoroughly enjoyed using the framework. Despite the docs not being as full featured as I would’ve liked, the internals were extremely hackable, so I didn’t mind at all reading the code to answer my questions.