Analysis

The motivation of this implementation seems to be around not relying on the sleep utility, despite this being part of the POSIX standard. Instead, it uses a pure bash solution with the SECONDS bash builtin (manpage reference).

Looking at the source of bash reveals that this builtin is implemented using the gettimeofday call (assign_seconds, get_seconds). If we rely on gettimeofday anyway, why not rely on sleep?

Indeed, Matthew Lugg points out that this approach is not only more complex, but buggy and is vulnerable to cases where the runner is highly loaded, the system clock is adjusted and so on. A straightforward fix for the deadlock was eventually merged - using a less-than comparator instead of equality.

But we are engineers, so let’s over-engineer this whole thing by building our own sleep utility from the ground up.

Sticking with bash

To start with, let’s sidestep the use of a realtime clock (through gettimeofday calls) and instead use the monotonic clock provided by the bash builtin BASH_MONOSECONDS. Introduced in bash 5.3 (release notes), new and not universally available. Such a solution would look like:

#!/usr/bin/env bash

duration=$1
start=$BASH_MONOSECONDS
end=$((start + duration))

while [[ $BASH_MONOSECONDS -lt end ]]; do
  :
done

Like the earlier implementation, this is a busy-wait loop. Let’s do better and take advantage of some alternatives.

Zig-a-zig-ah

Zig is a handy language for doing this kind of work - fast compilation and easy to do cross-platform work, perfect for building our own portable sleep.

First, let’s avoid the busy-wait loop by going with an asynchronous approach. In Linux, we have timefd_create, kqueue in BSD and CreateWaitableTimer in Windows. Let’s start with the kqueue implementation.

Kqueue

const std = @import("std");

fn sleepKqueue(ms: u64) !void {
    const posix = std.posix;

    const kq = try posix.kqueue();
    defer posix.close(kq);

    const kev = posix.Kevent{
        .ident = 0,
        .filter = posix.system.EVFILT.TIMER,
        .flags = posix.system.EV.ADD | posix.system.EV.ONESHOT,
        .fflags = 0,
        .data = @as(isize, @intCast(ms)),
        .udata = @intFromPtr(@as(?*anyopaque, null)),
    };

    _ = try posix.kevent(kq, &[_]posix.Kevent{kev}, &[_]posix.Kevent{}, null);

    var out: [1]posix.Kevent = undefined;
    const n = try posix.kevent(kq, &[_]posix.Kevent{}, &out, null);
    if (n != 1) return error.TimerWaitFailed;
}

kqueue is the BSD equivalent of epoll in Linux and provides a scalable way to monitor multiple file descriptors. In our case, we’re using it to wait for a timer event. With a bit of CLI wrapping:

$ time ./ksleep 2

real	0m2.019s
user	0m0.003s
sys	    0m0.008s

Contrast with the earlier bash implementation:

$ time /tmp/badsleep.sh 2

real	0m1.875s
user	0m1.854s
sys	    0m0.016s

The ksleep implementation is not only more efficient in CPU usage, but also more accurate in timing - we can specify millisecond precision:

$ time ./ksleep 2.5

real	0m2.523s
user	0m0.003s
sys	    0m0.010s

Next, lets look at the Linux implementation with timerfd:

Timerfd

fn sleepTimerFd(seconds: f64) !void {
    const sys = std.os.linux;
    const flags = sys.TFD{ .CLOEXEC = true };

    const rawfd = sys.timerfd_create(sys.timerfd_clockid_t.MONOTONIC, flags);
    const fd = @as(i32, @intCast(rawfd));
    if (fd < 0) return error.TimerFdCreateFailed;
    defer std.posix.close(fd);

    const sec: i64 = @intFromFloat(@floor(seconds));
    const nsec: i64 = @intFromFloat((seconds - @as(f64, @floatFromInt(sec))) * 1_000_000_000);

    var new_value = sys.itimerspec{
        .it_interval = .{ .sec = 0, .nsec = 0 },
        .it_value = .{ .sec = sec, .nsec = nsec },
    };

    const no_flags = sys.TFD.TIMER{};

    if (sys.timerfd_settime(fd, no_flags, &new_value, null) < 0)
        return error.TimerSetFailed;

    var buf: [8]u8 = undefined;
    const n = try std.posix.read(fd, &buf);
    if (n != 8) return error.TimerReadFailed;
}

As before, this implementation is efficient and accurate:

root@13738b47c4e3:/ksleep# time ./ksleep 2.5

real	0m2.507s
user	0m0.001s
sys	    0m0.002s

timerfd_create has been available since Linux 2.6.25, so it is widely supported.

CreateWaitableTimer

I don’t have a Windows machine, so this exercise is left up to the reader.

Wrap Up

It’s not that hard to build a pretty modern sleep utility without relying on the existing sleep command. The example I’ve shown here is not exactly production ready, but it builds statically and remains small, less than 128 KB. No need for horrible bash busy-wait loops.

Another implementation is to just call nanosleep directly (h/t Jihyeon Kim):

fn clockNanosleep(seconds: f64) void {
    const sec: usize = @intFromFloat(@floor(seconds));
    const nsec: usize = @intFromFloat((seconds - @floor(seconds)) * 1_000_000_000.0);
    const posix = std.posix;

    posix.nanosleep(sec, nsec);
}

But where’s the fun in that - that’s just what sleep does!

The final code looks like this:

// MIT License

// Copyright (c) 2025 Matthew Blewitt

// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:

// The above copyright notice and this permission notice shall be included in all
// copies or substantial portions of the Software.

// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.

const std = @import("std");
const builtin = @import("builtin");

pub fn main() !void {
    var buf: [512]u8 = undefined;
    var fba = std.heap.FixedBufferAllocator.init(&buf);
    const allocator = fba.allocator();
    const args = try std.process.argsAlloc(allocator);
    defer std.process.argsFree(allocator, args);

    if (args.len != 2) {
        std.debug.print("Usage: {s} <seconds>\n", .{args[0]});
        std.process.exit(1);
    }

    const seconds = std.fmt.parseFloat(f64, args[1]) catch {
        std.debug.print("Invalid number: {s}\n", .{args[1]});
        std.process.exit(1);
    };

    if (seconds < 0) {
        std.debug.print("Duration must be >= 0\n", .{});
        std.process.exit(1);
    }

    const millis = @as(u64, @intFromFloat(seconds * 1000.0));
    switch (builtin.os.tag) {
        .linux => try sleepTimerFd(seconds),
        .macos, .ios, .freebsd, .netbsd, .openbsd, .dragonfly => try sleepKqueue(millis),
        else => {
            std.debug.print("Unsupported OS: {s}\n", .{builtin.os.tag});
            std.process.exit(1);
        },
    }
}

fn sleepKqueue(ms: u64) !void {
    const posix = std.posix;

    const kq = try posix.kqueue();
    defer posix.close(kq);

    const kev = posix.Kevent{
        .ident = 0,
        .filter = posix.system.EVFILT.TIMER,
        .flags = posix.system.EV.ADD | posix.system.EV.ONESHOT,
        .fflags = 0,
        .data = @as(isize, @intCast(ms)),
        .udata = @intFromPtr(@as(?*anyopaque, null)),
    };

    _ = try posix.kevent(kq, &[_]posix.Kevent{kev}, &[_]posix.Kevent{}, null);

    var out: [1]posix.Kevent = undefined;
    const n = try posix.kevent(kq, &[_]posix.Kevent{}, &out, null);
    if (n != 1) return error.TimerWaitFailed;
}

fn sleepTimerFd(seconds: f64) !void {
    const sys = std.os.linux;
    const flags = sys.TFD{ .CLOEXEC = true };

    const rawfd = sys.timerfd_create(sys.timerfd_clockid_t.MONOTONIC, flags);
    const fd = @as(i32, @intCast(rawfd));
    if (fd < 0) return error.TimerFdCreateFailed;
    defer std.posix.close(fd);

    const sec: i64 = @intFromFloat(@floor(seconds));
    const nsec: i64 = @intFromFloat((seconds - @as(f64, @floatFromInt(sec))) * 1_000_000_000);

    var new_value = sys.itimerspec{
        .it_interval = .{ .sec = 0, .nsec = 0 },
        .it_value = .{ .sec = sec, .nsec = nsec },
    };

    const no_flags = sys.TFD.TIMER{};

    if (sys.timerfd_settime(fd, no_flags, &new_value, null) < 0)
        return error.TimerSetFailed;

    var buf: [8]u8 = undefined;
    const n = try std.posix.read(fd, &buf);
    if (n != 8) return error.TimerReadFailed;
}

The machine was necessary, but unremarkable. It tolerated the relative abuse I’d put it through, really through inaction rather than specific events. I rarely sought out physical sensation - the mind was far more interesting. It won’t surprise you to learn I was a bookworm.

Over the last couple of years, I’d felt a dissonance start to amplify, and this came to a head with the loss of my father-in-law. A little cliché that a death in the family forces you to stare at your own mortality. The machine was fallible. As much as I enjoy a good mecha story, I really needed to be in the world than just piloting the machine and experiencing life in the abstract.

Embodied cognition teaches us that thought emerges only in the full context of the body - the collection of organs that sense the world and ground us in it. Merleau-Ponty, godfather of phenomenology, takes this even further - “Truth does not inhabit only the inner man, or more accurately, there is no inner man, man is in the world, and only in the world does he know himself.”

Screens had compressed the world. From my family computer in the dinging room, to my personal laptops and smartphones (a term that now feels very dated), my first relationships formed. I was a richer, more vibrant person on the internet - the person I felt I truly was. The compression also compressed my thoughts, being ignorant to how powerful the “shower thought” was, or the thoughts I had while in those liminal spaces between the screen and whatever necessary location I had to pilot my machine to.

After tending to my father-in-law’s affairs, I resolved to stop living in my head quite so much. Partially out of fear of the inevitable, partially forced to reckon that I was no longer a twenty-something with unlimited possibilities and all the time to explore them.

I experienced a kind of humility in rejoining my body after years away. The nostalgia of climbing over rocks and feeling sand between my toes, combined with the acute realisation that you are capable of both more and less than you used to be. I realised that my body had held on to so much–tension long ignored, fatigue overruled, postponed joy and deferred sadness. It had always been trying to communicate these things, but I just wasn’t listening. The body keeps the score indeed.

A few things changed. The first was moving with intent. I started with walking, then kettlebells, and now my regular routine of exercise 6 days a week. Running, oddly enough, is the thing that has resonated with me the most, with lifting being an on-again-off-again relationship I’ve had for a long time. I was listening to my body, for what felt like the first time.

I began to think differently—not just with different thoughts, but through a different mode of being. Less control, more noticing. Less analysis, more relation. The world became thicker, stranger, more alive. I wasn’t watching it any more. I was in it.

I am no longer just a pilot. I am the machine, and the machine is me, being in the world with all we’ve got.

In a world being rapidly changed by simulacra of minds, being truly embodied has shown how stark the differences between our minds and these simulacra are. They can put on a good show, but they aren’t concious. Can a disembodied mind even perform things like spatial reasoning, with no concept of “space”? What would happen if there was an embodied simulacrum?

What if an LLM could touch grass?

All too often a request is put into an BI team or data analyst to “run a report” or something similar. Instead, pick up some power tools and do it yourself!

Table of Contents

• SQL and DuckDB
• Statistics
  • Applying Summary Statistics
  • Confidence Intervals
  • Bayesian Reasoning
• Wrap Up

SQL and DuckDB

DuckDB is probably going to be your best entry point into the world of SQL. It’s a lightweight, in-process SQL database that you can use to run queries against CSV files, Parquet files, and more. It’s a great way to get started with SQL without having to worry about setting up a database server or anything like that. It can also connect to remote databases, like Postgres or MySQL.

If you aren’t familiar with SQL, I’d recommend going through a brief tutorial - SQLTutorial is a pretty good place to start - look at using SQLite as a database to practice with. Once you’re comfortable with the basics, you can start using DuckDB to run queries against your data.

SQL is our lingua franca for data querying. Knowing how to write SQL queries is a powerful tool to have in your toolbox, and will make you more effective. It’s not just for data analysts or BI folks - it’s for anyone who needs to make decisions based on data. I’d recommend getting familiar with JOINs, GROUP BY, and HAVING clauses, as well as window functions. If you want to push the envelope, look into CTEs (Common Table Expressions) and recursive queries.

Lets say you have an extract of data from your production database, and you want to find out how many users signed up per day over the last month. Given a CSV file users.csv with the following columns:

user_id,created_at
1,2024-11-01
2,2024-11-01
3,2024-11-02
4,2024-11-02
5,2024-11-02
6,2024-11-03

You can run the following query in DuckDB:

SELECT
  created_at,
  COUNT(user_id) AS signups
FROM "users.csv"
GROUP BY created_at
ORDER BY created_at ASC;

This will give you a table with the number of signups per day:

A trivial example, but an example all the same. Now we can get the data and do some surface level aggregations, lets start working through some statistical methods. I’m going to cover 5 statistical methods that I think are especially useful for leaders: summary statistics, distributions, confidence intervals and Bayesian reasoning.

Statistics

Sorry, R fans - Python is my go-to for analysis. Before we get into the specific methods, lets talk a little about why you should bother with this stuff at all. I’m a big believer in data-informed decisions, and to do that we need ways to interpret the data and talk about it authoritatively. Statistical methods are your other set of tools.

Here’s a quick example, levering DuckDB again, but this time for summary statistics. Let’s grab some sample data from IMDB Non-Commercial Datasets - title.ratings.tsv.gz - and SUMMARIZE it:

SELECT
    column_name as col,
    approx_unique as uniq,
    round(avg::double,2) avg,
    min,
    round(q25::double,2) q25,
    round(q50::double,2) q50,
    round(q75::double,2) q75,
    max
FROM (SUMMARIZE "title.ratings.tsv.gz");

Descriptive summary statistics like this, including all-important percentiles, give you a quick overview of the data. Here, we can see we’ve got almost 1.4M entries, but only 73 unique ratings between 1.0 and 10.0. We can also see that although the mean average is 6.95, the median is 7.15.

I generally prefer using percentiles when discussing data like this. For example, saying something like “an above average number of 5xx error codes” is not particularly illuminating, but saying “an increase of 10% in the p95 count of 5xx error codes month on month” is more precise. For more on this, see Tyler Neely’s notes on E-Prime and precise language.

Applying Summary Statistics

Lets have a look at something a little more interesting, and maybe more directly relevant to engineering leaders - issue tracker data. I’m going to be using a cleaned dataset from logpai/bughub for Firefox. First of all, let’s load it into DuckDB and get an overview on what we’re looking at:

DESCRIBE TABLE 'mozilla_firefox.csv';

Alright, Duplicated_issue is the wrong type given it should be referring to Issue_id, but nothing we can’t work with.

We don’t have some ready made values to summarize, so we’ll have to derive some ourselves. Instead, lets break down time to resolve by Priority, and then by Component, excluding duplicates and non-resolved issues. We need to do a bit of cleanup with timestamp varchar fields first, though, as they aren’t in RFC 3339 format:

import csv
from datetime import datetime

def replace_timestamps(file):
    with open(file, "r") as f:
        reader = csv.reader(f)
        header = next(reader)
        rows = list(reader)

    for row in rows:
        for i in (9, 10):
            if row[i] == "":
                continue
            timestamp = datetime.strptime(row[i], "%Y-%m-%d %H:%M:%S %z")
            row[i] = timestamp.strftime("%Y-%m-%dT%H:%M:%S%z")

    with open(file, "w") as f:
        writer = csv.writer(f)
        writer.writerow(header)
        writer.writerows(rows)


if __name__ == "__main__":
    replace_timestamps("mozilla_firefox.csv")

Then load this into DuckDB.

-- We need to install this extension first to handle timezones.
install ICU;
load ICU;

-- I prefer CTEs over subqueries
with filtered as (
    select
        "Priority",
        ("Resolved_time"::timestamptz - "Created_time"::timestamptz) as time_to_resolve
    from "mozilla_firefox.csv"
    where "Status" = 'RESOLVED'
    and "Resolution" <> 'DUPLICATE'
)

select
    "Priority",
    quantile_disc(time_to_resolve, 0.25 order by "Priority") as p25_ttr,
    quantile_disc(time_to_resolve, 0.50 order by "Priority") as p50_ttr,
    quantile_disc(time_to_resolve, 0.75 order by "Priority") as p75_ttr,
    quantile_disc(time_to_resolve, 0.95 order by "Priority") as p95_ttr
from filtered
group by "Priority"
order by "Priority" ASC;

Neat. And using a similar query, lets look at it by Component:

Distributions And Histograms

DuckDB can also help us with reviewing histograms, before moving into using something like a Python notebook to render it out as a graph. Lets say we wanted to review the distribution of all the P1 issue closures:

with filtered as (
    select
        round(epoch(("Resolved_time"::timestamptz - "Created_time"::timestamptz)) / 3600) as time_to_resolve_hours
    from "mozilla_firefox.csv"
    where "Status" = 'RESOLVED'
    and "Resolution" <> 'DUPLICATE'
    and "Priority" = 'P1'
)

select * from histogram(filtered, time_to_resolve_hours, bin_count:=15);

This gives us an inline bar-chart:

If you tilt your head to the side, you can see this data exhibits a strong Pareto distribution, where ~80% of the entries fall within the first bucket of taking up to 5000 hours / 208 days to complete. A larger bin_count would demonstrate a further breakdown.

In my mind, there are about 5 core distributions that you should recognise by sight:

• Pareto distribution and associated power law
• Gaussian distribution
• Gamma distribution
• Log-normal distribution
• Beta distribution

By being able to grasp probabilistic thinking, this allows you to make better predictions and “be right more often”.

Confidence Intervals

As an engineering leader, you are going to need to be right a lot. Part of that is coming up with high-confidence estimations. The very best seem to have an almost magical way of providing high quality estimates out of thin air. For the rest of us, there are statistical methods.

Given our historical data above, imagine a Firefox PM wanting to know how long it will take for us to solve a bug in our “Tabbed Browser” component. We’ll cover two sides of this - calculating the confidence interval, and using a Monte Carlo simulation.

Lets Do Maths

Confidence intervals have a few different ways of calculating. As we know from above, our data is not normally distributed. Fortunately, we have enough samples to take advantage of the Central Limit Theorem and the distribution of the mean.

This lets us use the following formula:

CI = Mean ± Z * (stddev / √n)

where:

• Z is the z-statistic 1.96 for 95% confidence
• stddev is the standard deviation
• n is the number of samples

The confidence interval is best given as a tuple of the upper and lower bound. Lets prepare our data with DuckDB:

COPY (select
    "Issue_id",
    round(epoch(("Resolved_time"::timestamptz - "Created_time"::timestamptz)) / 3600) as time_to_resolve_hours
from "mozilla_firefox.csv"
where "Status" = 'RESOLVED'
and "Resolution" <> 'DUPLICATE'
and "Component" = 'Tabbed Browser') to 'tabbed_browser_ttr.csv' (header, delimiter ',');

select count(*) from 'tabbed_browser_ttr.csv';

Now we can leverage Python and polars to calculate our confidence interval:

import polars as pl

df = pl.read_csv("tabbed_browser_ttr.csv")
mean_ttr = df["time_to_resolve_hours"].mean()
std_ttr = df["time_to_resolve_hours"].std()
n = df.shape[0]

confidence_interval = (
    round(mean_ttr - 1.96 * (std_ttr / (n**0.5))),
    round(mean_ttr + 1.96 * (std_ttr / (n**0.5))),
)

print(f"Number of tickets: {n}")
print(f"Mean time to resolve a ticket: {round(mean_ttr)} hours")
print(f"Confidence interval: {confidence_interval} hours")

$ python confidence.py
Number of tickets: 2737
Mean time to resolve a ticket: 11678 hours
Confidence interval: (11075, 12281) hours

So we can say to our imaginary PM that “based on our historical data, we estimate that the bug will be resolved between 461 days and 511 days from now with 95% confidence.

That’s… not great for our users! Lets dig in a bit deeper based on the Resolution field:

SELECT "Resolution", Count("Issue_id")
FROM "mozilla_firefox.csv"
WHERE "Status" = 'RESOLVED'
AND "Component" = 'Tabbed Browser'
GROUP BY 1 ORDER BY 2 DESC;

Alright, lets just focus on FIXED issues, re-running the export query appropriately:

$ python confidence.py
Number of tickets: 585
Mean time to resolve a ticket: 9537 hours
Confidence interval: (8290, 10784) hours

Welp, sorry Firefox users who had issues with tabs during the period this dataset covers.

Lets Do Simulations

As easy as this was, for some datasets it is better to run a simulation. In this example, we’re going to use the Monte Carlo simulation. We’re going to use 10K iterations, and plot a graph to show the p10 and p90, showing that 80% of simulations falls between the two values:

import polars as pl
import matplotlib.pyplot as plt

df = pl.read_csv("tabbed_browser_ttr.csv")

n_iter = 10000
n = df.shape[0]
ttr = df["time_to_resolve_hours"].drop_nulls()

simulated_means = [ttr.sample(n, with_replacement=True).mean() for _ in range(n_iter)]
simulated_means_df = pl.DataFrame(simulated_means, schema=["mean_ttr"])

p10 = simulated_means_df.quantile(0.1).get_columns()[0][0]
p90 = simulated_means_df.quantile(0.9).get_columns()[0][0]

bg = "#f9ffff"
fg = "#33321a"

plt.figure(figsize=(10, 6), facecolor=bg)
ax = plt.axes()
ax.set_facecolor(bg)
ax.hist(
    simulated_means_df["mean_ttr"],
    bins=50,
    alpha=0.5,
    fill=True,
    color=fg,
    align="mid",
    width=75,
)
ax.axvline(
    x=p10,
    color=fg,
    linestyle="dotted",
    label="10th Percentile (~8743 hours)",
)
ax.axvline(
    x=p90,
    color=fg,
    linestyle="dashed",
    label="90th Percentile (~10370 hours)",
)
ax.tick_params(axis="both", colors=fg)
[ax.spines[spine].set_color(fg) for spine in ["left", "bottom"]]
[ax.spines[spine].set_color(bg) for spine in ["right", "top"]]
plt.title("Distribution of Simulated Mean Resolution Times", fontsize=12, color=fg)
plt.xlabel("Mean Time to Resolve Issues (hours)", fontsize=10, color=fg)
plt.ylabel("Frequency", fontsize=10, color=fg)
plt.legend(labelcolor=fg, shadow=False, frameon=False)
plt.tight_layout()
plt.savefig("monte_carlo.webp")

(Honestly, matplotlib is a power tool in itself - very much recommend learning how to plot your common distributions with it.)

Here, we can say that 80% of the simulations landed between 8743 hours and 10370 hours - a slightly tighter interval than our pure maths one above, but still pretty broad.

This method I find particularly useful in capacity estimation, though this relies on a relatively stable team to be useful. As the team changes, you have to throw out old data - past performance does not indicate guarantee future results, after all.

You can also apply Monte Carlo simulations to other useful domains: incident response, budgeting and estimating defect rates during the lifetime of a project.

Bayesian Reasoning

As engineering leaders, we must strive to be “good Bayesians” - that is, we update our beliefs about the world/processes/system/people as new data becomes available.

Frankly, getting into this in a serious level of detail is beyond the scope of this post. However, we can give a pretty close-to-home example at a high level - project estimation.

Estimating Probability of Project Completion

We need to start with an initial belief. Imagine we had a dataset similar to the above with all our previously completed projects, with estimated start and end dates, and actual start and end dates. From the data, the team completes projects within the initial estimate 70% of the time, so the probability of not completing on time is 0.3

We’re part way through our project, and we’ve completed 40% of the work so far. From our historical data, if we’re on time, normally 50% ± 10% of the work is complete by this point. If we’re off track, its 30% ± 10% of the work completed.

We now need to apply Bayes’ theorem:

P(H/D) = (P(D|H) * P(H) / P(D))

P(H/D) is our posterior probability, P(D|H) is our likelihood, P(H) is our prior probability and P(D) is our evidence.

We can now calculate each component. If we assume a normal distribution of probabilities (like from the Monte Carlo simulation above), our likelihood value for “still on time” is 0.24. Conversely, for “not on time”, we get 0.13.

Our evidence is a normalising constant, which evaluates to 0.207.

From there we can plug our figures into Bayes’ theorem:

For on time, we get (0.24 * 0.7) / 0.207 = 0.8115952, or ~81% probability.
For off time, we get (0.13 * 0.3) / 0.207 = 0.1884057, or ~19% probability.

When we complete another project, we can update our model. Again, this ties into providing high quality estimates to stakeholders.

Wrap Up

We’ve covered getting access to the data that is important to you, using tools like DuckDB and Python to analyse it, and apply statistical methods to understand the data and speak precisely about what the data means for you and your decision making. However, with great power comes great responsibility - always double check your work and ensure it aligns with the business goals.

If you’d like to get even more into stats (and why wouldn’t you!) consider this post by Ivan Savov on a a revised statistics curriculum. We’ve covered, briefly, probability distributions, confidence intervals and Bayesian reasoning (briefly), but it is worth looking into linear regression. I also recommend Think Stats and Python for Stats.

Preamble

These aren’t the “7 Best Databases” or something similar to power a Buzzfeed listicle - these are just 7 databases that I think are worth your time to really look into for a week or so. You might ask something like “why not Neo4j or MongoDB or MySQL/Vitess or <insert other db here>” - the answer is mostly that I don’t find them interesting. I’m also not covering Kafka or other similar streaming data services - definitely worth your time, but not covered.

This post was inspired by the original book 7 Databases In 7 Weeks by Luc Perkins with Eric Redmond and Jim Wilson.

Table of Contents

1. PostgreSQL
2. SQLite
3. DuckDB
4. ClickHouse
5. FoundationDB
6. TigerBeetle
7. CockroachDB

• Wrap Up

1. PostgreSQL

The Default Database

“Just use Postgres” is basically a meme at this point, and for good reason. PostgreSQL is the pinnacle of boring technology, and should be the database you reach for when you need a client-server model for your database. ACID compliant, plenty of interesting tricks for replication - both physical and logical - and incredibly well supported across all the major vendors.

My favourite feature of Postgres, however, are extensions. This is where I feel Postgres really comes alive in a way that few other databases can. There are extensions for almost everything you could want - AGE enables graph data structures and the user of the Cypher query language, TimescaleDB enables time-series workloads, Hydra Columnar provides an alternate columnar storage engine, and so on. I’ve written about writing an extension relatively recently if you’d like to give it a go yourself.

Postgres shines as a great “default” database for that reason, and we’re seeing even more non-Postgres services rely on the Postgres wire protocol as a general-purpose Layer 7 protocol to provide client compatibility. With a rich ecosystem, sensible default behaviour and that it can even be fit into a Wasm install makes it a database worth understanding.

Spend a week learning about whats possible with Postgres, but also some of its limitations - MVCC can be fickle. Implement a simple CRUD app in your favourite language. Maybe even build a Postgres extension.

2. SQLite

The Local-First Database

Moving on from a client-server model, we take a detour into “embedded” databases, starting with SQLite. I’ve termed this the “local-first” database, where the SQLite database is directly co-located with the application. One of the more famous examples of this usage is WhatsApp, which stored chats as local SQLite databases on the device being used. Signal also does the same thing.

Beyond that, we’re starting to see more creative uses of SQLite rather than “just” a local ACID-compliant database. With the advent of tools like Litestream enabling streaming backups and LiteFS to provide distributed access, we can devise more interesting topologies. Extensions like CR-SQLite allow the use of CRDTs to avoid needing conflict resolution when merging changesets, as used in Corrosion.

SQLite has also had a small resurgence thanks to Ruby on Rails 8.0 - 37signals has gone all in on SQLite, building a bunch of Rails modules like Solid Queue and configuring Rails to manipulate multiple SQLite databases via database.yml for this purpose. Bluesky uses SQLite for the Personal Data Servers - every user has their own SQLite database.

Spend a week experimenting with local-first architectures using SQLite, or even seeing if you can migrate a client-server model using Postgres to something that “just” needs SQLite instead.

3. DuckDB

The Query-Anything Database

Onto the next embedded database, we have DuckDB. Much like SQLite, DuckDB is intended to be an in-process database system, but more focused on online analytical processing (OLAP) versus online transaction processing (OLTP).

Where DuckDB shines is its use as a “query-anything” database, using SQL as its dialect of choice. It can natively pull data into its engine from CSVs, TSVs, JSON etc, but also formats like Parquet - just check out the list of data sources. This gives it extreme flexibility - check out this example of querying the Bluesky firehose.

Much like Postgres, DuckDB also has extensions, though not quite as rich an ecosystem - DuckDB is much younger, after all. Many contributed by the community can be found on the list of community extensions, though a particular favourite of mine is gsheets.

Spend a week doing some data analysis and processing with DuckDB - be it via a Python notebook or something like Evidence, maybe even see how it fits in with your “local-first” approach with SQLite by offloading analytics queries of your SQLite database to DuckDB, which can read it.

4. ClickHouse

The Columnar Database

Leaving the embedded database sphere, but sticking with the analytics theme, we come to ClickHouse. If I had to only pick two databases to deal with, I’d be quite happy with just Postgres and ClickHouse - the former for OLTP, the latter for OLAP.

ClickHouse specialises in analytics workloads, and can support very high ingest rates through horizontal scaling and sharded storage. It also supports tiered storage, allowing you to split “hot” and “cold” data - GitLab have a pretty thorough doc on this.

Where ClickHouse comes into its own is when you have analytics queries to run on a dataset too big for something like DuckDB, or you need “real-time” analytics. There is a lot of “benchmarketing” around these datasets, so I’m not going to repeat them here.

Another reason I suggest checking out ClickHouse is that it is a joy to operate - deployment, scaling, backups and so on are well documented - even down to setting the right CPU governor is covered.

Spend a week exploring some larger analytics datasets, or converting some of the DuckDB analytics from above into a ClickHouse deployment. ClickHouse also has an embedded version - chDB - that can offer a more direct comparison.

5. FoundationDB

The Layered Database

We now enter the “mind expanding” section of this list, with FoundationDB. Arguably, FoundationDB is not a database, but quite literally the foundation for a database. Used in production by Apple, Snowflake and Tigris Data, FoundationDB is worth your time because it is quite unique in the world of key-value storage.

Yes, it’s an ordered key-value store, but that isn’t what is interesting about it. At first glance, it has some curious limitations - transactions cannot exceed 10MB of affected data and they cannot take longer than five seconds after the first read in a transaction. But, as they say, limits set us free. By having these limits, it can achieve full ACID transactions at very large scale - 100+ TiB clusters are known to be in operation.

FoundationDB is architected for specific workloads and extensively tested using simulation testing, which has been picked up by other technologies, including another database on this list and Antithesis, founded by some ex-FoundationDB folks. For more notes on this, check out Tyler Neely’s and Phil Eaton’s notes on the topic.

As mentioned, FoundationDB has some very specific semantics that take some getting used to - their Anti-Features and Features docs are worth familiarising yourself with to understand the problems they are looking to solve.

But why is it the “layered” database? This is because of the Layers concept. Instead of tying the storage engine to the data model, instead the storage is flexible enough to be remapped across different layers. Tigris Data have a great post about building such a layer, and there are some examples such as a Record layer and a Document layer from the FoundationDB org.

Spend a week going through the tutorials and think about how you could use FoundationDB in place of something like RocksDB. Maybe check out some of the Design Recipes and go read the paper.

6. TigerBeetle

The Obsessively Correct Database

Flowing on from the deterministic simulation testing, TigerBeetle breaks the mold from our previous databases in that it is decidedly not a general purpose database - it is entirely dedicated to financial transactions.

Why is this worth a look? Single-purpose databases are unusual, and one that is as obsessively correct as TigerBeetle are a true rarity, especially considering it is open source. They include everything from NASA’s Power of Ten Rules and Protocol-Aware Recovery, through to strict serialisability and Direct I/O to avoid issues with the kernel page cache. It is seriously impressive - just go read their Safety doc and their approach to programming they call Tiger Style.

Another interesting point about TigerBeetle is that it’s written in Zig - a relative newcomer to the systems programming language school, but clearly has fit well with what the TigerBeetle folks are trying to accomplish.

Spend a week modelling your financial accounts in a local deployment of TigerBeetle - follow the Quick Start and take a look at the System Architecture docs on how you might use it in conjunction with one of the more general-purpose databases above.

7. CockroachDB

The Global Database

Finally, we come full circle. I struggled a little on what to put here in the last slot. Thoughts originally went to Valkey, but FoundationDB scratched the key-value itch. I thought about graph databases, or something like ScyllaDB or Cassandra. I thought about DynamoDB, but not being able to run it locally/for free put me off.

In the end, I decided to close on a globally distributed database - CockroachDB. It’s Postgres wire-protocol compatible, and inherits some of the more interesting features discussed above - large horizontal scaling, strong consistency - and has some interesting features of its own.

CockroachDB enables scaling a database across multiple geographies through being based on Google’s Spanner system, which relies on atomic and GPS clocks for extremely accurate time synchronisation. Commodity hardware, however, doesn’t have such luxuries, so CockroachDB has some clever solutions where reads are retried or delayed to account for clock sync delay with NTP, and nodes also compare clock drift amongst themselves and terminate members if they exceed the maximum offset.

Another interesting feature of CockroachDB is how multi-region configurations are used, including table localities, where there are different options depending on the read/write tradeoffs you want to make.

Spend a week re-implementing the movr example in a language and framework of your choice.

Wrap Up

We’ve explored a bunch of different databases, all used in production by some of the largest companies on the planet, and hopefully this will have exposed you to some technologies you weren’t familiar with before. Take this knowledge with you as you look to solve interesting problems.

The 7 languages we’re going to look at are, notably, ones that aren’t overlapping with 7 Obscure Languages in 7 Weeks. These are going to be a bit more mainstream. I’ll go through each language explaining why you should have it in your toolbox, followed by a small demo of what I feel makes the language unique. The first 3 languages are common, the latter 4 much less so.

Table of Contents

1. Python
2. TypeScript
3. Go
4. Gleam
5. Zig
6. Racket
7. Odin

• Wrap Up

1. Python

Really, Python is about as mainstream as it gets - being ranked #1 in GitHub’s Octoverse report for 2024 and #3 in Stack Overflow’s survey for 2024. So why include this at all?

The reason is twofold - interoperability with the hottest technology of the last decade, and Python notebooks (with associated statistics packages). The latter, in my opinion, is Python’s USP - competing against R and Julia, but with significantly more ubiquity and flexibility than the former two.

To give an example of the latter - there are plenty of tutorials on machine learning models and integrating LLMs if you search around - lets leverage DuckDB and matplotlib to quickly generate a graph using notebooks in VSCode.

import duckdb
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
%config inlineBackend.figure_format = 'retina'

# inline type hints!
GRAPHRC: dict[str, tuple | str] = {
    "figure.figsize": (11.7, 8.27),
    "figure.facecolor": "#f9ffff",
    "axes.facecolor": "#f9ffff"
}
GRAPHSTYLE: str = "dark"
GRAPHFONT: str = "sans-serif"

sns.set_theme(style=GRAPHSTYLE, rc=GRAPHRC, font=GRAPHFONT)

db = duckdb.read_csv("title.ratings.tsv.gz", sep="\t")
query = """
SELECT
    tconst,
    averageRating
FROM
    ratings
WHERE
    numVotes > 50
""".strip()
df = db.query("ratings", query).pl()

ax = plt.axes()
ax.xaxis.label.set_color("#33321a")
ax.yaxis.label.set_color("#33321a")
ax.tick_params(which="both", colors="#33321a")
ax.title.set_color("#33321a")
ax.margins(0)
pt = sns.histplot(
    df,
    x="averageRating",
    bins=50,
    kde=True,
    color="#33321a",
    ax=ax
)
plt.tight_layout()
plt.title("IMDb Ratings")
plt.xlabel("Rating")
plt.ylabel("Count")
plt.show()

A handful of code and we’ve got a histogram plotted out with a kernel density estimate, with our filtering done via SQL and reading straight from the gzipped TSV from IMDb Non-Commercial Datasets. We can go even further with Simon Willison’s wonderful Datasette for exploratory data analysis and presentation. Wow your colleagues and bosses with quick and dirty notebooks. Spend a week getting familiar with these integrations and keep it in your back pocket the next time you reach for Google Sheets or Excel.

2. TypeScript.

TypeScript gives us typed JavaScript. Why should you care? Perhaps best summed up in this meme from Annie Sexton:

Typed languages are, fundamentally, safer languages to write in - being able to declare expected types and enforce their usage lets you have increased confidence that the software you write is correct.

What I find unique about TypeScript is its approach to gradual typing. If you don’t have a greenfield project, slowly introducing types lets you reduce risk piecemeal. Here’s an example:

const compact = (arr) => {
  if (orr.length > 10) {
    return arr.trim(0, 10);
  }
  return arr;
};

We’ve got a clear error above - orr is not arr and therefore would crash at runtime:

> compact([1,2,3])
Uncaught ReferenceError: orr is not defined
    at compact (REPL6:2:3)

It’s also ambiguous what arr is - multiple object instances have a length property, and trim() is not a valid Array method. So, we can gradually type this function, first in our editor:

// @ts-check
const compact = (arr) => {
  if (orr.length > 10) {
    // Cannot find name 'orr'.
    return arr.trim(0, 10);
  }
  return arr;
};

Then declare our param type via JSDoc comments:

// @ts-check

/** @param {any[]} arr */
const compact = (arr) => {
  if (arr.length > 10) {
    return arr.trim(0, 10); // Property 'trim' does not exist on type 'any[]'.
  }
  return arr;
};

Then finally convert it over with real type declarations:

const compact = (arr: string[]) => {
  if (arr.length > 10) {
    return arr.slice(0, 10);
  }
  return arr;
};

This approach to gradual typing, I feel, is pretty special. Other languages that have introduced typing, either via annotations like Python or separate systems like Ruby, do not pull it off as elegantly as this. Besides, TypeScript is one of the very few “universal” languages to write a complete application in. Consider spending a week writing a “full stack” application in TypeScript using Next.js.

3. Go

Go remains probably my personal favourite language to write, and it is worth your time too. So why Go? Its most compelling points are a “batteries-included” standard library, small surface area to learn (much like my favourite small language Lua) and easy binary shipping, including generating static binaries.

However, the “killer app” for Go is that coupled with a great type system, it is phenomenally easy to write applications that are easy to maintain. The Go team’s commitment to iterative improvement to the language without breaking backwards compatibility is extremely commendable.

Rather than providing a direct demo, I’m going to take a sample of code from my redis-rest-api project:

func newServer(ctx context.Context, cfg serverCfg, rc *redis.Client, um UserMap, auth authFunc) *http.Server {
	const maxBytes int64 = 1048576 // 1 MiB

	mux := http.NewServeMux()
	mux.HandleFunc("/", rootHandler(ctx, rc, um, auth))
	mux.HandleFunc("/pipeline", pipelineHandler(ctx, rc, um, auth))
	mux.HandleFunc("/multi-exec", txHandler(ctx, rc, um, auth))
	wrappedMux := http.MaxBytesHandler(mux, maxBytes)

	server := &http.Server{
		Addr:              cfg.ListenAddr,
		Handler:           wrappedMux,
		ReadTimeout:       cfg.ReadTimeout,
		ReadHeaderTimeout: cfg.ReadHeaderTimeout,
		IdleTimeout:       cfg.IdleTimeout,
		TLSConfig:         cfg.TLSConfig,
	}

	return server
}

Again, a handful of lines, some dependency injection, and baby you’ve got a stew server going with routing, limits and authentication. This also demonstrates another pattern I enjoy about Go - imperative programming making things easy to test. Oh right, testing! Unlike many other languages, Go comes with a test harness that is great without external libraries. An example from the above repo:

func TestAllowedCommands(t *testing.T) {
	testCases := []struct {
		expectedCommands map[string]int
		desc             string
		role             Role
	}{
		{
			desc:             "rw",
			role:             "rw",
			expectedCommands: AllowedRWCommands,
		},
		{
			desc:             "ro",
			role:             "ro",
			expectedCommands: AllowedROCommands,
		},
		{
			desc:             "missing_role",
			role:             "gg",
			expectedCommands: nil,
		},
	}
	for _, tC := range testCases {
		t.Run(tC.desc, func(t *testing.T) {
			if !reflect.DeepEqual(tC.expectedCommands, tC.role.AllowedCommands()) {
				t.Errorf("expected: %+v, got: %+v", tC.expectedCommands, tC.role.AllowedCommands())
			}
		})
	}
}

This is a table-driven test - a pattern that I don’t see used enough, but extremely common in Go codebases.

Go (hah!) spend a week implementing a small proxy, or similar networked service, or work through the fantastic Learn Go With Tests.

4. Gleam

I wanted to include something in the spirit of Erlang on this list, but wanted something a bit more “modern”. Elixir would have been the clear choice, but I decided on Gleam. Gleam is a relative newcomer, but there are some interesting ideas here - and interoperability - that make it worth exploring.

To quote the Gleam page:

The power of a type system, the expressiveness of functional programming, and the reliability of the highly concurrent, fault tolerant Erlang runtime, with a familiar and modern syntax.

Gleam applies many of the lessons learned in programming language design over the years, such as avoiding the billion dollar mistake and including a strong type system (notice a theme?)

Gleam can additionally compile to JavaScript, which is a particularly neat trick.

Here’s a small Gleam program that reads some logs in from stdin and outputs them as JSON:

import gleam/io
import gleam/iterator
import gleam/json
import gleam/string
import stdin

type LogEntry {
  Entry(level: String, message: String)
}

fn entry_to_json(entry: LogEntry) -> String {
  json.object([
    #("level", json.string(entry.level)),
    #("message", json.string(entry.message)),
  ])
  |> json.to_string
}

fn parse_to_entry(input: String) -> LogEntry {
  case string.split_once(input, on: ":") {
    Ok(parts) -> {
      Entry(level: parts.0, message: string.trim(parts.1))
    }
    Error(_) -> {
      Entry(level: "UNKNOWN", message: input)
    }
  }
}

// read logs from stdin, output as json
// input is something like "ERROR: something went wrong"
// output is something like {"level": "ERROR", "message": "something went wrong"}
pub fn main() {
  stdin.stdin()
  |> iterator.map(parse_to_entry)
  |> iterator.map(entry_to_json)
  |> iterator.each(io.print)
}

Pipelines are something every functional language should have - so much nicer than nested parens.

We can then export this program as an erlang-shipment, which compiles all the modules ready to be run by any system with Erlang:

$ gleam export erlang-shipment
  Compiling gleam_stdlib
  Compiling gleam_json
  Compiling gleeunit
  [snip]

And then execute it:

$ printf "ERROR: something went wrong\nINFO: no it didnt" | ./build/erlang-shipment/entrypoint.sh run | jq
{
  "level": "ERROR",
  "message": "something went wrong"
}
{
  "level": "INFO",
  "message": "no it didnt"
}

Gleam is a young language, being developed by a small community under the careful stewardship of Louis Pilfold. Spend a week helping out on some issues, or consider writing some packages to add to the ecosystem (at least, something more meaningful than left-pad, please).

5. Zig

I wanna really, really, really wanna “zig-a-zig”, ah
~ Spice Girls

Zig is next up. I was sorely tempted to go with a “You were expecting Rust, but it was me, Zig!” opening, but we’ll have to do with a meta-commentary instead.

To me, Zig is more interesting than Rust for three primary reasons - comptime, passing allocators, and incremental improvement of C/C++ codebases through Zig. If Rust is a “better C++”, then Zig is a “better C”, where the lack of macros etc are a feature, rather than an oversight.

Much like my earlier recommendation of Go, the surface area of Zig is very small, with no hidden control flow and a frankly tiny PEG grammar. It is very much worth your time to read the features overview - it covers pretty much all the unique aspects of Zig.

My examples here will be with Zig 0.13.0 - Zig is a reasonably unstable language (folks who write nightly Rust will be familiar) and so these examples may not run with newer or older versions.

First up, a fun comptime trick - initialising a set of static data that would be expensive at runtime (from the comptime documentation):

const first_25_primes = firstNPrimes(25);
const sum_of_first_25_primes = sum(&first_25_primes);

fn firstNPrimes(comptime n: usize) [n]i32 {
    var prime_list: [n]i32 = undefined;
    var next_index: usize = 0;
    var test_number: i32 = 2;
    while (next_index < prime_list.len) : (test_number += 1) {
        var test_prime_index: usize = 0;
        var is_prime = true;
        while (test_prime_index < next_index) : (test_prime_index += 1) {
            if (test_number % prime_list[test_prime_index] == 0) {
                is_prime = false;
                break;
            }
        }
        if (is_prime) {
            prime_list[next_index] = test_number;
            next_index += 1;
        }
    }
    return prime_list;
}

fn sum(numbers: []const i32) i32 {
    var result: i32 = 0;
    for (numbers) |x| {
        result += x;
    }
    return result;
}

test "variable values" {
    try @import("std").testing.expect(sum_of_first_25_primes == 1060);
}

We can run this with zig test --verbose-llvm-ir=llvm.ir primes.zig, and then inspect the LLVM IR for the compiled values:

$ rg '@primes\.' llvm.ir
255:@primes.first_25_primes = internal unnamed_addr constant [25 x i32] [i32 2, i32 3, i32 5, i32 7, i32 11, i32 13, i32 17, i32 19, i32 23, i32 29, i32 31, i32 37, i32 41, i32 43, i32 47, i32 53, i32 59, i32 61, i32 67, i32 71, i32 73, i32 79, i32 83, i32 89, i32 97], align 4, !dbg !10
256:@primes.sum_of_first_25_primes = internal unnamed_addr constant i32 1060, align 4, !dbg !11

This is a useful trick if you have something like precomputed hash tables, where you can amortise the cost over the life of the program by doing this at compile time. Another example is a generic queue.

Spend a week re-implementing some basic C programs in Zig, and see how far you get. Something like coreutils/basename is a small enough domain to work through.

6. Racket

Racket is next up - you didn’t think I’d complete this list without a Lisp, right? Racket is interesting in that unlike pretty much all other languages, Racket is language-oriented - its a language for making languages. Additionally, its strengths lie in its ability to convey programming principles, exemplified in the book How To Design Programs.

Given it’s a language to build languages, its no surprise that it has some in-built languages to ease development in specific domains. For example, for a web server, you might consider the web-server/insta language:

#lang web-server/insta
(define (start request)
  (response/xexpr
   '(html
     (head (title "My Blog"))
     (body (h1 "Under construction")))))

We can compile and run this (with compilation optional):

$ raco make web.rkt
$ racket compiled/web_rkt.zo
Your Web application is running at http://localhost:56355/servlets/standalone.rkt.
Stop this program at any time to terminate the Web Server.

Or what about making a language for web publishing or generating a Racket AST? The concept of language-oriented programming is fascinating to me, especially given that so much of business domains are their own language. We see new DSLs pop up all the time, such as Cedar to express access control.

Spend a week writing a language in Racket to solve a trivial problem - I like the example of stacker for a stack-based calculator. Why not implement one for Reverse Polish Notation?

7. Odin

Odin is a fascinating language that has a bunch in common with Zig, but with some very specific applications. As mentioned in the splash page, JangaFX are the creators and extensively use Odin in their applications, which are computer graphics related. As such, Odin holds a pretty unique niche in having bindings for literally all the graphics runtimes out there, from Vulkan to WebGPU.

Of course, not just games are made with Odin. A great example is the web version of Spall - a flamegraph profiler that supports their own native format as well as Google’s tracing format.

Rather than give you some demo code, I’ll instead link to Karl Zylinski’s wonderful series where they walk through making a game with Odin and raylib. Making small games is a joy. Spend a week working through that.

If you must have some code, here you go:

package game

import rl "vendor:raylib"

main :: proc() {
	rl.InitWindow(1280, 720, "A window appears!")

	for !rl.WindowShouldClose() {
		rl.BeginDrawing()
		rl.DrawText("Look ma, no engine!", 190, 200, 20, rl.BLACK)
		rl.ClearBackground(rl.RAYWHITE)
		rl.EndDrawing()
	}

	rl.CloseWindow()
}

In this example, we load the bindings to raylib and create a window, draw some text and set the background colour.

Wrap Up

We’ve covered 7 languages - Python, TypeScript, Go, Gleam, Zig, Racket and Odin - as something to explore in 7 weeks. Maybe one of these will become your daily driver, or you’ll make significant contributions to the language or community. Maybe you’ll hate them and never touch them again. At the very least, I hope you learn something from exploring them.

For a long time, I felt the gold standard for a career ladder was RentTheRunway’s, shared by Camille Fournier. Combined with something like levels.fyi, we can get a scope of what the “median average” software engineer looks like at each level.

I’m going to take a bit of an unconventional path and talk very little about technology. In an industrial programming environment, we should be hiring folks who can program.

However you check for that in your interview loop is your choice - I’m a fan of the work sample test. This, combined with a strong rubric, helps with levelling.

For the purposes of this article, I’m going to use Levels.fyi’s “standard” mapping for levelling to provide an easy comparison. I’m also going to be primarily talking about scope and impact.

I’ve worked at companies from ~20 employees through to ~80K employees, and at the time of writing I manage a team that covers everything from L1 through the top of L4.

We’re going to start at L1 and work our way up - the context of previous levels helps frame what “senior” means.

L1 - The Early Career Engineer

Scope

Congratulations on what is likely their first job as a software engineer! Their scope of responsibility is near zero - they are here to learn. That said, this is a real “sink or swim” test - I generally don’t expect folks to stay at L1 longer than one performance review.

At this stage, an L1 is expected to spend the majority of their time pairing with more senior engineers and making supervised changes to production code. I generally push with a “one week to meaningful PR” standard, even if “meaningful” here is a small bug fix or feature improvement.

The bulk of the time should be learning how software engineering works beyond the classroom and being a sponge. This is a great time to take advantage of any education bursaries / cert funding that the company may offer.

Impact

Minimal impact expected - the goal is going from being a net-negative addition to the team to break-even as fast as possible.

L2 - The Core

Scope

L2s are the core of any software engineering team. As per the “pyramid distribution”, this should be your widest step - L1s don’t really count as they should quickly graduate into becoming an L2.

An L2 should be considered a wholly independent engineer, able to take well-scoped, small items off the kanban/sprint backlog and deliver to the definition of done without much more supervision than code review. They should be encouraged to share responsibility of delivery of larger, more complex items with another L2 or a more senior engineer. They are generally not held accountable of delivery of larger OKR items, but may be held accountable of delivery of parts of those items.

Learning continues here, but primarily self-driven or through pairing with more senior engineers are on more complex problems. I expect L2s to start engaging in writing culture, and should generally be reading a lot of code and asking questions about why things are done a particular way.

Impact

As mentioned earlier, an L2 is generally about delivering small, well-scoped chunks of larger projects delivered by the immediate team. At this stage, I would expect them to start participating within a team’s/org’s culture of writing, if even only as a reviewer rather than instigator.

L3 - The Senior Engineer

Scope

L3s are your go-to executors, leading small groups of L2s to tackle larger, more complex projects - the ones that typically appear as key results in your OKRs (assuming you are a line manager/tech lead).

On top of the skills of L2, this is where I’d expect to start to see some level of specialisation - deep knowledge of a specific domain, and understanding how to apply that knowledge for the greatest effect. I also expect L3s to evolve beyond being an order-taker and start to generate work for themselves - improving the codebase, practices and so on. Similarly, I expect them to engage in writing culture, gathering consensus where decisions need to be made that impact the immediate team.

On top of that, I expect L3s to own delivery of small-medium projects from design through to delivery, with help from L4s and up. They should also ensure they are assisting in the growth of L2s into more L3s.

Impact

L3s are “get-it-done” folks - they are given a task and drive it to completion in a high quality way. Their contributions should always be beyond the immediate scope of the task they took on - becoming somewhat of a “1.1x” engineer.

Beyond that, this is where working only within the scope of the immediate team becomes a career limiting factor. To progress beyond that, scope and impact needs to exceed that of the immediate team boundaries.

Generally, I would expect an L3 and above to “go a little further” on everything - kind of like practising “5 Why’s” for each thing. This is especially true for things like requirements.

I would consider this a terminal level for some engineers.

L4 - The Staff Engineer

Scope

L4 gets in the murky world of “staff” engineering. At this level, scope expands beyond the immediate team and there is an expectation to own delivery of larger, complex projects that cut across many teams within a business function. L4s are likely wearing different hats different days of the week, and should generally be part of planning and longer term initiatives. “Planting trees they will not sit in” is a good mantra for the scope of work an L4 will be doing.

This is where it gets much harder to define what the appropriate scope is. In my experience, an L4 is generally owning design-through-delivery of projects that appear at the Senior Director/Vice President level. They are also where the buck typically stops for technical decision making within their team. For many, the scope is self-defining.

Impact

L4s should have outsized impact with most things they do - their job is making the whole team more effective and are essentially a right hand to the leaders of the team, if not a business unit leader. I would expect frequent contributions to writing culture, and ensuring that there is a focus on outcomes rather than output.

This is a terminal level for the majority of engineers.

L5+ - Architects, Principals, Etc

Beyond the L4, it gets harder to define the role according to of scope and impact so narrowly. Generally speaking, an L5 owns technical direction for a whole business unit or many business units, and are typically technical “faces” of an organisation at conferences and other industry events.

How an org defines this level is very org dependent, and so becomes hard to generalise. Ask your friendly neighbourhood architect what they see the role as - you’ll probably get different answers from different architects, even under the same org.

As this level rarely leaves time for day-to-day coding, many L4s are happy not stepping into it. There is also a level of politics at play at this level that is normally only exposed to the management track, which some engineers might also find unpalatable.

Wrap Up

So what is a senior engineer, anyway? It’s about scope and impact. Once an engineer takes on a wider scope of responsibility and is driving outcomes that are larger than they are, its safe to call them senior.

And how should we manage them? Generally by getting out of their way, by laying down track for them to follow to the next level, providing opportunities to own bigger things, and the support necessary to make them successful.

Regular restarts of compute units bring one obvious benefit - resolving memory leaks. Broadly, this is what the documentation means by “maintain the health of applications” running on the platform. Slow memory leaks are insidious, only appearing when taking a sufficiently zoomed out view of memory utilisation by the application, and a regular restart is one way of papering over that behaviour.

Historically, Rails applications used to be particularly hit by this problem, mostly due to how easy it is to accumulate objects. Real ones will remember the introduction of frozen_string_literal: true magic comment. Given Heroku’s history as a home for Rails apps, there should be no surprise as to why this regular cycling exists.

But there are other benefits, and for that, we need to re-visit The Twelve-Factor App.

The Twelve-Factor App is a little long in the tooth - Heroku CTO Gail Frederick is speaking about what a re-imagined set would look like for today at KubeCon Americas 2024 - but there are two factors that remain as powerful today as they did 13 years ago: Processes and Disposability.

The Art of Throwing Things Away

The combination of an ephemeral local filesystem and regular restarts means that you will not have local state and you will like it, dammit. And honestly, no local state is the best way to be - don’t accumulate cruft.

By enforcing the lack of local state, we can ensure that state lives somewhere more appropriate - a queue, a database, an object store and so on. This makes coordination significantly easier - shared-nothing services can be thrown away, scaled up and down without having to worry about state.

After all - always make state and coordination someone else’s problem.

We can take this concept further and architect our processes under the assumption that they can, and will, be interrupted at any time. This is why this “feature” gets a lot of flack - what to do about longer-running tasks?

If we’re fully embracing this model, we’re putting state elsewhere, like a queue. It’s entirely possible you will be mid-way through running a job and have your compute unit terminated. This introduces us to the architectural properties we need - idempotency, atomicity and reentrancy.

I’ve written about this elsewhere. By ensuring a given operation is idempotent and atomic, and the collection of operations are reentrant, we don’t need to concern ourselves mid-task interruptions - “just” retry. Embrace “at least once” processing. Kill your darlings. Let the cruft be swept away by a pod reaper or the dyno scheduler. By ensuring repeatable, predictable side effects and the ability to safely resume processing from a given state without leaving half-baked results lying around, we reduce our headaches considerably.

Time Keeps On Slipping

But what about tasks that are going to take a long amount of time anyway? The classic example is the “billing run” or similar end-of-month batch processes that require loading larger volumes of data into memory, doing some number crunching, then spitting out a result.

For these, we must instead decompose into smaller tasks. Providing an intermediate representation that can be stored (again, not locally!) somewhere and loaded, while keeping track of where in the sequence we are to resume processing after cancellation. If you are thinking “this sounds a lot like the MapReduce pattern” - you are right! MapReduce isn’t just for “big data”.

Then we run into the hard problem - long tasks that cannot be broken up into small, atomic tasks. In my line of work, the most familiar is the database dump.

Although Postgres natively supports backing up through the type of process outlined above (through continuous WAL archival, enabling “hot” backups), this isn’t very useful if you just want the data. Outside of logical replication, typically the only alternative is to use something like pg_dump.

Unfortunately, dumping a database, or even parts of a database, using pg_dump is a monolithic task - there are no valid “partial” dumps due to needing access to the transaction snapshot, which if pg_dump is interrupted, no longer exists. For very large datasets, it is entirely possible that even on adequate hardware and running in parallel mode pg_dump, the odds of the process being interrupted are very high.

So what to do about these monolithic tasks? “Just don’t restart” I hear you cry. And for these very particular types of work, you are correct. However, such behaviour should be opt-in instead of opt-out - discourage this kind of task in favour of composable sequences of tasks.

Update 2024-11-09: An earlier version of this post referred to Gail Frederick as Heroku CEO instead of Heroku CTO. This has been corrected.

So this Okta security advisory was doing the rounds, and was pretty transparent, all things considered:

On October 30, 2024, a vulnerability was internally identified in generating the cache key for AD/LDAP DelAuth. The Bcrypt algorithm was used to generate the cache key where we hash a combined string of userId + username + password. During specific conditions, this could allow users to authenticate by only providing the username with the stored cache key of a previous successful authentication.

bcrypt has a well-known limitation that it has a maximum password length of 72 bytes (or 72 ASCII characters). By using bcrypt as a hash function for a cache key that is composed by concatenation, we can infer that clientId is 20 bytes, and therefore a username of 52 or more bytes results in the password becoming irrelevant.

We can easily demonstrate this in code:

import bcrypt
import secrets
import sys

client_id = secrets.token_hex(10)
client_username = secrets.token_hex(26)
client_password = secrets.token_hex(32)

cache_key = client_id + client_username + client_password

hashed_key = bcrypt.hashpw(cache_key.encode(), bcrypt.gensalt())

check_password = sys.argv[1]
check_key = client_id + client_username + check_password

print("Correct Password: {}".format(client_password))
print("Entered Password: {}".format(check_password))

if bcrypt.checkpw(check_key.encode(), hashed_key):
    print("Password is correct.")

$ python main.py foobar
Correct Password: 3b34021604ace062aa21a77f3461dc0289191988c248467f2401c9540939ec5c
Entered Password: foobar
Password is correct.

Alternatively, you can hash the input with a constant-length hash function, like SHA-256, producing a 64 character hex digest:

@@ -1,4 +1,5 @@
 import bcrypt
+import hashlib
 import secrets
 import sys

@@ -7,14 +8,18 @@ client_username = secrets.token_hex(26)
 client_password = secrets.token_hex(32)

 cache_key = client_id + client_username + client_password
+hashed_input = hashlib.sha256(cache_key.encode()).hexdigest()

-hashed_key = bcrypt.hashpw(cache_key.encode(), bcrypt.gensalt())
+hashed_key = bcrypt.hashpw(hashed_input.encode(), bcrypt.gensalt())

 check_password = sys.argv[1]
 check_key = client_id + client_username + check_password
+hashed_check_input = hashlib.sha256(check_key.encode()).hexdigest()

 print("Correct Password: {}".format(client_password))
 print("Entered Password: {}".format(check_password))

-if bcrypt.checkpw(check_key.encode(), hashed_key):
+if bcrypt.checkpw(hashed_check_input.encode(), hashed_key):
     print("Password is correct.")

Of course, this introduces a possibility of a collision attack, though at the time of writing not feasible. Another problem with this approach is password shucking, as well as a litany of other issues of combining bcrypt with other hash functions. Just use a better hash algorithm designed for password storage.

Interestingly, this was “fixed” by changing the hash function to use PBKDF2 - this is materially worse than bcrypt, but it is FIPS compliant. I can only hope this was implemented with a minimum work factor of 600K and using HMAC-SHA-256 as the internal hash function. My preference would be to use argon2id or scrypt:

from argon2 import PasswordHasher
from argon2.low_level import Type
from argon2.exceptions import VerifyMismatchError
import sys
import secrets

client_id = secrets.token_hex(10)
client_username = secrets.token_hex(26)
client_password = secrets.token_hex(32)

cache_key = client_id + client_username + client_password

ph = PasswordHasher(type=Type.ID)
hashed_key = ph.hash(cache_key)

check_password = sys.argv[1]
check_key = client_id + client_username + check_password

print("Correct Password: {}".format(client_password))
print("Entered Password: {}".format(check_password))

try:
    ph.verify(hashed_key, check_key)
    print("Password is correct.")
except VerifyMismatchError:
    print("Password is incorrect.")

$ python argon2id.py foobar
Correct Password: 720d5d3959f20cbee3f05efc7289f4c307ae7add38c821cfa5708631f4ee77c6
Entered Password: foobar
Password is incorrect

Read this post from Latacora for more “cryptographic right answers”.

The use of bcrypt here with an input that is not well restricted and can easily blow through the known 72-byte limitation is a flaw that should have been caught through design review, but I can understand why - if you don’t have the cryptographic expertise, this is easily missed.

That said, some bcrypt libraries refuse to accept inputs longer than 72 bytes, such as the Go implementation:

package main

import (
	"fmt"

	bcrypt "golang.org/x/crypto/bcrypt"
)

func main() {
	password := []byte("894d8772dcff261840ba07e6ceb271a3803483a1900c2719c76561de735b0dab5318c6895085675d5ee315cf10c0a1a6df4e3bbc7681d2ea502ca4de899b6195de410b20")
	fmt.Println("password_length: ", len(password))

	_, err := bcrypt.GenerateFromPassword(password, bcrypt.DefaultCost)
	if err != nil {
		panic(err)
	}
}

$ go run main.go
password_length:  136
panic: bcrypt: password length exceeds 72 bytes

This reinforces the need to have sensible defaults.

Update 2024-11-03: include SHA-2 hashing as alternative mitigation.

GraphQL Is Insecure By Default

Well, at least in the JavaScript implementation. Cue the bug bounties:

• GraphQL Information Disclosure
• GraphQL Information Disclosure
• GraphQL Information Disclosure
• GraphQL Incorrect Cost Handling
• GraphQL Denial of Service

It is well known that there are several easily accessible Denial of Service vectors on GraphQL endpoints.

Unlimited Power Tokens

One of the most egregious was CVE-2022-37734, where it is possible to construct a query with a very large number of tokens to put pressure on the lexer - a variant of the Billion Laughs attack. This was fixed in graphql-java, by stopping parsing after the configured maximum number of tokens (including whitespace) is reached, extending the original fix (https://github.com/graphql-java/graphql-java/pull/2549) that adds a default limit of 15,000 tokens.

However, in the reference implementation graphql/graphql-js, a similar fix was added that did not include a default limit - by default it is left undefined, and so the DoS mitigation is not active by default.

Some other implementations also lack this:

• graphql-python/graphql-core
• absinthe-graphql/absinthe

There are no doubt other implementations of the spec that don’t set this by default.

In many cases this can be mitigated by setting a limit on the size of the request body, but having it configured and built into the lexer is vital.

Aliases and Directives

As mentioned in the blog post earlier, a less easily solved problem is that of alias overloading and directive overloading.

In short, alias overloading allows performing the same operation multiple times within the same request - batching. Directive overloading “spams” the same directive multiple times in the same operation.

In order to mitigate against these, we need to traverse the parsed query AST and count against them. Again, this is something that most implementations do not have guards for by default, often falling back to query complexity analysis, as well as complex rate limiting implementations, such as GitHub’s. I’ll wait while you digest that particular document. However, there be dragons: GraphQL Incorrect Cost Handling disclosed to Shopify indicates that this is a hard thing to get right.

Some popular GraphQL frameworks such as RedwoodJS and Apollo provide some defaults, but in the case of the latter, this is an Enterprise only feature.

Again, this kind of thing is surprising.

Information Disclosure By Default

Many GraphQL implementations enable schema introspection by default, which allows attackers to perform easy reconnaissance.

It is considered good practice to disable introspection in production, and requires an explicit opt in. The linked docs from Apollo show how for their framework, and in vanilla graphql/graphql-js:

import { createHandler } from "graphql-http/lib/use/express";
import { NoSchemaIntrospectionCustomRule } from "graphql";

const graphQLHandler = createHandler({
  // ...
  validationRules: [NoSchemaIntrospectionCustomRule],
});

I kind of understand why this is enabled by default - it is helpful in development - but like a lot of things GraphQL, feels like a loaded footgun.

Introspection Abuse

Besides introspection leaking data, it can also be used in attacks that result in deeply recursive queries. As a result, controlling query depth is also required. Reviewing graphql-python/graphene, this is something that needs to be enabled explicitly through an additional validator.

What Does A “Secure By Default” Endpoint Looking Like Anyway?

Taking graphql/graphql-js as the reference implementation, lets take the “getting started” example and work on making it secure.

Here is the starting example:

import express from "express";
import morgan from "morgan";
import { createHandler } from "graphql-http/lib/use/express";
import { schema } from "./schema";

const root = {
  hello: () => "Hello, world!",
};

const app = express();

app.use(morgan("common"));
app.use("/graphql", createHandler({ schema, rootValue: root }));
app.listen(3000, () => console.log("Server running on port 3000"));

First of all, we need to protect ourselves against large request bodies. Straight forward enough with body-parser:

 import morgan from "morgan"
 import { createHandler } from "graphql-http/lib/use/express"
 import { schema } from "./schema"
+import bodyParser from "body-parser"

 const root = {
     hello: () => "Hello, world!"
@@ -10,5 +11,7 @@ const root = {
 const app = express()

 app.use(morgan("common"))
+app.use(bodyParser.json({ limit: "64kb" }))
+app.use(bodyParser.urlencoded({ extended: true, limit: "64kb" }))
 app.use("/graphql", createHandler({ schema, rootValue: root }))
 app.listen(3000, () => console.log("Server running on port 3000"))

Next, we need to add some default validation rules and set our maxToken to some sensible value:

-app.use(morgan("common"))
-app.use(bodyParser.json({ limit: "64kb" }))
-app.use(bodyParser.urlencoded({ extended: true, limit: "64kb" }))
-app.use("/graphql", createHandler({ schema, rootValue: root }))
-app.listen(3000, () => console.log("Server running on port 3000"))
+const handler = createHandler({
+  schema,
+  parse: (query) => parse(query, { maxTokens: 5000 }),
+  validationRules: [
+    ...specifiedRules,
+    NoSchemaIntrospectionCustomRule,
+  ],
+});
+
+app.use(morgan("common"));
+app.use(bodyParser.json({ limit: "64kb" }));
+app.use(bodyParser.urlencoded({ extended: true, limit: "64kb" }));
+app.post("/graphql", handler);

Now, we need to add some specific guards. The nice people over at escape.tech have created a practically mandatory package - graphql-armor - that exposes some helpful rules to mitigate the above:

 validationRules: [
   ...specifiedRules,
   NoSchemaIntrospectionCustomRule,
+    maxDepthRule({
+      exposeLimits: false,
+    }),
+    costLimitRule({
+      exposeLimits: false,
+    }),
+    maxDirectivesRule({
+      n: config.graphqlMaxDirectives,
+      exposeLimits: false,
+    }),
+    maxAliasesRule({
+      n: config.graphqlMaxAliases,
+      exposeLimits: false,
+    }),
 ],

We set exposeLimits to false to prevent leaking information to attackers about what limits are in place.

The final setup is shown in this repo: mble/graphql-tirefire.

This leaves us with something very similar to RedwoodJS’s configuration.

In an ideal world, these packages and their default configuration would be baked into the underlying GraphQL implementations themselves, to provide more of a “zero config” approach.

By providing sensible defaults, we can enable our users to be more successful out of the gate, and keep the web safer by default.

Undercommunication Will Destroy You

Shortly after I met my partner, circumstances required us to maintain a long distance relationship between London, UK and sunny California, USA - a time difference of 8 hours. As my day finished, her day was just beginning, requiring us to be deliberate about maximising the time we could spend with each other during our waking hours, and figuring out ways for us to communicate effectively when we didn’t cross over - voice notes, letters, email and so on. Leaving little things for her to wake up to.

Operating a remote team is a lot like that! Especially with the time zone differences I’ve had to overcome. There are a few key tenets, including using the tools for a culture of writing.

There Is No Communication Like Overcommunication

The old adage is to say something several times before it is truly heard, and the goes doubly for remote teams. Say something in a meeting? That meeting better be recorded, transcribed, summarised (see bottom line, up front) and the key points posted in your collaboration tool of choice. LLM-based tools can help a lot with this, but it is still your responsibility to ensure the summary is accurate and useful - I personally prefer to write my executive summaries by hand.

Why do this? Team members may be unable to make a given meeting, either temporarily or permanently, but no one should ever be out of the loop. Some folks may consider holding “off the record” meetings where no recording takes place - in those cases, take notes, respecting the speaker(s) by following the Chatham House Rule and/or being judicious about omitting information from notes, but share it with team members who could not make it verbally. Don’t let information fall into a hole.

As a result of the timezone disparities, the team has opted for a written-first, asynchronous-first approach to information sharing. Daily standups are written, most projects are driven through task trackers, documents and IM, with sparing use of synchronous meetings - usually for our retrospectives and weekly team meetings to ensure we have some regular face time.

On that note, we strive to have regular in-person off-sites. These are partially structured, to allow for high value synchronous collaboration, but also leave some unstructured time to allow the team to “be people” together.

The Team Is The Unit Of Delivery

I think I first heard this term back in 2014, maybe 2015? It comes from the UK’s Government Digital Service, and has been the way I feel about teams ever since.

What does this mean? Through individual performance and collaboration, we deliver value to our customers. “The team” transcends my reports into our wider organisation and the cross-functional relationships we build.

What About The Individual?

This is not to say I don’t value the individual - we must recognise that the team is successful because of individual skillsets and ability, and that we cover each other’s bases. Shipping a product as complex as ours is simply not possible by an individual. I generally recommend that my staff maintain a brag doc, and I track their individual contributions within the context of providing feedback and performance management.

Performance management aside, I try and avoid focusing on “individual” failures - we succeed and fail as one unit.

Session Musicians, Not Rockstars

Assembling a team that can function as one unit and be healthy is tricky, especially in a remote environment - there can be interactions that you are not privy to via DMs, private channels and so on. As a result, there must be a high level of trust between each team member, as well as the staff and the management layer. It is a lot like constructing a high performing sports team, and the relationship between the front office and the back office.

I’m often heard saying that I’ve learned more from the great sports coaches of our time (Phil Jackson, Nick Saban, Pep Guardiola) than most management trainings and material, and part of this is construction of a strong team. I don’t enjoy managing egos, so intentionally select for folks who display high levels of coachability and curiosity with the right amount of appropriate professional drive.

As an example of this practice, consider session musicians - consummate professionals quietly behind some of the most influential music produced in the 20th and 21st centuries. A great example of this were the Swampers, with Rick Hall conjouring a kind of magic rarely seen before or since.

Sure, rockstars are great for the spark of inguinity and allowing lightning to strike, but software engineering is a long game, and niche of managed database services and database service reliability engineering even more so. I need the longevity of a good session group.

Work Is Fungible, Team Members Are Not

Along the lines of longevity, one of the more painful lessons learned is that work is fungible, team members are not.

In short, teams can learn (as a function of time and funding) pretty much anything you need them to, so moving work to different teams is “free”. However, breaking that team up to align closer to the work is extremely expensive.

The thesis is that, as a team is one unit, breaking that unit creates a new team. Think of a team as a graph of nodes and edges - when you pop a node from the graph, the graph has changed. Adding a new node or nodes to the graph? A new graph.

Thinking this through even more, a software engineering team is generally in a metastable state. There is probably a theoretic stable state of extremely strong bonds between individuals, but in real life this is rarely, if ever, achieved - even the very long term session groups or sports teams eventually break up due to internal or external forces.

We want to maximise the stability of the team, and to do that, we bring the work to the team, rather than take the team members to the work. As an example, a team that ostensibly works on an integration product picked up work to enable billing for new regions. The originally proposed option was to “air drop” members from that team into the billing the team, but doing so would have unacceptably disrupted the effectiveness of the original team. Moving the work has much lower cost.

The PR Machine

Last but not least, one of the key lessons I’ve learned in remote leadership has been that as all interactions are intentional, you have to be the PR machine for your team. This loops back around to overcommunication - if a major project is shipped and no one hears about it, did it even happen?

Your job is to take your team’s accolades and broadcast them wide and far. Have your team be known as one that Gets Stuff Done. Have your team be known as one that is a benchmark for quality and reliability.

A lot of this work is a continuation of generating artefacts for your teams work, and using those artefacts as part of your PR campaign towards other teams and upper leadership. This is not entirely altruistic - a successful, high profile team reflects well on you as a manager, furthering your career.

A previous version of this article used an AI-generated header image. This has been replaced with something hand drawn.

You can see the following implementation in this diff on my fork of the Postgres repository.

Contents

• Adding A New Option
• Filtering Event Triggers
• Filtering Event Trigger Returning Functions
• Testing

Adding A New Option

To begin with, we need to add a new switch to pg_dump that will allow us to exclude event triggers on command. The switches generally come in two varieties - short and long. As pg_dump already uses most of the alphabet in the short options (abBcCdeEfFhjnNOpRsStTUvwWxZ), we’ll use a long option: --no-event-triggers. This aligns with similar options in pg_dump like --no-comments and --no-subscriptions.

This is fortunately straightforward to implement - in pg_backup.h we add a new boolean variable to the _dumpOptions struct:

typedef struct _dumpOptions
{
    ConnParams cparams;
    /* ... */

    /* flags for various command-line long options */
	int			disable_dollar_quoting;
	int			column_inserts;
	int			if_exists;
    /* ... */
    int			outputNoEventTriggers;
    /* ... */
} DumpOptions;

In pg_dump.c, we add the option to the long_options array:

static struct option long_options[] = {
    /* ... */
    {"no-table-access-method", no_argument, &dopt.outputNoTableAm, 1},
    {"no-tablespaces", no_argument, &dopt.outputNoTablespaces, 1},
    {"no-event-triggers", no_argument, &dopt.outputNoEventTriggers, 1},
    /* ... */
};

And finally, we add the option to the help output:

static void
help(const char *progname)
{
    printf(_("%s dumps a database as a text file or to other formats.\n\n"), progname);
    /* ... */
    printf(_("  --no-event-triggers          do not dump event triggers\n"));
}