Procella - YouTube’s analytical column store

Procella is a horizontally scalable, eventually consistent, distributed column store leveraging lambda architecture to support both realtime and batch queries [1]. Let’s define those terms one at a time:

  • Horizontally scalable means YouTube can spin up more machines and Procella will distribute queries to the new machines automagically.
  • Procella doesn't support strong database isolation levels (the I in ACID). Queries support read uncommitted isolation which can cause dirty reads. A dirty read occurs when a transaction sees uncommitted changes from another transaction.
  • The lambda architecture means there are two write paths. The first path, called the real-time write path, aims for low latency and writes into an unoptimized row-store that’s immediately available for queries. The second, called the batch write path, ingests large quantities of data into an optimized columnar format. The query path merges changes from both real-time and batch storage to unify results.
  • Distributed means the data is sharded across multiple servers.
  • A column store refers to how the database physically stores data. See the figure below for how we might store a table with three columns: event_id, time, and path.
Row store versus column store
Figure: The difference in data layout between a row store and a column store.
Continue reading

Passing on Lastpass: migrating to 1Password

After implicating Lastpass as a culprit in browser slowness, I ripped it out and replaced it with the significantly snappier 1Password. Lasspass slows down the browser in a litany of ways:

  • Chrome compiles the Lastpass JavaScript bundle on every page load. Crucially, Lastpass injects the script so that Chrome blocks the first contentful paint. Lastpass takes 70ms to compile on a relatively high-end MacBook pro. This means Lastpass adds a 70ms delay to all page loads.
  • The Lastpass vault is hilariously slow. Opening up account properties in the Lastpass vault pegs the CPU at 100% for 15 seconds.

  • On a more subjective note, interacting with the vault and form-filling with Lastpass feels much more janky than 1Password.

The migration to 1Password was surprisingly straightforward. The 1Password documentation breaks down into two easy steps with nine total subtasks. I completely migrated to 1Password in less than 10 minutes.

Lastpass performance

Let’s examine Lastpass performance on a simple page,, to evaluate the performance impact of the extension.

Lastpass Chrome performance timeline
Figure: The performance timeline of loading with Lastpass. Notice that the "Evaluate Script" for Lastpass blocks the first contentful paint (FCP). Onloadwff.js is part of the Lastpass extension.
Continue reading

Cutting down the Ubuntu MOTD down to size

The Ubuntu message of the day (MOTD) is a chatty affair. A MOTD sends information to all users on login—A recent login message greeted me with 42 lines of questionable value.

Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1021-aws x86_64)

 * Documentation:
 * Management:
 * Support:

  System information as of Mon Apr 20 01:01:24 UTC 2020

  System load:  28.97              Processes:             10
  Usage of /:   28.9% of 48.41GB   Users logged in:       0
  Memory usage: 61%                IP address for enp4s0:
  Swap usage:   0%

 * Kubernetes 1.18 GA is now available! See
   for docs or install it with:

     sudo snap install microk8s --channel=1.18 --classic

 * Multipass 1.1 adds proxy support for developers behind enterprise
   firewalls. Rapid prototyping for cloud operations just got easier.

  Get cloud support with Ubuntu Advantage Cloud Guest:

 * Canonical Livepatch is available for installation.
   - Reduce system reboots and improve kernel security. Activate at:

99 packages can be updated.
1 update is a security update.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

*** System restart required ***
Last login: Sun Apr 19 03:41:04 2020 from
Continue reading

Gorilla Time Series Database

Gorilla is an in-memory, time series database from Facebook optimized for writes, reading data in a few milliseconds, and high availability. Facebook open-sourced the code as Beringei, but the maintainers archived the repo. At its core, Gorilla is a 26-hour write-through cache backed by durable storage in HBase, a distributed key-value store. Gorilla’s contributions include a novel, streaming timestamp compression scheme.

Paper: Gorilla: A Fast, Scalable, In-Memory Time Series Database

Gorilla architecture
Figure: The architecture of a Gorilla cluster. A cluster runs in 2 regions. Each region contains multiple instances. Each instance contains shards that store time series data.
Continue reading

Fix sluggish ZSH shells with lazy loading for slow scripts

I care immensely for having a snappy terminal startup and rebel against demands to blindly source scripts in my ~/.zshrc.

The most recent challenge was Node Version Manager. NVM commits the mortal sin of automatically adding itself to the ~/.zshrc file in the following incantation:

curl -o- "${URL}" | bash

# Added to ~/.zshrc.
[ -s "$NVM_DIR/" ] && \. "$NVM_DIR/"

The NVM script requires 800ms to load, a debt paid every time I startup a shell. Consulting internet canon, we calculate how much time it’s worth spending to reclaim the 800ms of shell startup.

Continue reading

Create a Go web server from scratch with Linux system calls

A web-server with Linux syscalls.

One itch I’ve wanted to scratch for a while is to create a web-server from scratch without relying on libraries and without first inventing the universe. I’ve also wanted a chance to take Go for a spin. I’ll cover how to create a web server in Go using Linux system calls.

Completed Code at Github: scratch_server.go


The Go net package is a full-featured, production ready library. We’ll skip the following features:

  • HTTP 100 Continue support
  • TLS
  • Most error checking
  • Persistent and chunked connections
  • HTTP Redirects
  • Deadline and cancellation
  • Non-blocking sockets
Continue reading

Advanced queries with Bazel

I often need to query complex things with Bazel, an open-source build system from Google that focuses on performance and correctness by enforcing hermetic builds. For a more complete list of examples, see the official Bazel query how-to.

Find all tests marked as size = small that have a database dependency

Google tests have a specific size (small, medium, large) with strict time-outs. If a small test exceeds 60 seconds, the test fails. For tests involving a database, the tests need to be marked as medium to avoid flaky timeouts.

bazel query '
  attr(size, small, //server/...:all)
Bazel test sizes that depend on a test database
Figure: Dependency graph of tests on the test database with the size attribute.
Continue reading