Torsten Curdt writing about tech

Scanning QR Codes on Mac for Automation

2020-01-02T00:00:00Z

It started with an invoice. I wanted to scan the QR code on a German Überweisung (payment transfers) and have it open directly in my banking software. No typing, no copy-paste errors on those long IBANs. Seemed like it should be easy.

It wasn’t. On iOS, the camera app handles QR codes natively. But not payment transfers either. On the Mac? Nothing. There was no straightforward way to scan a barcode from the built-in camera and do something useful with the result.

Then I ran into the same wall again with two-factor authentication. A service showed me a QR code containing a TOTP secret key and I was supposed to scan it with my phone. But I was sitting right in front of my Mac, staring at the code on my screen. Why couldn’t I just scan it right there?

That frustration is what led me to build ScanCode.

What ScanCode does

ScanCode scans QR codes and barcodes on macOS. It reads from the built-in camera or directly from the screen. Once it decodes the content, it can act on it immediately or pass it along to scripts for automation.

The first version scanned codes and handed the raw data to external actions via Automator workflows or shell scripts. Everything else was up to you.

Version 2 is different. Most common actions are now built in. ScanCode recognizes the type of content and offers the right action automatically. You can still pipe data into custom scripts, but for everyday use cases you no longer need to.

ScanCode triggering a Bash Script

Here is what it handles out of the box:

Type	What’s encoded	What ScanCode does
URL	`http://` or `https://` links	Opens in your default browser
SEPA	Payment data (EPC, Bezahlcode, payto)	Opens a pre-filled transfer in your banking app
WiFi	Network name, encryption, password	Copies the password to your clipboard
OTP	`otpauth://` with secret key and parameters	Copies/opens the key in your authenticator
Email	`mailto:` with address, subject, body	Opens your email client with message ready
Contacts	vCard or MeCard data	Adds to your address book
Events	Calendar dates, times, locations	Creates a calendar entry
Geo	`geo:` URI with coordinates	Opens in Maps
Bitcoin	`bitcoin:` URI with wallet address	Copies address or opens wallet app

For anything that doesn’t match a known pattern, ScanCode can copy the raw text to the clipboard or save it to a file. Combined with shell scripts, you can build custom workflows for whatever you need.

SEPA transfers

This is still the main use case for me. QR codes on European invoices encode the payment information for a bank transfer. No manual entry, no transcription errors.

What’s inside the QR code

There are several standards for encoding SEPA payment data in QR codes.

EPC QR codes are the most widespread in the EU. The European Payments Council defined the standard and it’s used in Austria, Belgium, Finland, Germany and The Netherlands. The encoding is straightforward. Lines of text, one field per line:

BCD
002
1
SCT
RLNWATWW
Doctors Without Borders
AT973200000000518548
EUR1500.99


Emergency Donation

BCD identifies it as an EPC code. If you want to generate one yourself, there are online tools for that.

Bezahlcode was a German standard using a bank:// URI scheme:

bank://singlepaymentsepa?name={NAME}&reason={REASON}&iban={IBAN}&amount={AMOUNT}

The official website is long gone but the format is still understood by some banking apps.

The Swiss QR bill uses its own format, optimized for the Swiss payment system. If you handle Swiss invoices, you’ll encounter this variant instead of EPC.

The payto URI defined in RFC 8905 is the standards-track approach:

payto://iban/{IBAN}?amount=EUR:{AMOUNT}&message={MESSAGE}

Adoption is still limited, but it’s the one backed by an RFC.

Getting it into your banking app

ScanCode recognizes all these formats. For MoneyMoney (my banking software of choice on Mac), it converts the scanned data into a URL scheme that MoneyMoney understands and opens a pre-filled SEPA transfer. No intermediate steps.

MoneyMoney supports both the bank:// and payto:// URL schemes. ScanCode handles the conversion from EPC or whatever format the QR code uses.

Other banking apps with URL scheme support work the same way. The point is that you go from paper invoice to pre-filled transfer form in a single scan.

A word on security

Be careful with QR codes from untrusted sources. A QR code is just a way to encode text and that text could be a malicious URL or a crafted payload. The same caution you’d apply to clicking unknown links applies to scanning unknown codes.

ScanCode allows you to show the decoded content before taking action, so you always get a chance to inspect what you’re about to open.

From camera to workflow

The gap between phones and Macs in terms of QR code handling has always been strange. Phones scan codes effortlessly. On the Mac - not so much. ScanCode fills that gap. Whether it’s a payment on an invoice, a WiFi password at a hotel, or a TOTP key for your authenticator, the flow is the same. Point, scan, done.

Modulo of Negative Numbers

2011-06-28T00:00:00Z

The modulo (often just called “mod”) gives the remainder of a division. In 1801 Gauss published a book covering modular arithmetics. Later a widely accepted mathematical definition was given by Donald Knuth:

mod(a, n) = a - n * floor(a / n)

An integer division followed by multiplication finds the largest multiple of n that fits into a. Subtracting that from a yields the remainder - the modulo.

That sounds straightforward enough. But there’s a subtlety hiding in that floor that trips up programmers across languages.

Remainder vs modulo

Most programming languages use the % operator and call it “modulo”, but what they actually implement is the remainder of a truncated division. Truncated division rounds toward zero, while the mathematical modulo uses floored division (rounding toward negative infinity). For positive numbers the results are identical. For negative numbers, they diverge.

Consider -13 mod 3. The floored division gives floor(-13/3) = -5, so the modulo is -13 - 3 * (-5) = 2. A positive result. The truncated division gives trunc(-13/3) = -4, so the remainder is -13 - 3 * (-4) = -1. A negative result.

This is the root of a whole class of bugs.

Restricting bounds

In programming, the modulo operator is commonly used to wrap an index around the bounds of an array or ring buffer.

values = [ 3, 4, 5 ]
index = 5
value_at_index = values[ index % values.length ]

For the above example 5 mod 3 = 2 following the definition is 5 - floor(5/3)*3 = 2. No matter the value of index, the array bounds are met.

But is that really the case?

What happens if the dividend is negative? Turns out the result of the modulo operation on negative numbers depends on the language you are using.

While the code looks pretty much the same across languages, the results fall into mostly two camps.

Language	13 mod 3	-13 mod 3	13 mod -3	-13 mod -3
C	1	-1	1	-1
C#	1	-1	1	-1
C++	1	-1	1	-1
Elixir	1	-1	1	-1
Erlang	1	-1	1	-1
Go	1	-1	1	-1
Java	1	-1	1	-1
Javascript	1	-1	1	-1
Kotlin	1	-1	1	-1
Nim	1	-1	1	-1
OCaml	1	-1	1	-1
PHP	1	-1	1	-1
Rust	1	-1	1	-1
Scala	1	-1	1	-1
Swift	1	-1	1	-1
V	1	-1	1	-1
Crystal	1	2	-2	-1
Haskell	1	2	-2	-1
Lua	1	2	-2	-1
Python	1	2	-2	-1
Ruby	1	2	-2	-1
Dart	1	2	1	2

The first group (C, Java, JavaScript, etc.) implements truncated division and the remainder takes the sign of the dividend. The second group (Python, Ruby, etc.) implements floored division and the result takes the sign of the divisor, matching the more mathematical definition.

Dart is a bit of an outlier. Its % operator always returns a non-negative result, regardless of the sign of either operand.

Zig’s principled approach

Zig deserves a special mention for being the most explicit about the distinction. Instead of picking one behavior and hoping developers remember which one it is, Zig provides two separate builtins: @rem for the truncated remainder and @mod for the floored modulo. If you pass a negative divisor, both operations produce a compile-time error. No ambiguity, no surprises.

Zig	13 mod 3	-13 mod 3	13 mod -3	-13 mod -3
@rem	1	-1	error	error
@mod	1	2	error	error

This is arguably the most clear design there is.

Language-level fixes

Some languages provide built-in alternatives to the default % behavior. Java added Math.floorMod() in Java 8, which gives the mathematically correct floored modulo. So if you’re a Java developer bitten by this before - there’s a one-liner fix now.

Math.floorMod(-13, 3) // returns 2, not -1

Python and Ruby don’t need this since their % already does floored division by default.

For languages without a built-in, a simple and efficient workaround is to check the sign.

int mod(a, b) {
  c = a % b
  return (c < 0) ? c + b : c
}

Alternatively, you can apply the modulo twice.

int mod(a, b) {
  return ((a % b) + b) % b
}

The even/odd trap

Another related pitfall is testing whether a number is odd using the modulo operator. Given that -13 % 2 returns -1 in many languages, you should always compare against 0 rather than 1.

bool is_odd(int n) {
    return n % 2 != 0; // correct: works for both 1 and -1
}

If performance matters and you’re dealing with powers of two, the modulo can be replaced with a bitwise operation since x % 2^n == x & (2^n - 1) for positive divisors. So a faster is_odd becomes:

bool is_odd(int n) {
    return (n & 1) != 0; // note the parentheses to ensure precedence
}

In summary

The modulo operator is one of those things that seems simple until it isn’t. The majority of languages implement truncated remainder rather than the mathematical modulo. The difference only shows up with negative numbers, exactly the path that is often not tested for.

Know which camp your language falls into, use built-in alternatives like Math.floorMod() where available, and when in doubt - write a small wrapper. For a deeper dive, see the Wikipedia article on the modulo operation.

Time Series Rollups and Aggregation at Scale

2011-05-15T00:00:00Z

Many sites provide statistics for their users’ content. Play counts, page views, API calls. Usually presented as a graph over time. This is easy when aggregating over the raw events data. But that’s neither particularly fast nor scalable. As soon as the number of events or the time window gets too big this approach is no longer feasible.

Today there are purpose-built time series databases that handle much of the heavy lifting. But underneath, they all rely on the same fundamental patterns. Understanding those patterns helps you pick the right tool, configure it properly, and know when to build something custom.

The landscape

The number of time series databases has exploded over the past decade. Here’s a rough map.

TimescaleDB extends PostgreSQL with continuous aggregates that automatically maintain pre-computed rollups. As new data arrives, only affected time buckets are recalculated. If you’re already running Postgres, it’s the easiest on-ramp.

ClickHouse takes a different angle. Its materialized views trigger on insert, shifting aggregation work from query time to write time. You define a raw table, a rollup table, and a view that connects them. Fast column-oriented storage makes it popular for analytics workloads.

InfluxDB was built specifically for time series from the start. Version 3.x includes a downsampler plugin that aggregates measurements over configurable time intervals using functions like avg, sum, min, max. It also supports external stream processing through Kafka-based pipelines.

VictoriaMetrics positions itself as a faster, more scalable drop-in replacement for Prometheus. While Prometheus itself has no built-in downsampling or long-term storage, extensions like Thanos add exactly that: automatic downsampling (5-minute resolution after 40 hours, 1-hour after 10 days) plus object storage for retention beyond what a single Prometheus instance can hold.

For large-scale batch processing of time series, Apache Spark and libraries like Flint offer distributed computation across clusters. This is the territory where you’re processing terabytes of historical data rather than serving real-time dashboards.

On the managed side, AWS Timestream offers a serverless time series database that automatically moves data between in-memory and cold storage tiers based on age.

Different tools, same underlying ideas.

The fundamentals

Every one of these systems implements some variation of three core concepts: pre-aggregation, rollup strategies, and resolution tradeoffs.

Pre-aggregation

Querying raw events is O(n) where n is the number of events. For a billion events, that’s a billion rows to scan. Pre-aggregation collapses events into time buckets ahead of time. Instead of counting a million page views for March 2024, you store a single counter: 2024-03 = 1000000. Query time becomes nearly constant regardless of how many events actually occurred.

Rollup strategies

A natural choice is to roll up data into multiple time bucket granularities. By creating counters per different time units, the number of operations can be kept at a minimum while still providing fast aggregation over large time windows.

Calculating an aggregate then becomes as simple as adding a few counters. It’s the sum of the most covering aggregates that fit into the given time interval.

Getting the total count for the interval from 2008-11-30 until 2011-02-03 means getting 8 values and adding them.

The 8 rollup values used by the query

Choosing the time units defines the read pattern and the number of counters needed. A natural combination would be year+month+day but year+week+day could work too. Even just rolling up into day buckets might be enough. For an informed decision, check against real user queries and calculate the required storage.

Resolution tradeoffs

More granularity means more storage. Fewer rollup levels mean faster writes but slower reads for large time ranges. Most systems let you define retention policies: keep full-resolution data for 30 days, hourly rollups for a year, daily rollups forever. This is exactly what Thanos does for Prometheus, and what TimescaleDB’s continuous aggregate policies automate.

The right balance depends entirely on the use case. Real-time dashboards need second-level granularity for the last hour but nobody needs per-second data from three years ago.

The fundamentals cover the read path, but the events still need to be rolled up. How that happens has evolved significantly.

Lambda architecture

The simplest approach is a batch job that periodically recomputes all rollups from raw events. But pure batch creates lag. The Lambda architecture closes this gap by running two parallel systems. A batch layer provides rollups up until a certain point in time. Everything past that is served from a fast in-memory layer.

Combining Rollups and Real Time Counters

The in-memory layer can even be transient since the eventual source of truth is always the batch layer. SoundCloud used a similar architecture for their play count statistics, backed by Cassandra for storage and Memcached for caching. Their system called “Stitch” would regenerate aggregated data buckets through daily and monthly batch jobs while a speed layer handled real-time updates using distributed counters.

The downside of Lambda is maintaining two separate code paths for the same business logic. The batch pipeline and the streaming pipeline need to produce consistent results, and keeping them in sync is tricky to get right.

The batch layer in practice

A naive approach is to increment counters as events arrive. A single event fans out into multiple writes:

event at 2011-03-14 =>
    2011 += 1
    2011-03 += 1
    2011-03-14 += 1

This puts a lot of write pressure on the store. An increment is not just a write but also a read. And all increments should ideally happen in a single transaction.

Another option is to generate the counters from log files using a MapReduce job. The mapper emits data for per-track and global counters:

def mapper(lines):
    result = []
    for date, track in lines:
        result.append((track, date))   # per-track counter
        result.append(("*", date))     # global counter
    return result

After mapping, the data gets sorted. The reducer detects changes in the incoming data, does the rollups and emits the final counts:

def reducer(sorted_lines):
    prev = {"d": "", "m": "", "y": "", "t": ""}
    count = {"d": 0, "m": 0, "y": 0}

    for track, date in sorted_lines:
        d, m, y = date[:10], date[:7], date[:4]

        if prev["t"]:
            if d != prev["d"]:
                emit(prev["t"], prev["d"], count["d"])
                count["d"] = 0
            if m != prev["m"]:
                emit(prev["t"], prev["m"], count["m"])
                count["m"] = 0
            if y != prev["y"]:
                emit(prev["t"], prev["y"], count["y"])
                count["y"] = 0
            if track != prev["t"]:
                count = {"d": 0, "m": 0, "y": 0}

        count["d"] += 1
        count["m"] += 1
        count["y"] += 1
        prev = {"d": d, "m": m, "y": y, "t": track}

    if prev["t"]:
        emit(prev["t"], prev["d"], count["d"])
        emit(prev["t"], prev["m"], count["m"])
        emit(prev["t"], prev["y"], count["y"])

Ideally this job creates data ready for bulk import into a fast store that can then serve web requests in a timely manner.

The partitioning problem

In a distributed MapReduce setup, a partitioner splits data across machines. For this to work correctly, all events for a given track within a full year must reach the same reducer. Otherwise you can’t calculate complete yearly rollups.

This is fine for per-track counters. But for global counters (the "*" key), it means all events of an entire year funnel through a single machine. That’s a hard bottleneck. Adding more machines to the cluster won’t help because one reducer still has to see everything.

The fix is a secondary job. The first pass computes partial rollups per partition. The second pass combines them. It adds complexity, so whether it’s worth it depends on the data volume and how fast the batch needs to complete.

Kappa architecture

The Kappa architecture, proposed by Jay Kreps, simplifies things by eliminating the batch layer entirely. Instead of maintaining two pipelines, all data flows through a single stream processing system. Historical reprocessing is handled by replaying the event log from the beginning. One codebase, one set of semantics. With modern stream processing frameworks like Apache Flink or Kafka Streams, this has become practical at significant scale.

But Kappa has real limitations. Reprocessing from raw events works well for recent data and low-cardinality dimensions. It breaks down when you need aggregations over long time ranges or across high-cardinality keys like per-user statistics across millions of users. Good luck deriving yearly rollups from raw events when the event log holds billions of entries. The throughput and parallelism requirements for a Kappa system can be substantial, and replaying months or years of history through a stream processor is expensive if it’s even feasible.

The right choice depends on the use case. Kappa works well for systems where the time window is bounded, the cardinality is manageable, and you can afford the infrastructure to keep up with replay demands. Lambda remains the better fit when you need deep historical aggregations over high-cardinality data, where a batch layer can churn through the volume efficiently in a way that stream replay cannot.

Choosing your approach

For most teams, a purpose-built time series database is the right answer. TimescaleDB if you want to stay in the PostgreSQL ecosystem, ClickHouse for heavy analytics, InfluxDB or VictoriaMetrics for metrics and monitoring.

Custom rollup pipelines still make sense when your data doesn’t fit neatly into the time series model, when you need full control over the aggregation logic, or when you’re operating at a scale where managed solutions become prohibitively expensive.

Either way, the fundamentals don’t change. Pre-aggregate. Roll up into time buckets. Trade resolution for speed and storage. These patterns were solid in 2011 and they’re solid now. The tools just got a lot better at hiding the machinery.

Random Lines from Large Files

2011-05-13T00:00:00Z

When working with big data, taking samples is the only road to quick answers. Unfortunately that already poses a bigger hurdle than it should. Ask around how to get random lines from a file and you’ll most likely hear something like this:

shuf -n 10 file.txt
cat file.txt | sort --random-sort | head -n 10

Note that passing the file as an argument instead of piping through cat would avoid the useless use of cat.

Why these don’t scale

The sort approach is the worst offender. It reads the entire file, shuffles all lines, and only then lets you pick from the top. For a 10 GB file that means sorting billions of lines just to grab 10.

shuf is smarter but still not fast. Looking at the source code, it reads the entire input sequentially. Even with -n 1 it processes every single line. Recent versions use reservoir sampling internally when combined with -n, which at least keeps memory usage bounded. But it still has to scan through the whole file from start to finish.

For a 10 GB file, that means waiting minutes for what should take milliseconds.

Seeking instead of reading

If the file is on disk (not stdin), we can do something much faster. Instead of reading every line, we pick random byte positions, seek directly to them, skip to the next newline, and read the following complete line.

import os
import random

def random_lines(filename, n=10):
    size = os.path.getsize(filename)
    positions = sorted(random.randint(0, size - 1) for _ in range(n))

    lines = []
    with open(filename, 'rb') as f:
        for pos in positions:
            f.seek(max(0, pos - 1))
            f.readline()          # skip partial line
            line = f.readline()   # read full line
            if line:
                lines.append(line.decode().strip())
    return lines

for line in random_lines("data.txt"):
    print(line)

Sorting the positions before seeking means we sweep through the file in one direction, which is friendlier to the disk cache. The whole thing runs in O(n) seeks regardless of file size.

The catch - the line length bias

This approach picks random byte positions, not random line positions. That means longer lines occupy more bytes and are more likely to be “hit” by a random position. If all lines are roughly the same length, the bias is negligible. If your file has a mix of 10-character and 10,000-character lines, the long lines will be heavily overrepresented.

For quick exploratory sampling this usually doesn’t matter. But if statistical correctness is important, you need a different approach.

Reservoir sampling

Reservoir sampling (Algorithm R, published by Jeffrey Vitter in 1985) is the theoretically correct way to get a uniform random sample from data of unknown size. It works in a single pass and uses O(k) memory, where k is the sample size.

The idea is simple. Keep the first k lines in a “reservoir”. For every subsequent line j, generate a random number between 1 and j. If it falls within k, replace that slot in the reservoir. When you reach the end of the file, the reservoir contains a perfectly uniform sample.

import random

def reservoir_sample(filename, k=10):
    reservoir = []
    with open(filename) as f:
        for i, line in enumerate(f):
            if i < k:
                reservoir.append(line.strip())
            else:
                j = random.randint(0, i)
                if j < k:
                    reservoir[j] = line.strip()
    return reservoir

The downside is obvious: it still reads every line in the file, just like shuf. The advantage is that it’s guaranteed to produce a uniform sample regardless of line lengths, and it works on streams where you can’t seek.

Which one to pick

It depends on what you need.

For quick sampling from large files on disk where line lengths are reasonably uniform, the seek-based approach is hard to beat. It runs in milliseconds on files of any size.

For statistically correct uniform sampling, or when reading from a pipe or stream, reservoir sampling is the right choice. It’s slower but mathematically sound.

For small files, just use shuf -n 10. It’s simple and good enough.

Building Debian Packages in Java

2007-01-19T00:00:00Z

Using native packaging formats to install Java applications leverages existing and well-tested infrastructure for software deployment. But one of the perks of Java has always been the cross-platform build. Using the standard native tools like dpkg-deb to create those packages breaks the promise of a platform independent build.

With the jdeb project there is a way to create Debian packages directly from your build - whether you use Maven, Gradle, or Ant.

Build Debian packages without native tools - on Linux, Windows, or macOS.

The project has been maintained since 2007 and is used by over 3,500 repositories on GitHub - including projects at Apache, Netflix, and Eclipse.

Why native packages in the container era?

With containers being everywhere, the question comes up whether native packages are still relevant. I think the answer is yes - for several reasons.

First, even inside containers you often want proper package management. A clean apt install with declared dependencies beats copying files around. System services, users, permissions, and config files are all things that packages handle well.

Second, not everything runs in containers. Bare-metal deployments, embedded systems, appliances, and developer tooling all benefit from native packages. And even when containers are the delivery vehicle, building the underlying image from proper packages keeps things reproducible.

You could of course use Docker to run the native packaging tools during the build. But that adds considerable complexity to what should be a straightforward Java build. The whole point of jdeb is to avoid that.

The control file

The only requirement is a control file that provides metadata about the package. It declares things like name, version, and most importantly dependencies.

Package: [[name]]
Version: [[version]]
Section: misc
Priority: optional
Architecture: all
Description: [[description]]
Maintainer: Your Name <you@example.com>
Depends: default-jre (>= 11)

jdeb also handles maintainer scripts (preinst, postinst, prerm, postrm) - just place them next to the control file in your control directory. These scripts are often the tricky part of packaging, and jdeb passes them through without getting in the way.

Maven

With Maven, jdeb attaches to the package phase. A build creates the jar, then builds the .deb and attaches it as a secondary artifact.

<plugin>
  <artifactId>jdeb</artifactId>
  <groupId>org.vafer</groupId>
  <version>1.14</version>
  <executions>
    <execution>
      <phase>package</phase>
      <goals>
        <goal>jdeb</goal>
      </goals>
      <configuration>
        <controlDir>${basedir}/src/deb/control</controlDir>
        <dataSet>
          <data>
            <src>${project.build.directory}/${project.build.finalName}.jar</src>
            <type>file</type>
            <mapper>
              <type>perm</type>
              <prefix>/usr/share/myapp/lib</prefix>
              <user>loader</user>
              <group>loader</group>
              <filemode>640</filemode>
            </mapper>
          </data>
        </dataSet>
      </configuration>
    </execution>
  </executions>
</plugin>

Running mvn package now produces both the jar and the .deb.

Gradle

jdeb itself does not ship a Gradle plugin, but Netflix’s nebula ospackage plugin uses jdeb under the hood and integrates beautifully with Gradle’s copy spec.

plugins {
  id 'com.netflix.nebula.ospackage' version '12.2.0'
}

ospackage {
  packageName = 'myapp'
  version = '1.0'
  release = '1'

  into '/usr/share/myapp/lib'

  from(jar.outputs.files) {
    fileMode = 0640
    user = 'loader'
    permissionGroup = 'loader'
  }
}

Running gradle buildDeb creates the package.

Ant

For Ant, jdeb provides a task directly. Specify the paths and provide the data that should get included.

<taskdef name="deb"
  classname="org.vafer.jdeb.ant.DebAntTask"
  classpath="jdeb.jar" />

<deb
  destfile="${build.dir}/${ant.project.name}.deb"
  control="${build.dir}/deb/control"
  verbose="true">
  <data type="file"
    src="${build.dir}/jar/${ant.project.name}-${version}.jar">
    <mapper type="perm" prefix="/usr/share/jdeb/lib"/>
  </data>
</deb>

GitHub Actions

Since jdeb runs as part of your normal build, creating packages in CI is straightforward. No special actions or Docker containers needed.

- name: Build with Maven
  run: mvn package

- name: Upload .deb artifact
  uses: actions/upload-artifact@v4
  with:
    name: debian-package
    path: target/*.deb

That’s it. The .deb gets built on whatever runner GitHub provides - Linux, macOS, or even Windows.

Alternatives

Naturally, jdeb is not the only option. Depending on your setup, these might be worth considering:

nfpm is a Go tool that builds .deb, .rpm, and .apk packages from a simple YAML config. No native tools required. I have contributed to nfpm and it is a great choice for all kinds of projects.
Native tools in CI. If your CI already runs on Debian, using dpkg-deb directly is always an option. You lose the cross-platform build, but for some workflows that is fine.
Nix takes a fundamentally different approach to packaging and deployment. A bigger commitment but very powerful.

Getting started

Check out the documentation and the examples on GitHub. Issues and discussions are welcome in the repository.

ditto vs zip vs tar on macOS

2006-05-02T00:00:00Z

Files on macOS carry more data than what you see in Finder. Beyond the file content there is metadata - and not all archiving tools preserve it. This has been a source of subtle bugs for decades.

Resource forks and extended attributes

Back in the classic Mac OS days, files had resource forks - a second data stream that stored structured metadata like icons, dialog layouts, or code. This was a feature of the HFS file system and pretty unique to the Mac.

When Apple moved to Mac OS X and eventually to APFS, resource forks didn’t really disappear. They evolved into extended attributes. Arbitrary key-value pairs attached to any file. APFS supports them natively, and macOS uses them everywhere.

Some concrete examples of extended attributes you encounter daily:

com.apple.quarantine is the flag that triggers “this file was downloaded from the internet” warnings
com.apple.metadata:kMDItemWhereFroms records which URL a file was downloaded from
com.apple.FinderInfo stores Finder tags and other Finder-specific metadata

You can inspect them with the xattr command line tool:

$ xattr -l sample.txt
com.apple.quarantine: 0081;12345678;Safari;ABC123
user.custom: my-custom-value

Other operating systems have similar concepts. Linux has extended attributes via getfattr and setfattr (used by SELinux for security labels, for instance). Windows has alternate data streams on NTFS. But macOS relies on them more heavily than most, which is why archiving them matters.

The archiving problem

Not all archive tools treat extended attributes the same way. Checking the behaviour of zip, ditto, and tar on macOS 15 (Sequoia) reveals some interesting differences.

zip

$ zip -r archive.zip testfolder/
$ unzip -l archive.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
       12  02-23-2026 23:42   testfolder/sample.txt
---------                     -------
       12                     1 file

Standard zip creates a clean, portable archive. But it drops extended attributes. What you get back after extracting is the file content - nothing more.

ditto

$ ditto -ck testfolder archive.zip
$ unzip -l archive.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
       12  02-23-2026 23:42   sample.txt
      264  02-23-2026 23:42   ._sample.txt
---------                     -------
      276                     2 files

ditto is what Finder uses when you right-click and choose “Compress”. It stores extended attributes in AppleDouble ._ files alongside the actual content. On a Mac, extracting this archive restores the full metadata. This has been the default behavior since Mac OS X 10.4.

The catch: when someone unpacks a ditto archive on Linux or Windows, those ._ files just show up as regular files. Annoying and unexpected for the non-Mac users.

tar (bsdtar)

$ tar -cf archive.tar testfolder/
$ tar -tf archive.tar
testfolder/
testfolder/sample.txt

The tar that ships with macOS is actually bsdtar, not GNU tar. And it preserves extended attributes by default - no special flags needed. The metadata is stored natively in the archive format.

# disable metadata explicitly if you don't want it
$ tar --no-mac-metadata -cf clean.tar testfolder/

This makes tar a solid choice when you’re archiving on a Mac for use on a Mac. The metadata travels with the archive transparently and doesn’t produce any ._ clutter.

Quick comparison

Tool	xattrs	Format	Cross-platform
`zip`	no	zip	clean everywhere
`ditto -ck`	yes	zip (with `._` files)	`._` files on non-Mac
`tar` (bsdtar)	yes	tar (native metadata)	only macOS bsdtar restores metadata
`tar --no-mac-metadata`	no	tar	clean everywhere

When to use what

Sharing with non-Mac users? Use zip or tar --no-mac-metadata. The ._ files from ditto archives tend to cause confusion on the receiving end.

Archiving for yourself or other Mac users? Use ditto -ck or just tar. Both preserve the full metadata. ditto gives you a zip that Finder can preview, tar is more natural for the command line.

Backing up files where metadata matters? tar is the first choice. It preserves everything without the ._ file overhead. Though for real backups you probably want Time Machine or tmutil, which handle extended attributes as well.

For the last 20 years the advice has been to use ditto over zip. That hasn’t changed. But it’s worth knowing that tar on macOS has quietly become a solid alternative that avoids the trade-off entirely.

TLDR: Don’t use zip if metadata matters. Use ditto for zip archives, tar for everything else.

The Duff Device

2006-01-10T00:00:00Z

Duff’s Device was discovered by Tom Duff in November 1983 while working at Lucasfilm. He needed to speed up a real-time animation program that was running 50% too slowly. The program had to copy data into the Programmed IO data register of an Evans & Sutherland Picture System II and the straightforward loop just wasn’t fast enough.

What he came up with is one of the most famous pieces of C code ever written. It’s also one of the most confusing.

The idea

Let’s start with a simple loop that copies some memory.

for (int i = 0; i < len; i++) {
    *output++ = *input++;
}

For longer loops, the overhead of checking the loop condition adds up. On every iteration the loop checks the counter against len, increments it, and jumps back. That’s a lot of bookkeeping for a single copy operation.

Loop unrolling reduces this overhead by doing multiple operations per iteration.

// assuming len is a multiple of 8
int n = len / 8;
do {
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
} while (--n > 0);

Now we only pay the loop overhead once for every 8 copies. But what if len isn’t a multiple of 8? You’d need to handle the remainder separately. This is where Duff’s Device gets creative.

The trick

int n = (len + 8 - 1) / 8;
switch (len % 8) {
    case 0: do { *output++ = *input++;
    case 7:      *output++ = *input++;
    case 6:      *output++ = *input++;
    case 5:      *output++ = *input++;
    case 4:      *output++ = *input++;
    case 3:      *output++ = *input++;
    case 2:      *output++ = *input++;
    case 1:      *output++ = *input++;
    } while (--n > 0);
}

It looks like someone tricked the parser but it really is valid C. It exploits two features of the language: the fall-through behavior of switch statements and the fact that switch and do-while can share a compound statement.

The switch acts as a jump table. On the first pass, it jumps to the right case label based on the remainder of len % 8, executing only the leftover copies. After that, the do-while loop takes over, copying 8 elements at a time until done.

As Duff himself wrote: “after 10 years of writing C there are still little corners that I haven’t explored fully.”

Making it readable

If we disentangle the two constructs, it becomes easier to follow.

int n = (len + 8 - 1) / 8;

// first: handle the remainder
switch (len % 8) {
    case 0: *output++ = *input++;
    case 7: *output++ = *input++;
    case 6: *output++ = *input++;
    case 5: *output++ = *input++;
    case 4: *output++ = *input++;
    case 3: *output++ = *input++;
    case 2: *output++ = *input++;
    case 1: *output++ = *input++;
}

// then: copy 8 at a time
while (--n > 0) {
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
    *output++ = *input++;
}

Same logic, no interleaved control structures. The switch handles the odd remainder, and the while loop does the rest in chunks of 8. In both versions, the loop overhead is reduced to 1/8 of the original.

I chose a memory copy operation only for demonstration purposes. In practice, the standard C library memcpy should always be preferred - it contains architecture-specific optimizations that will be faster than anything you write by hand.

What the compiler killed

Here’s the twist. In the late 1990s, the XFree86 project removed Duff’s Device from their codebase. The result? The server got faster and the binary got smaller.

The hand-unrolled code was actually preventing the compiler from applying its own optimizations. The interleaved switch and do-while created a control flow pattern that confused the optimizer. Simpler, straightforward loops gave the compiler room to do what it does best.

Modern compilers like GCC and Clang perform loop unrolling automatically. The -funroll-loops flag enables it, and with profile-guided optimization the compiler can make much better decisions about when and how much to unroll than a programmer writing code by hand. You can see exactly what the compiler does on Compiler Explorer.

But compilers don’t just unroll loops. They also apply SIMD vectorization - that’s processing multiple data elements in a single instruction. This is essentially what Duff’s Device was trying to achieve, but done at the hardware level. A single MOVDQA instruction can copy 16 bytes at once. That’s a fundamentally different level of optimization than reducing branch overhead.

Clever tricks compilers have replaced

The Duff Device is not alone. Many hand-crafted optimizations that were once essential have been absorbed by compilers.

Strength reduction replacing multiplication with bit shifts (i * 4 becomes i << 2). Compilers do this automatically.
Common subexpression elimination caching repeated calculations. The compiler tracks these across entire functions.
Constant folding computing 3 * 60 * 24 at compile time rather than runtime.
Dead code elimination removing branches that can never execute.
Register allocation the register keyword in C is now ignored by most compilers because they do a better job of register allocation than programmers.

What it comes down to: as compilers got smarter, hand-written optimizations went from helpful to harmful. Code that looks clever to a human can look opaque to an optimizer.

When to optimize by hand

There are still cases where manual optimization matters. Performance-critical inner loops in game engines, signal processing, and scientific computing sometimes benefit from hand-tuned SIMD intrinsics or carefully structured memory access patterns. But these are exceptions, not the rule.

For most code, the best optimization strategy is to write clear, straightforward loops and let the compiler do its job. Measure first, optimize second - and check the compiler output before rewriting anything by hand.

The Duff Device remains a wonderful piece of programming history. It teaches us about the dark corners of C, the evolution of compilers, and perhaps most importantly - that the clever hack you’re proud of today might be the thing slowing you down tomorrow.

Why Generated API Docs Suck

2005-03-23T00:00:00Z

Over the years I’ve been exposed to a lot of code bases across many languages. And over time it has become painfully clear that

Generated API docs just suck. Not as a concept, but what most teams make out of them.

The core problem of documentation is that it needs to be kept up to date. Having it separate from the code makes it really hard to maintain. So the idea of keeping docs close to the code isn’t bad at all. But somewhere along the way, teams confused “having documentation” with “having useful documentation.”

The boilerplate epidemic

Here’s a Java snippet that should look familiar to anyone who has worked on a larger code base.

/**
 * Construct an entry with only a name.
 *
 * @param name the entry name
 */
public FooEntry(String name) {
    this.name = name;
}

/**
 * Get this entry's name.
 *
 * @return This entry's name.
 */
public String getName() {
    return name;
}

What’s the benefit here? Every comment just restates what the code already says. It’s noise dressed up as documentation.

This isn’t a Java problem though. TypeScript has the same disease.

/**
 * Gets the user by ID.
 * @param id - The user ID.
 * @returns The user object.
 */
function getUser(id: string): User {
  return users.get(id);
}

Go isn’t immune either.

// GetUser returns the user for the given id.
func GetUser(id string) *User {
    return users[id]
}

And even Rust, despite having one of the best documentation cultures, still attracts this pattern.

/// Returns the name of the entry.
fn name(&self) -> &str {
    &self.name
}

In all four cases the type system already tells you everything the comment does. The signature is the documentation. Adding a comment that just rephrases it creates maintenance burden with zero informational value.

Why do we keep doing this?

For one, there’s a good chance developers didn’t write much of these docs in the first place. IDEs generate them. Linters enforce them. Checkstyle rules fail the build if they’re missing. And now AI assistants happily generate /** Gets the name. @return the name */ for every getter without anyone asking.

The incentive is to satisfy the tooling, not the reader. That’s backwards.

When you write documentation for the linter instead of the human, the human loses.

What good documentation looks like

The best documentation tells a story. It explains the why, outlines the important parts of the how, and gives notice of anything that might surprise you. Method signatures already cover the what.

Here’s a better version of that Java class.

/**
 * Represents a single entry in a Foo archive.
 *
 * Entries are created during archive construction and
 * written sequentially. The header format follows the
 * Foo specification v2.1, section 4.3.
 *
 * Instances are not thread-safe. If shared between threads,
 * external synchronization is required.
 */
public class FooEntry implements ArchiveEntry {

    private String name = "";

    public FooEntry(String name) {
        this.name = name;
    }

    public String getName() {
        return name;
    }

    public void writeEntryHeader(OutputStream out)
        throws IOException {
        out.write(1);
        out.write(2);
    }
}

The class-level comment does the heavy lifting. It explains what this thing is, where it fits in the bigger picture, and what you need to watch out for. The individual methods don’t need comments because their names and signatures are self-explanatory.

Focus on self-explanatory class, function and variable names. Often that’s even better than comments because the explanation is visible not only at declaration time but with every use. That’s the path to code as documentation. A lot of comments then become duplication and can be removed. Less visual clutter, more signal.

Languages that got it right

Some ecosystems have found clever ways to make documentation both useful and trustworthy.

Rust’s doc-tests let you write examples in documentation comments that the compiler actually runs as tests. If the example breaks, the build breaks. This means documentation stays in sync with the code by force, not by discipline.

Go’s testable examples follow a similar philosophy. You write functions named ExampleFoo in test files. They show up in the generated docs and run as part of the test suite. The output is verified against a comment.

Elixir’s doctests take this even further. Examples in @doc attributes are executed as part of the test suite. The documentation is the test. It’s the closest thing to documentation that can’t go stale.

Python’s doctest module has been doing this since 1999. Interactive examples in docstrings get extracted and verified. The idea is old but the principle is timeless: documentation that gets tested stays correct.

What all of these have in common is that documentation must be verified, not just present.

Documentation needs structure

The Divio documentation system makes a useful distinction between four types of documentation: tutorials, how-to guides, reference, and explanation. Most auto-generated docs only cover reference. That’s the least useful type when you’re trying to understand a new codebase.

What developers actually need when they encounter unfamiliar code is explanation and context. Why does this module exist? How does it fit into the architecture? What are the trade-offs?

Projects like Django and Stripe are known for documentation that reads like a guided tour, not a phone book. Elixir’s ecosystem has built this into its culture. Modules in Phoenix often start with pages of discussion about what something is, how to use it, and what to watch out for. The API reference comes after.

AI won’t save us

AI coding assistants can generate documentation at scale. But they tend to produce exactly the kind of boilerplate this article argues against. They’re very good at restating what code does. They’re not good at explaining why it exists or how it fits into the bigger picture.

That said, AI can help with reading and understanding undocumented code. It can summarize, explain patterns, and answer questions about a codebase in ways that static docs never could. But even AI needs to “understand” the code the same way a human does. If there’s no narrative structure, no explanation of intent, the AI is guessing just like the next developer would be.

Write for the reader

First and foremost, empathize with the person reading and trying to understand your code.

If you enforce documentation through linting, do it only for public APIs and focus on class-level or module-level docs. Nobody benefits from a mandatory @param name the name on every setter.

Write documentation at the highest useful level. Explain the module, not the getter. Tell the story of why this code exists. If you want to enforce documentation quality, verify your examples compile and run. Rust, Go, Elixir and Python have shown that this works.

The best documentation isn’t the most complete. It’s the most useful.