tiny.tripwire

Overview · API · Code relationships · Verification · Audit

Overview

The excerpt below, from BufferPublication.stageAndPublish, shows one operation that takes a private copy of a payload, holds it, and then publishes it, with an error path open between the copy and the publish. The worked example near the end of this page gives the operation in full.

1pub fn stageAndPublish(self: *BufferPublication, payload: []const u8) !void {2    if (self.published != null) return error.AlreadyPublished;3    const staged = try self.allocator.dupe(u8, payload);4    errdefer self.allocator.free(staged);56    try Points.check(.before_publish);78    self.published = staged;9}

When a unit test runs that operation with Points unconfigured and the allocation succeeding, check returns void, so the errdefer cleanup never runs and the error recovery goes unverified. This unverified error recovery is the obstacle the package answers. A place in the code under test where a call to check is written, named by one tag of an enum, is a check site, and the package lets a test choose the error a check site delivers while the test runs. One evaluation of a configured site is a visit. The caller decides when a site delivers, by setting how many visits pass first. Once the test body has run, the caller confirms that the site delivered its error.

Named check sites and the threshold counter

A codebase declares an enum whose tags name its check sites, the failpoint enum, and passes it to module, which republishes it as FailPoint.

1const Points = tripwire.module(enum {2    before_publish,3    after_flush,4}, anyerror);

An operation can reach the same site more than once before it reaches the state the test is after, for example while it parses earlier records or runs earlier loop iterations. How many visits a site lets through before it delivers its error (min) is the threshold. errorAfter(point, err, min) lets min visits pass and delivers err on visit min + 1 and on every visit after. Inside check, a site with no configuration returns void at once and touches no state. A configured site adds 1 to its visit count, tripwire.reached, inside usize range. While the visit count stands at or below the threshold, check returns void. Once the visit count passes the threshold, the site trips: it sets tripped to true, the flag end reads, and hands back err. A tripped site keeps returning err on every later call, as long as the visit count stays inside usize range. errorAlways(point, err) calls errorAfter(point, err, 0), which leaves the threshold at zero and trips the site on its first visit.

Execution trace

The record one configured site carries, holding the error to deliver, its threshold, its visit count, and whether the error has gone out (the type Tripwire), is a failpoint. A call to end has three outcomes:

• With no failpoint configured, it succeeds at once.
• Where every configured failpoint has tripped == true, it returns void.
• Where any configured failpoint has tripped == false, that site is an untripped site, and the call returns error.UntrippedError.

Configuration lifecycle and verification

The one map of site names to failpoints that the generated type owns (TripwireMap, a std.EnumMap) is the state map. Calling errorAfter or errorAlways on a site that is already configured replaces its record in the state map, putting reached back to 0 and tripped back to false. reset() empties the state map.

The argument to end choosing whether it empties the state map after it verifies, one of .reset or .retain, is the reset mode:

• end(.reset) confirms that every configured site tripped and then empties the state map through reset(), and it empties the map on the run that returns error.UntrippedError as well.
• end(.retain) runs the same check and leaves the state map and the counters as they were. Because the accumulated tripped flags stay set under .retain, a later end(.retain) with no check in between succeeds again at once.

Calling end proves that configured errors were delivered: each active site took more visits than its threshold and gave back the error it holds. Whether the caller handled the error, released its resources, or held its domain invariants is left to the test's own assertions, such as checking that private staged state stayed unpublished and that memory came back.

Compilation and runtime mechanics

A compilation whose root is a test, the case builtin.is_test reports, is a test build. The compile-time flag that turns the checks on is enabled, and it equals builtin.is_test, so in a test build a check performs its state map lookup, raises its counter, and delivers the error. Any other build has builtin.is_test == false, which fixes enabled at false during compilation and makes callingConvention() yield .@"inline". Each check begins with the guard if (comptime !enabled) return;, so the compiler folds the call into its caller and drops the rest of the body as an inlined no-op. An expression written as an argument to check is evaluated by the caller at runtime before the call, unless a general compiler optimization removes it. Whether the compilation is a test (builtin.is_test) is a separate question from the optimization mode (-O Debug, -O ReleaseFast). errorAfter, reset, and end stay in the failpoint module's namespace and remain callable where enabled is false, and there they work on a state map that check leaves unread. The counter uses platform usize addition, written tripwire.reached += 1, which detects overflow in a safety-checked build. The persistent failure behavior holds as long as reached stays inside usize range.

Namespace and type system contracts

A call to module(P, E) returns a struct type, the failpoint module, holding the site names, the resolved error set, the state map, and the functions that configure, evaluate, and verify the sites.

1var tripwires: TripwireMap = .{};

That map belongs to the generated type, so every caller referencing the same (P, E) specialization shares the one state map, and no struct instance holds it. The state map is a std.EnumMap held under no mutex, no atomic primitive, and no thread-local storage. Calling check, errorAfter, reset, or end from more than one thread at once races on that map, and the result is undefined behavior. A test with concurrency serializes its failpoint operations or keeps them on one thread.

The second argument to module, written E, is the error specification, and the error set read out of it at compile time, the type check returns (Error), is the resolved error set. The error specification E accepts four forms:

1. An error set type, written anyerror or error{OutOfMemory, DiskFull}.
2. An error union type, written anyerror!void or error{DiskFull}!u32.
3. A function type returning an error union, written @TypeOf(myFunc).
4. A function value, written myFunc, whose declared return type carries the error union.

An individual error value such as error.DiskFull is refused as E, because the error set case hands the value back and Error has to be a type. Where enabled is false, checkConstrained(point, ConstrainedError) becomes an inlined no-op that returns void. In a test build, once its body is instantiated, checkConstrained executes return tripwire.err;, and tripwire.err has type Error, so returning it requires Error to coerce to ConstrainedError. Zig coerces an error set only into a set that holds it, so in a test build ConstrainedError has to be Error or a superset of Error. A ConstrainedError smaller than Error, say error{DiskFull} where the failpoint module was built with anyerror, fails compilation.

Worked caller example

The example stages private memory, checks a failpoint before publishing, cleans up with errdefer, and asserts its postconditions apart from end.

1const std = @import("std");2const tripwire = @import("tripwire");34pub const Points = tripwire.module(enum {5    before_publish,6}, anyerror);78pub const BufferPublication = struct {9    allocator: std.mem.Allocator,10    published: ?[]u8 = null,1112    pub fn stageAndPublish(self: *@This(), payload: []const u8) !void {13        if (self.published != null) return error.AlreadyPublished;14        const staged = try self.allocator.dupe(u8, payload);15        errdefer self.allocator.free(staged);1617        try Points.check(.before_publish);1819        self.published = staged;20    }2122    pub fn deinit(self: *@This()) void {23        if (self.published) |buffer| {24            self.allocator.free(buffer);25            self.published = null;26        }27    }28};2930test "staged buffer allocation and cleanup under fault injection" {31    const testing = std.testing;3233    Points.reset();34    defer Points.reset();35    Points.errorAlways(.before_publish, error.DiskFull);3637    var publication: BufferPublication = .{ .allocator = testing.allocator };38    defer publication.deinit();3940    // 1. Confirm that the injected error propagates:41    try testing.expectError(error.DiskFull, publication.stageAndPublish("payload"));4243    // 2. Independently verify domain postconditions:44    //    Private staged memory was freed by errdefer (verified by testing.allocator),45    //    and published state remains unmutated:46    try testing.expect(publication.published == null);4748    // 3. Verify probe execution:49    try Points.end(.reset);50}

Boundaries and limitations

The package tests explicit, localized failure points:

• Placement: errors arrive at the program locations where a check call is written, and the package synthesizes none anywhere else.
• Thread scheduling: the package models no race condition, no lock contention, and no thread preemption interleaving.
• Heap exhaustion: the package intercepts no memory allocation globally, and it reaches the sites written into the code. An allocator built for the job, std.testing.FailingAllocator among them, refuses an allocation on a count the caller sweeps.
• Hardware-level corruption: the package models no bit rot, no truncated write, and no kernel panic, and one of those reaches a test only where the code turns it into an application-level error code delivered at a site.

Context and related work

Work on injecting faults reaches the machine at different layers:

• Fault injection methodologies: Arlat et al. (1990) defined the FARM framework, which characterizes a fault injection experiment by a fault set F, an activation set A, readouts R, and derived measures M, and applied it experimentally at the physical pin level. Hsueh, Tsai, and Iyer (1997) surveyed hardware-based fault injection alongside software-implemented fault injection (SWIFI), which spans low-level memory and register corruption through application-level error injection. Carreira, Madeira, and Silva (1998) described Xception, which performs software-implemented fault injection through processor debugging features. Set beside them, a failpoint module names the simulated fault set, caller workloads supply the activation, and end reads out whether the activation reached the designated site and tripped it.
• SQLite anomaly testing: the SQLite Project exercises how its code handles running out of memory and how it handles input and output errors, by intercepting allocations with sqlite3_config(SQLITE_CONFIG_MALLOC, ...) and by routing filesystem calls behind custom Virtual File System (VFS) shims (SQLite Project documentation, Sections 3.1 and 3.2). Inside its test loops, an instrumented interface tallies the calls it sees, and the harness sets it either to fail once and then resume normally or to keep failing after that first failure. Those harnesses raise the failure counter ($N=1,2,3\dots$) one step at a time until the operation runs clean. This package places named check sites in the code by hand, and errorAfter models failure that persists past a threshold.
• Named failpoints in kernels and distributed systems:
• FreeBSD fail(9) provides the kernel KFAIL_POINT_CODE macro and a sysctl interface for dynamic failure injection.
• The Linux kernel provides a fault injection engine whose configurable knobs include failslab and fail_page_alloc.
• Storage and consensus engines in distributed systems reach chosen error paths through named failpoints, TiKV's fail-rs and CoreOS's gofail among them.
• Error handling as a source of outages: across an empirical study of 198 user-reported failures in five distributed data-intensive systems (Cassandra, HBase, HDFS, Hadoop MapReduce, and Redis), of which catastrophic failures formed a subset, Yuan et al. (OSDI '14) reported that a critical failure frequently begins in code that mishandles a non-fatal error while recovering, which is the argument for exercising error-handling branches systematically.

References:

• J. Arlat, M. Aguera, L. Amat, Y. Crouzet, J.-C. Fabre, J.-C. Laprie, E. Martins, and D. Powell. "Fault injection for dependability validation: a methodology and some applications." IEEE Transactions on Software Engineering, 16(2):166–182, 1990. DOI: 10.1109/32.44380, Author PDF.
• M.-C. Hsueh, T. K. Tsai, and R. K. Iyer. "Fault injection techniques and tools." IEEE Computer, 30(4):75–82, 1997. DOI: 10.1109/2.585157.
• J. Carreira, H. Madeira, and J. G. Silva. "Xception: a technique for the experimental evaluation of dependability in modern computers." IEEE Transactions on Software Engineering, 24(2):125–136, 1998. DOI: 10.1109/32.666826.
• SQLite Project. "How SQLite Is Tested" (Section 3: Anomaly Testing). https://www.sqlite.org/testing.html.
• Ding Yuan, Yu Luo, Xin Zhuang, Guilherme Renna Rodrigues, Xu Zhao, Yongle Zhang, Pranay U. Jain, and Michael Stumm. "Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems." In Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI '14), pages 249–265, 2014. https://www.usenix.org/conference/osdi14/technical-sessions/presentation/yuan.

Definitions

Actions

Public operations.

• module: Creates a failpoint module at compile time for a given failpoint enum and error specification.

Namespaces

Public namespaces.

• failpoint: The implementation of the package, holding the generic constructor and the runtime logic behind every check site.

Code relationships

Direct static dependencies extracted from parsed source by semantic graph analysis.

Uses: None
Used by: None

Verification

No verification records are cataloged for this module in this build.

Audit