lib/tripwire/src/root.zig

daab053ee43316e1809a84551d573ddd1e5bf3d2

  1 //! The excerpt below, from `BufferPublication.stageAndPublish`, shows one
  2 //! operation that takes a private copy of a payload, holds it, and then
  3 //! publishes it, with an error path open between the copy and the publish. The
  4 //! worked example near the end of this page gives the operation in full.
  5 //!
  6 //! ```zig
  7 //! pub fn stageAndPublish(self: *BufferPublication, payload: []const u8) !void {
  8 //!     if (self.published != null) return error.AlreadyPublished;
  9 //!     const staged = try self.allocator.dupe(u8, payload);
 10 //!     errdefer self.allocator.free(staged);
 11 //!
 12 //!     try Points.check(.before_publish);
 13 //!
 14 //!     self.published = staged;
 15 //! }
 16 //! ```
 17 //!
 18 //! When a unit test runs that operation with `Points` unconfigured and the
 19 //! allocation succeeding, `check` returns `void`, so the `errdefer` cleanup
 20 //! never runs and the error recovery goes unverified. This unverified error
 21 //! recovery is the obstacle the package answers. A place in the code under test
 22 //! where a call to `check` is written, named by one tag of an enum, is a *check
 23 //! site*, and the package lets a test choose the error a check site delivers
 24 //! while the test runs. One evaluation of a configured site is a *visit*. The
 25 //! caller decides when a site delivers, by setting how many visits pass first.
 26 //! Once the test body has run, the caller confirms that the site delivered its
 27 //! error.
 28 //!
 29 //! ## Named check sites and the threshold counter
 30 //!
 31 //! A codebase declares an enum whose tags name its check sites, the *failpoint
 32 //! enum*, and passes it to `module`, which republishes it as `FailPoint`.
 33 //!
 34 //! ```zig
 35 //! const Points = tripwire.module(enum {
 36 //!     before_publish,
 37 //!     after_flush,
 38 //! }, anyerror);
 39 //! ```
 40 //!
 41 //! An operation can reach the same site more than once before it reaches the
 42 //! state the test is after, for example while it parses earlier records or runs
 43 //! earlier loop iterations. How many visits a site lets through before it
 44 //! delivers its error (`min`) is the *threshold*. `errorAfter(point, err, min)`
 45 //! lets `min` visits pass and delivers `err` on visit `min + 1` and on every
 46 //! visit after. Inside `check`, a site with no configuration returns `void` at
 47 //! once and touches no state. A configured site adds 1 to its visit count,
 48 //! `tripwire.reached`, inside `usize` range. While the visit count stands at or
 49 //! below the threshold, `check` returns `void`. Once the visit count passes the
 50 //! threshold, the site *trips*: it sets `tripped` to `true`, the flag `end`
 51 //! reads, and hands back `err`. A tripped site keeps returning `err` on every
 52 //! later call, as long as the visit count stays inside `usize` range.
 53 //! `errorAlways(point, err)` calls `errorAfter(point, err, 0)`, which leaves
 54 //! the threshold at zero and trips the site on its first visit.
 55 //!
 56 //! ## Execution trace
 57 //!
 58 //! | Configuration | Call | Result |
 59 //! | :--- | :--- | :--- |
 60 //! | unconfigured | 1 | `void` |
 61 //! | `errorAlways(pt, err)` | 1 | `err` |
 62 //! | `errorAlways(pt, err)` | 2 | `err` |
 63 //! | `errorAfter(pt, err, 2)` | 1 | `void` |
 64 //! | `errorAfter(pt, err, 2)` | 2 | `void` |
 65 //! | `errorAfter(pt, err, 2)` | 3 | `err` |
 66 //! | `errorAfter(pt, err, 2)` | 4 | `err` |
 67 //!
 68 //! The record one configured site carries, holding the error to deliver, its
 69 //! threshold, its visit count, and whether the error has gone out (the type
 70 //! `Tripwire`), is a *failpoint*. A call to `end` has three outcomes:
 71 //! - With no failpoint configured, it succeeds at once.
 72 //! - Where every configured failpoint has `tripped == true`, it returns `void`.
 73 //! - Where any configured failpoint has `tripped == false`, that site is an
 74 //!   *untripped site*, and the call returns `error.UntrippedError`.
 75 //!
 76 //! ## Configuration lifecycle and verification
 77 //!
 78 //! The one map of site names to failpoints that the generated type owns
 79 //! (`TripwireMap`, a `std.EnumMap`) is the *state map*. Calling `errorAfter` or
 80 //! `errorAlways` on a site that is already configured replaces its record in
 81 //! the state map, putting `reached` back to 0 and `tripped` back to `false`.
 82 //! `reset()` empties the state map.
 83 //!
 84 //! The argument to `end` choosing whether it empties the state map after it
 85 //! verifies, one of `.reset` or `.retain`, is the *reset mode*:
 86 //! - `end(.reset)` confirms that every configured site tripped and then empties
 87 //!   the state map through `reset()`, and it empties the map on the run that
 88 //!   returns `error.UntrippedError` as well.
 89 //! - `end(.retain)` runs the same check and leaves the state map and the
 90 //!   counters as they were. Because the accumulated `tripped` flags stay set
 91 //!   under `.retain`, a later `end(.retain)` with no check in between succeeds
 92 //!   again at once.
 93 //!
 94 //! Calling `end` proves that configured errors were delivered: each active site
 95 //! took more visits than its threshold and gave back the error it holds.
 96 //! Whether the caller handled the error, released its resources, or held its
 97 //! domain invariants is left to the test's own assertions, such as checking
 98 //! that private staged state stayed unpublished and that memory came back.
 99 //!
100 //! ## Compilation and runtime mechanics
101 //!
102 //! A compilation whose root is a test, the case `builtin.is_test` reports, is a
103 //! *test build*. The compile-time flag that turns the checks on is *enabled*,
104 //! and it equals `builtin.is_test`, so in a test build a check performs its
105 //! state map lookup, raises its counter, and delivers the error. Any other
106 //! build has `builtin.is_test == false`, which fixes `enabled` at `false`
107 //! during compilation and makes `callingConvention()` yield `.@"inline"`. Each
108 //! check begins with the guard `if (comptime !enabled) return;`, so the
109 //! compiler folds the call into its caller and drops the rest of the body as an
110 //! *inlined no-op*. An expression written as an argument to `check` is
111 //! evaluated by the caller at runtime before the call, unless a general
112 //! compiler optimization removes it. Whether the compilation is a test
113 //! (`builtin.is_test`) is a separate question from the optimization mode (`-O
114 //! Debug`, `-O ReleaseFast`). `errorAfter`, `reset`, and `end` stay in the
115 //! failpoint module's namespace and remain callable where `enabled` is false,
116 //! and there they work on a state map that `check` leaves unread. The counter
117 //! uses platform `usize` addition, written `tripwire.reached += 1`, which
118 //! detects overflow in a safety-checked build. The persistent failure behavior
119 //! holds as long as `reached` stays inside `usize` range.
120 //!
121 //! ## Namespace and type system contracts
122 //!
123 //! A call to `module(P, E)` returns a struct type, the *failpoint module*,
124 //! holding the site names, the resolved error set, the state map, and the
125 //! functions that configure, evaluate, and verify the sites.
126 //!
127 //! ```zig
128 //! var tripwires: TripwireMap = .{};
129 //! ```
130 //!
131 //! That map belongs to the generated type, so every caller referencing the same
132 //! `(P, E)` specialization shares the one state map, and no struct instance
133 //! holds it. The state map is a `std.EnumMap` held under no mutex, no atomic
134 //! primitive, and no thread-local storage. Calling `check`, `errorAfter`,
135 //! `reset`, or `end` from more than one thread at once races on that map, and
136 //! the result is undefined behavior. A test with concurrency serializes its
137 //! failpoint operations or keeps them on one thread.
138 //!
139 //! The second argument to `module`, written `E`, is the *error specification*,
140 //! and the error set read out of it at compile time, the type `check` returns
141 //! (`Error`), is the *resolved error set*. The error specification `E` accepts
142 //! four forms:
143 //! 1. An error set type, written `anyerror` or `error{OutOfMemory, DiskFull}`.
144 //! 2. An error union type, written `anyerror!void` or `error{DiskFull}!u32`.
145 //! 3. A function type returning an error union, written `@TypeOf(myFunc)`.
146 //! 4. A function value, written `myFunc`, whose declared return type carries
147 //!    the error union.
148 //!
149 //! An individual error value such as `error.DiskFull` is refused as `E`,
150 //! because the error set case hands the value back and `Error` has to be a
151 //! type. Where `enabled` is false, `checkConstrained(point, ConstrainedError)`
152 //! becomes an inlined no-op that returns `void`. In a test build, once its body
153 //! is instantiated, `checkConstrained` executes `return tripwire.err;`, and
154 //! `tripwire.err` has type `Error`, so returning it requires `Error` to coerce
155 //! to `ConstrainedError`. Zig coerces an error set only into a set that holds
156 //! it, so in a test build `ConstrainedError` has to be `Error` or a superset of
157 //! `Error`. A `ConstrainedError` smaller than `Error`, say `error{DiskFull}`
158 //! where the failpoint module was built with `anyerror`, fails compilation.
159 //!
160 //! ## Worked caller example
161 //!
162 //! The example stages private memory, checks a failpoint before publishing,
163 //! cleans up with `errdefer`, and asserts its postconditions apart from `end`.
164 //!
165 //! ```zig
166 //! const std = @import("std");
167 //! const tripwire = @import("tripwire");
168 //!
169 //! pub const Points = tripwire.module(enum {
170 //!     before_publish,
171 //! }, anyerror);
172 //!
173 //! pub const BufferPublication = struct {
174 //!     allocator: std.mem.Allocator,
175 //!     published: ?[]u8 = null,
176 //!
177 //!     pub fn stageAndPublish(self: *@This(), payload: []const u8) !void {
178 //!         if (self.published != null) return error.AlreadyPublished;
179 //!         const staged = try self.allocator.dupe(u8, payload);
180 //!         errdefer self.allocator.free(staged);
181 //!
182 //!         try Points.check(.before_publish);
183 //!
184 //!         self.published = staged;
185 //!     }
186 //!
187 //!     pub fn deinit(self: *@This()) void {
188 //!         if (self.published) |buffer| {
189 //!             self.allocator.free(buffer);
190 //!             self.published = null;
191 //!         }
192 //!     }
193 //! };
194 //!
195 //! test "staged buffer allocation and cleanup under fault injection" {
196 //!     const testing = std.testing;
197 //!
198 //!     Points.reset();
199 //!     defer Points.reset();
200 //!     Points.errorAlways(.before_publish, error.DiskFull);
201 //!
202 //!     var publication: BufferPublication = .{ .allocator = testing.allocator };
203 //!     defer publication.deinit();
204 //!
205 //!     // 1. Confirm that the injected error propagates:
206 //!     try testing.expectError(error.DiskFull, publication.stageAndPublish("payload"));
207 //!
208 //!     // 2. Independently verify domain postconditions:
209 //!     //    Private staged memory was freed by errdefer (verified by testing.allocator),
210 //!     //    and published state remains unmutated:
211 //!     try testing.expect(publication.published == null);
212 //!
213 //!     // 3. Verify probe execution:
214 //!     try Points.end(.reset);
215 //! }
216 //! ```
217 //!
218 //! ## Boundaries and limitations
219 //!
220 //! The package tests explicit, localized failure points:
221 //! - Placement: errors arrive at the program locations where a `check` call is
222 //!   written, and the package synthesizes none anywhere else.
223 //! - Thread scheduling: the package models no race condition, no lock
224 //!   contention, and no thread preemption interleaving.
225 //! - Heap exhaustion: the package intercepts no memory allocation globally, and
226 //!   it reaches the sites written into the code. An allocator built for the
227 //!   job, `std.testing.FailingAllocator` among them, refuses an allocation on a
228 //!   count the caller sweeps.
229 //! - Hardware-level corruption: the package models no bit rot, no truncated
230 //!   write, and no kernel panic, and one of those reaches a test only where the
231 //!   code turns it into an application-level error code delivered at a site.
232 //!
233 //! ## Context and related work
234 //!
235 //! Work on injecting faults reaches the machine at different layers:
236 //! - Fault injection methodologies: Arlat et al. (1990) defined the FARM
237 //!   framework, which characterizes a fault injection experiment by a fault set
238 //!   F, an activation set A, readouts R, and derived measures M, and applied it
239 //!   experimentally at the physical pin level. Hsueh, Tsai, and Iyer (1997)
240 //!   surveyed hardware-based fault injection alongside software-implemented
241 //!   fault injection (SWIFI), which spans low-level memory and register
242 //!   corruption through application-level error injection. Carreira, Madeira,
243 //!   and Silva (1998) described Xception, which performs software-implemented
244 //!   fault injection through processor debugging features. Set beside them, a
245 //!   failpoint module names the simulated fault set, caller workloads supply
246 //!   the activation, and `end` reads out whether the activation reached the
247 //!   designated site and tripped it.
248 //! - SQLite anomaly testing: the SQLite Project exercises how its code handles
249 //!   running out of memory and how it handles input and output errors, by
250 //!   intercepting allocations with `sqlite3_config(SQLITE_CONFIG_MALLOC, ...)`
251 //!   and by routing filesystem calls behind custom Virtual File System (VFS)
252 //!   shims ([SQLite Project
253 //!   documentation](https://www.sqlite.org/testing.html), Sections 3.1 and
254 //!   3.2). Inside its test loops, an instrumented interface tallies the calls
255 //!   it sees, and the harness sets it either to fail once and then resume
256 //!   normally or to keep failing after that first failure. Those harnesses
257 //!   raise the failure counter ($N = 1, 2, 3 \dots$) one step at a time until
258 //!   the operation runs clean. This package places named check sites in the
259 //!   code by hand, and `errorAfter` models failure that persists past a
260 //!   threshold.
261 //! - Named failpoints in kernels and distributed systems:
262 //!   - FreeBSD
263 //!     [fail(9)](https://man.freebsd.org/cgi/man.cgi?query=fail&sektion=9)
264 //!     provides the kernel `KFAIL_POINT_CODE` macro and a sysctl interface for
265 //!     dynamic failure injection.
266 //!   - The Linux kernel provides a
267 //!     [fault injection engine](https://docs.kernel.org/fault-injection/fault-injection.html)
268 //!     whose configurable knobs include `failslab` and `fail_page_alloc`.
269 //!   - Storage and consensus engines in distributed systems reach chosen error
270 //!     paths through named failpoints, TiKV's
271 //!     [fail-rs](https://github.com/tikv/fail-rs) and CoreOS's
272 //!     [gofail](https://github.com/etcd-io/gofail) among them.
273 //! - Error handling as a source of outages: across an empirical study of 198
274 //!   user-reported failures in five distributed data-intensive systems
275 //!   (Cassandra, HBase, HDFS, Hadoop MapReduce, and Redis), of which
276 //!   catastrophic failures formed a subset,
277 //!   [Yuan et al. (OSDI '14)](https://www.usenix.org/conference/osdi14/technical-sessions/presentation/yuan)
278 //!   reported that a critical failure frequently begins in code that mishandles
279 //!   a non-fatal error while recovering, which is the argument for exercising
280 //!   error-handling branches systematically.
281 //!
282 //! References:
283 //!
284 //! - J. Arlat, M. Aguera, L. Amat, Y. Crouzet, J.-C. Fabre, J.-C. Laprie,
285 //!   E. Martins, and D. Powell. "Fault injection for dependability validation:
286 //!   a methodology and some applications." *IEEE Transactions on Software
287 //!   Engineering*, 16(2):166–182, 1990.
288 //!   [DOI: 10.1109/32.44380](https://doi.org/10.1109/32.44380),
289 //!   [Author PDF](https://homepages.laas.fr/arlat/documents/89124/89124.pdf).
290 //! - M.-C. Hsueh, T. K. Tsai, and R. K. Iyer. "Fault injection techniques and
291 //!   tools." *IEEE Computer*, 30(4):75–82, 1997.
292 //!   [DOI: 10.1109/2.585157](https://doi.org/10.1109/2.585157).
293 //! - J. Carreira, H. Madeira, and J. G. Silva. "Xception: a technique for the
294 //!   experimental evaluation of dependability in modern computers." *IEEE
295 //!   Transactions on Software Engineering*, 24(2):125–136, 1998.
296 //!   [DOI: 10.1109/32.666826](https://doi.org/10.1109/32.666826).
297 //! - SQLite Project. "How SQLite Is Tested" (Section 3: Anomaly Testing).
298 //!   [https://www.sqlite.org/testing.html](https://www.sqlite.org/testing.html).
299 //! - Ding Yuan, Yu Luo, Xin Zhuang, Guilherme Renna Rodrigues, Xu Zhao,
300 //!   Yongle Zhang, Pranay U. Jain, and Michael Stumm. "Simple Testing Can Prevent
301 //!   Most Critical Failures: An Analysis of Production Failures in Distributed
302 //!   Data-Intensive Systems." In *Proceedings of the 11th USENIX Symposium on
303 //!   Operating Systems Design and Implementation (OSDI '14)*, pages 249–265, 2014.
304 //!   [https://www.usenix.org/conference/osdi14/technical-sessions/presentation/yuan](https://www.usenix.org/conference/osdi14/technical-sessions/presentation/yuan).
305 
306 /// The implementation of the package, holding the generic constructor and the
307 /// runtime logic behind every check site.
308 pub const failpoint = @import("failpoint.zig");
309 
310 /// Creates a failpoint module at compile time for a given failpoint enum and
311 /// error specification.
312 pub const module = failpoint.module;