tiny.machine.explore.distributed
Defined in explore.
A tool that searches for bugs earns trust by finding a real one in a running system and replaying it on demand.
API (86)
Actions
Public operations.
Conjunct.bothConjunct.eitherConjunct.mergeCursors.advanceCursors.initDriver.momentDriver.runDriver.runnerDriver.startProperty.idQueue.cancelQueue.findQueue.messagesQueue.pushQueue.takeReplayer.applyReplayer.beginReplayer.initReplayer.pathReplayer.runnerReplayer.sealSemantic.idState.assertValidState.backingState.initState.suspectingState.votesWitness.forkWitness.runbuildsinitialRootoriginseedProvenanceselectedProfilestep
Types and contracts
Public types and contracts.
CapsuleConfigConjunctCursors: The input, schedule, and fault sources that offer the alternatives for one kind of choice, one step at a time (generators), each positioned where a settled run the search keeps (retained branch) left it.DeliveryDiagnosticDiffDriver: One search of the three-node commit protocol under test (workload) that covers every branch of the fixed six levels of choices every workload search walks.ErrorEventsExplorationHistoryMessageMessageKindNodePersistencePropertyProvenanceQueue: Stores sent messages awaiting delivery for one position of the three-node protocol: at most 12, twice the depth of the choice tree.ReducerReductionReplayReplayer: Runs one given list of alternatives taken at a step (decisions) against the three-node commit protocol under test (workload) from its initial state, in order and with no search, and records an append-only record of the run (causal history), whichseallater seals.ReportSearchSemanticSignatureState: One position of the three-node protocol.VariantWitness: One run of the whole defect gate finds the bug planted on purpose in the defective variant: the coordinator counts every acknowledgement, late or repeated, so it can announce a commit that fewer than two nodes back.
Values and defaults
Public values and defaults.
Driver.budgetDriver.capacityQueue.capacitybudgetcapsule_capacitycoordinatordeclarationsdefectivedelayed_onlydepthfreshness_rulehistory_capacitykindsnode_countpersistent_onlyquorumquorum_rulerepairedsearch_capacitysuspicion_stepstree
Source
Source: lib/machine/src/explore/distributed/driver.zig:52
/// The input, schedule, and fault sources that offer the alternatives for one/// kind of choice, one step at a time (*generators*), each positioned where a/// settled run the search keeps (*retained branch*) left it. The driver keeps/// one set for every retained branch, so it can offer each child the choices/// that follow its own parent's history. `init` builds the three over 8 steps/// and virtual ticks 1 through 4096 from the selected machine configuration and/// the 32-byte value the generators draw from (*seed*), and returns/// `GeneratorUnavailable` when a generator cannot start. `advance` moves the/// generator that matches the kind of one alternative taken at a step/// (*decision*) past that decision. `advance` returns `StaleChoiceSite` when/// the decision's step, alternative count, or chosen value differs from what/// the generator offers, and `ChoiceOutOfRange` for an alternative past the/// offered count. A topology decision returns `DecisionStreamMismatch`, because/// the fixed six levels of choices every workload search walks have no topology/// level, and a generator with no next step returns `GeneratorUnavailable`. A/// child's set starts as a copy of its parent's.pub const Cursors = struct { input: Input, schedule: Schedule, faults: Fault, pub fn init(selected: profile.Profile, seed: explore.Seed) types.Error!Cursors { return .{ .input = try ready(Input, try Input.init(selected, seed, temporal)), .schedule = try ready(Schedule, try Schedule.init(selected, seed, .{ .temporal = temporal, .nodes = types.node_count, })), .faults = try ready(Fault, try Fault.init(selected, seed, .{ .temporal = temporal, .kinds = &types.kinds, })), }; } pub fn advance( self: *Cursors, decision: explore.SearchDecision, ) types.Error!void { switch (decision.site.stream) { .input => { const site = try requireSite(self.input.next()); try match(site, decision); const chosen = try self.input.choose(site, decision.alternative); if (!std.meta.eql(chosen, decision.choice.input)) { return error.StaleChoiceSite; } }, .schedule => { const site = try requireSite(self.schedule.next()); try match(site, decision); const chosen = try self.schedule.choose(site, decision.alternative); if (!std.meta.eql(chosen, decision.choice.schedule)) { return error.StaleChoiceSite; } }, .fault => { const site = try requireSite(self.faults.next()); try match(site, decision); const chosen = try self.faults.choose(site, decision.alternative); if (!std.meta.eql(chosen, decision.choice.fault)) { return error.StaleChoiceSite; } }, .topology => return error.DecisionStreamMismatch, } }};Source: lib/machine/src/explore/distributed/driver.zig:128
/// One search of the three-node commit protocol under test (*workload*) that/// covers every branch of the fixed six levels of choices every workload search/// walks. The witness gate calls `start` and `run` once for each workload/// variant and compares the summaries. The driver holds the search plus one/// protocol state and one cursor set for each of 3584 settled runs the search/// keeps (*retained branches*), all inside the value, so a caller allocates it/// in memory it owns. `start` resets the driver for one variant, the selected/// machine configuration, and the 32-byte value the generators draw from,/// stores the initial protocol state as branch 0, and queues its first choices./// `run` steps the search until the queue of waiting choices empties or a/// budget runs out, then returns a summary. A branch that violates a rule that/// fails at the first event that breaks it (*safety rule*) is kept and counted,/// and the driver grows no children from it. A step that prunes a choice or/// ends incomplete returns `SearchPruned` or `SearchIncomplete`, so `run`/// returns a summary only when no branch was skipped. The summary gives the/// executions, the retained branches, the pruned count, the failures, the/// reason the search stopped, and how many branches reached the long hang/// alone, the late delivery alone, both together, or a queued acknowledgement./// `runner` gives the search a callback that restarts from a copy of the/// parent's saved state and cursors, applies one alternative taken at a step,/// and reports the position in a world's history (*moment*) reached, or a/// failure when a safety rule is violated. `moment` computes the moment of one/// retained branch from its saved state.pub const Driver = struct { config: types.Config, origin: world.Root, search: Search, states: [types.search_capacity.retained]state_owner.State, cursors: [types.search_capacity.retained]Cursors, pending: state_owner.State, pending_cursors: Cursors, parent: explore.SearchBranchId, leaf: bool, failures: u32, pub const capacity: explore.SearchCapacity = types.search_capacity; pub const budget: explore.SearchBudget = types.budget; pub fn start( self: *Driver, config: types.Config, selected: profile.Profile, seed: explore.Seed, ) types.Error!void { const contract = try profile.contractFingerprint(selected); self.config = config; self.origin = canon.origin(contract); self.failures = 0; self.parent = 0; self.leaf = false; self.states[0] = state_owner.State.init(canon.initialRoot(contract)); self.cursors[0] = try Cursors.init(selected, seed); self.search = try Search.init(try self.moment(0), budget); std.debug.assert(self.search.branches().len == 1); try self.expand(0); std.debug.assert(self.search.frontier().len > 0); } pub fn run(self: *Driver) types.Error!types.Exploration { for (0..budget.executions + 1) |_| { const frontier = self.search.frontier(); std.debug.assert(frontier.len <= capacity.frontier); if (frontier.len == 0) return self.summary(.frontier_empty); self.parent = frontier[0].parent; std.debug.assert(self.parent < self.search.branches().len); self.leaf = (try self.search.branch(self.parent)).depth + 1 == types.depth; switch (self.search.step(self.runner())) { .completed => |value| try self.settle(value.branch, true), .failed => |value| try self.settle(value.branch, false), .exhausted => |value| return self.summary(value.reason), .pruned => return error.SearchPruned, .incomplete => return error.SearchIncomplete, } } return error.SearchIncomplete; } pub fn runner(self: *Driver) explore.SearchRunner { return .{ .context = self, .execute = execute }; } pub fn moment(self: *const Driver, id: explore.SearchBranchId) types.Error!world.Moment { std.debug.assert(id < self.states.len); return world.prepareMoment(self.origin, self.states[id].fabric); } fn settle(self: *Driver, id: explore.SearchBranchId, expand_next: bool) types.Error!void { std.debug.assert(id > 0); std.debug.assert(id < self.states.len); std.debug.assert(id < capacity.retained); self.states[id] = self.pending; self.cursors[id] = self.pending_cursors; if (!expand_next) { self.failures += 1; return; } try self.expand(id); } fn expand(self: *Driver, id: explore.SearchBranchId) types.Error!void { const branch = try self.search.branch(id); if (branch.depth >= types.depth) return; std.debug.assert(branch.depth < types.tree.len); const cursors = &self.cursors[id]; const expansion = switch (types.tree[branch.depth]) { .input => try self.search.expandInput( id, try requireSite(cursors.input.next()), ), .schedule => try self.search.expandSchedule( id, try requireSite(cursors.schedule.next()), ), .fault => try self.search.expandFault( id, try requireSite(cursors.faults.next()), ), .topology => return error.DecisionStreamMismatch, }; switch (expansion) { .expanded => {}, .exhausted => return error.SearchIncomplete, .pruned => return error.SearchPruned, } } fn summary( self: *const Driver, reason: explore.SearchExhaustedReason, ) types.Exploration { const progress = self.search.progress(); std.debug.assert(progress.retained == self.search.branches().len); std.debug.assert(progress.executions <= budget.executions); var result: types.Exploration = .{ .config = self.config, .executions = progress.executions, .retained = progress.retained, .pruned = progress.pruned, .settled_failures = 0, .reported_failures = self.failures, .exhausted = reason, .hang_only_branches = 0, .delay_only_branches = 0, .conjunction_branches = 0, .queued_acknowledgement_branches = 0, }; for (self.search.branches(), 0..) |branch, index| { const conjunct = self.states[index].conjunct; if (conjunct.queued_acknowledgement) { result.queued_acknowledgement_branches += 1; } if (conjunct.both()) { result.conjunction_branches += 1; } else if (conjunct.persistent_hang) { result.hang_only_branches += 1; } else if (conjunct.delayed_delivery) { result.delay_only_branches += 1; } switch (branch.settlement) { .failed => { result.settled_failures += 1; std.debug.assert(conjunct.both()); }, .origin, .completed => {}, } } std.debug.assert(result.settled_failures == self.failures); std.debug.assert(result.conjunction_branches >= result.settled_failures); return result; }};Source: lib/machine/src/explore/distributed/queue.zig:14
/// Stores sent messages awaiting delivery for one position of the three-node/// protocol: at most 12, twice the depth of the choice tree. The three-node/// commit protocol under test (*workload*) pushes each message it sends here/// and takes it out at delivery. `push` appends one message and returns/// `MessageCapacityExceeded` when the queue is full. Messages stay in the order/// sent: `find` returns the index of the earliest message for one destination,/// and `take` removes it and keeps the others in order. `cancel` drops every/// message of one kind from one sender. The workload calls `cancel` to withdraw/// the coordinator's old proposals when the coordinator, node 0, opens a new/// round, a new attempt to get a value accepted.pub const Queue = struct { storage: [types.message_capacity]types.Message = undefined, count: u8 = 0, pub const capacity: u8 = types.message_capacity; pub fn push(self: *Queue, message: types.Message) types.Error!void { self.assertValid(); std.debug.assert(message.sender < types.node_count); std.debug.assert(message.destination < types.node_count); if (self.count == capacity) return error.MessageCapacityExceeded; self.storage[self.count] = message; self.count += 1; self.assertValid(); } pub fn find(self: *const Queue, destination: u8) ?u8 { self.assertValid(); std.debug.assert(destination < types.node_count); for (self.messages(), 0..) |message, index| { if (message.destination == destination) return @intCast(index); } return null; } pub fn take(self: *Queue, index: u8) types.Message { self.assertValid(); std.debug.assert(index < self.count); const message = self.storage[index]; const remaining = self.count - index - 1; std.mem.copyForwards( types.Message, self.storage[index .. index + remaining], self.storage[index + 1 .. self.count], ); self.count -= 1; self.assertValid(); return message; } pub fn cancel(self: *Queue, sender: u8, kind: types.MessageKind) void { self.assertValid(); std.debug.assert(sender < types.node_count); var kept: u8 = 0; var index: u8 = 0; while (index < self.count) : (index += 1) { const message = self.storage[index]; if (message.sender == sender and message.kind == kind) continue; std.debug.assert(kept <= index); self.storage[kept] = message; kept += 1; } self.count = kept; self.assertValid(); } pub fn messages(self: *const Queue) []const types.Message { std.debug.assert(self.count <= capacity); return self.storage[0..self.count]; } fn assertValid(self: *const Queue) void { std.debug.assert(self.count <= capacity); std.debug.assert(self.count <= self.storage.len); }};Source: lib/machine/src/explore/distributed/replay.zig:39
/// Runs one given list of alternatives taken at a step (*decisions*) against/// the three-node commit protocol under test (*workload*) from its initial/// state, in order and with no search, and records an append-only record of the/// run (*causal history*), which `seal` later seals. The witness gate uses one/// to replay a record of one failing run that replays it exactly (*capsule*)/// and to replay a chosen path, and reads the sealed causal history afterward./// Each recorded step of the history (*frame*) holds the step's events recorded/// for checking rules and the results of checking both rules that fail at the/// first event that breaks them (*safety rules*), and the rule declarations go/// into the first frame alone. The history holds at most 8 frames and 64/// records, and the replayer keeps it and its byte encoding inside the value./// `init` binds one workload variant and one profile and starts an empty/// history, and `begin` restarts from the initial state. `apply` runs one/// decision and reports the position reached in a world's history (*moment*),/// or reports a failure when a safety rule is violated. `seal` finishes the/// history as exhausted and returns its identity. `path` restarts, applies 1 to/// 8 decisions, and seals. `runner` adapts the replayer for capsule replay: its/// preparation accepts only the replayer's own profile, the identity of the/// build each of the workload's three nodes ran, and outside results from host/// services or effects, and it restarts at the capsule's start moment. The/// runner's step requires the parent to be the current moment, reports a full/// history as an incomplete trace, and reports any other error as a rejected/// replay.pub const Replayer = struct { config: types.Config, selected: profile.Profile, origin: world.Root, state: state_owner.State, history: History, wire: History.Wire, sequence: u16, frames: u16, failure: ?explore.PropertyEvaluation, preparations: u32, pub fn init( self: *Replayer, config: types.Config, selected: profile.Profile, ) types.Error!void { self.config = config; self.selected = selected; self.preparations = 0; try self.begin(); } pub fn begin(self: *Replayer) types.Error!void { const contract = try profile.contractFingerprint(self.selected); self.origin = canon.origin(contract); self.state = state_owner.State.init(canon.initialRoot(contract)); self.history = try History.init(try self.moment()); self.sequence = 0; self.frames = 0; self.failure = null; std.debug.assert(self.history.traceState() == .open); } pub fn apply( self: *Replayer, decision: explore.SearchDecision, ) types.Error!explore.SearchExecution { var events: types.Events = .{}; const verdicts = try workload.evaluate( self.config, &self.state, decision, &events, false, ); const reached = try self.moment(); std.debug.assert(reached.fabric.entry_frontier <= self.frames + 1); try self.record(decision, reached, &events, verdicts); std.debug.assert(self.frames <= types.history_capacity.frames); if (property.violation(verdicts)) |failure| { self.failure = failure; return .{ .failed = .{ .root = reached, .evaluation = failure } }; } return .{ .completed = reached }; } pub fn seal(self: *Replayer) types.Error!explore.query.Identity { std.debug.assert(self.frames > 0); return self.history.finish(.exhausted, &self.wire); } pub fn runner(self: *Replayer) explore.CapsuleRunner { return .{ .context = self, .prepare = prepare, .step = .{ .context = self, .execute = execute }, }; } pub fn path( self: *Replayer, decisions: []const explore.SearchDecision, ) types.Error!explore.query.Identity { std.debug.assert(decisions.len > 0); std.debug.assert(decisions.len <= types.history_capacity.frames); try self.begin(); for (decisions) |decision| _ = try self.apply(decision); std.debug.assert(self.frames == decisions.len); return self.seal(); } fn moment(self: *const Replayer) types.Error!world.Moment { return world.prepareMoment(self.origin, self.state.fabric); } fn record( self: *Replayer, decision: explore.SearchDecision, reached: world.Moment, events: *const types.Events, verdicts: property.Verdicts, ) types.Error!void { std.debug.assert(self.frames < types.history_capacity.frames); const frame = try self.history.appendFrame(decision, reached); std.debug.assert(frame == self.frames); self.frames += 1; for (events.events()) |event| { const declaration = std.meta.activeTag(event.value) == .property_declaration; if (declaration and frame != 0) continue; try self.append(frame, event.virtual_time_tick, event.value); } for (verdicts) |verdict| { try self.append( frame, self.state.now, .{ .property_evaluation = verdict }, ); } } fn append( self: *Replayer, frame: u16, tick: u64, value: explore.EventValue, ) types.Error!void { try self.history.appendSemantic(frame, .{ .sequence = self.sequence, .virtual_time_tick = tick, .value = value, }); std.debug.assert(self.sequence < std.math.maxInt(u16)); self.sequence += 1; }};Source: lib/machine/src/explore/distributed/state.zig:24
/// One position of the three-node protocol. The state records the ledger root/// reached so far, the pending messages, what each node accepted, the/// coordinator's round, proposal, and acknowledgement count, the commit, which/// node is hung and for how long, the simulated clock, and which of the two/// facts the planted defect needs have happened. The driver and the replayer/// hand one to each workload step and read it back afterward. Only a workload/// step changes the state. Each step that advances folds the previous ledger/// digest, the decision, the protocol fields, and the pending messages into the/// next digest, so two equal histories reach equal *ledger roots* (the digest/// and counters naming the position of the ledger, the ordered record of every/// admitted input and fault decision). `init` starts on a given ledger root at/// tick 1, with no messages, no *round* (one attempt by the coordinator, node/// 0, to get a value accepted), and no commit. `backing` counts the nodes that/// accepted the current round's proposal. `votes` counts the coordinator's own/// vote plus every *acknowledgement* (a node's reply that it accepted a round's/// proposal) it counted in the current round. `suspecting` holds while a round/// is open and a node other than the coordinator has stayed hung for at least/// two fault steps.pub const State = struct { fabric: fabric.Root, messages: queue.Queue = .{}, nodes: [types.node_count]types.Node = @splat(types.Node{}), round: u16 = 0, open: bool = false, proposal: u64 = 0, acks: u8 = 0, ack_mask: u8 = 0, committed: bool = false, committed_round: u16 = 0, committed_value: u64 = 0, hang_streak: u8 = 0, hung: ?u8 = null, delivery_blocked: bool = false, turn_node: ?u8 = null, request: u64 = 0, now: u64 = types.start_tick, conjunct: types.Conjunct = .{}, pub fn init(root: fabric.Root) State { const value: State = .{ .fabric = root }; value.assertValid(); return value; } pub fn backing(self: *const State) u8 { self.assertValid(); var count: u8 = 0; for (self.nodes) |node| { if (!node.accepted) continue; if (node.accepted_round != self.round) continue; if (node.accepted_value != self.proposal) continue; count += 1; } std.debug.assert(count <= types.node_count); return count; } pub fn votes(self: *const State) u8 { self.assertValid(); return 1 + self.acks; } pub fn suspecting(self: *const State) bool { self.assertValid(); if (!self.open) return false; if (self.hang_streak < types.suspicion_steps) return false; const hung = self.hung orelse return false; return hung != types.coordinator; } pub fn assertValid(self: *const State) void { std.debug.assert(self.acks <= types.depth); std.debug.assert(self.committed_round <= self.round); std.debug.assert(self.now >= types.start_tick); std.debug.assert(self.now <= types.end_tick); if (self.hung) |node| std.debug.assert(node < types.node_count); if (self.turn_node) |node| std.debug.assert(node < types.node_count); for (self.nodes) |node| std.debug.assert(node.accepted_round <= self.round); }};Source: lib/machine/src/explore/distributed/types.zig:46
pub const Config = struct { variant: Variant, persistence: Persistence = .retained, delivery: Delivery = .queued,};Source: lib/machine/src/explore/distributed/types.zig:157
pub const Conjunct = packed struct(u8) { persistent_hang: bool = false, delayed_delivery: bool = false, queued_acknowledgement: bool = false, padding: u5 = 0, pub fn both(self: Conjunct) bool { return self.persistent_hang and self.delayed_delivery; } pub fn either(self: Conjunct) bool { return self.persistent_hang or self.delayed_delivery; } pub fn merge(self: Conjunct, other: Conjunct) Conjunct { return .{ .persistent_hang = self.persistent_hang or other.persistent_hang, .delayed_delivery = self.delayed_delivery or other.delayed_delivery, .queued_acknowledgement = self.queued_acknowledgement or other.queued_acknowledgement, }; }};Source: lib/machine/src/explore/distributed/types.zig:41
pub const Delivery = enum(u8) { queued = 1, immediate = 2,};Source: lib/machine/src/explore/distributed/types.zig:90
pub const Diagnostic = enum(u32) { turn_skipped = 1, inbox_empty = 2,};Source: lib/machine/src/explore/distributed/types.zig:196
pub const Exploration = struct { config: Config, executions: u32, retained: u16, pruned: u64, settled_failures: u16, reported_failures: u32, exhausted: explore.SearchExhaustedReason, hang_only_branches: u32, delay_only_branches: u32, conjunction_branches: u32, queued_acknowledgement_branches: u32,};Source: lib/machine/src/explore/distributed/types.zig:100
pub const Message = struct { kind: MessageKind, round: u16, value: u64, sender: u8, destination: u8,};Source: lib/machine/src/explore/distributed/types.zig:95
pub const MessageKind = enum(u8) { propose = 1, ack = 2,};Source: lib/machine/src/explore/distributed/types.zig:108
pub const Node = struct { accepted: bool = false, accepted_round: u16 = 0, accepted_value: u64 = 0, hung: bool = false,};Source: lib/machine/src/explore/distributed/types.zig:36
pub const Persistence = enum(u8) { retained = 1, transient = 2,};Source: lib/machine/src/explore/distributed/types.zig:63
pub const Property = enum(explore.SemanticId) { commit_quorum = 0x51_0001, accept_freshness = 0x51_0002, pub fn id(self: Property) explore.SemanticId { return @backingInt(self); }};Source: lib/machine/src/explore/distributed/types.zig:181
pub const Provenance = struct { profile: profile.ProfileFingerprint, contract: profile.ContractFingerprint, determinism: profile.DeterminismIdentity, dialect: u16, seed: explore.Seed,};Source: lib/machine/src/explore/distributed/types.zig:210
pub const Reduction = struct { original_frames: u16, reduced_frames: u16, statistics: explore.CapsuleStatistics, minimal: bool,};Source: lib/machine/src/explore/distributed/types.zig:217
pub const Replay = struct { identity: explore.CapsuleIdentity, first: explore.query.Identity, second: explore.query.Identity, evaluation: explore.PropertyEvaluation, conjunct: Conjunct,};Source: lib/machine/src/explore/distributed/types.zig:225
pub const Report = struct { provenance: Provenance, defective: Exploration, repaired: Exploration, persistent_only: Exploration, delayed_only: Exploration, reduction: Reduction, replay: Replay, branch: Signature, repeat_matched: bool,};Source: lib/machine/src/explore/distributed/types.zig:72
pub const Semantic = enum(explore.SemanticId) { round_opened = 0x64_0001, proposal_accepted = 0x64_0002, acknowledged = 0x64_0003, delayed_delivery = 0x64_0004, persistent_hang = 0x64_0005, node_suspected = 0x64_0006, commit_announced = 0x64_0007, unbacked_commit = 0x64_0008, stale_accept = 0x64_0009, delivery_deferred = 0x64_000a, host_operation = 0x64_000b, pub fn id(self: Semantic) explore.SemanticId { return @backingInt(self); }};Source: lib/machine/src/explore/distributed/types.zig:189
pub const Signature = struct { frame: u16, reason: explore.query.DivergenceReason, roots: explore.query.RootDifference, reference: explore.query.RefWire,};Source: lib/machine/src/explore/distributed/types.zig:31
pub const Variant = enum(u8) { defective = 1, repaired = 2,};Source: lib/machine/src/explore/distributed/witness.zig:40
/// One run of the whole defect gate finds the bug planted on purpose in the/// defective variant: the coordinator counts every acknowledgement, late or/// repeated, so it can announce a commit that fewer than two nodes back. The/// gate shrinks that record of one failing run that replays it exactly/// (*capsule*), replays it twice, branches from it, and checks both/// single-fault variants and the repaired variant. A test calls `run` once and/// checks the report it returns. The witness holds the search driver, three/// replayers, two reducers, a history comparison, both capsules, and the/// failing and sibling paths inside the value, so a caller allocates it in/// memory it owns. The call to `run` returns `FailureMissing` when the/// defective search settles no failure. The first failure becomes a capsule,/// which the gate shrinks and then tests by deleting in turn each remaining/// step of the capsule (*frame*), to report whether any single frame can still/// go. The shortened capsule replays on two replayers, and `run` returns/// `ReplayRejected` unless both reach the same history identity, their/// histories never diverge, and both reach the same result of checking one/// rule. Branching replays the failing path and a sibling that takes another/// alternative at the last decision, compares the two histories, and identifies/// the first step where two histories differ, with the reason (*first/// divergence*). The gate encodes that first divergence as the history's/// identity, a target, and an index, and `run` returns `SiblingMissing` when no/// sibling or divergence exists. A failure in either single-fault search or in/// the repaired search returns `RepairRegressed`. The report gives the record/// of where the seed came from, the four search summaries, the shrinking/// result, the replay identities, and the signature of the first divergence./// The `fork` function reads the paths that `run` records, so a caller calls it/// after `run`.pub const Witness = struct { driver: driver_owner.Driver, primary: replay.Replayer, repeat: replay.Replayer, alternate: replay.Replayer, reducer: replay.Reducer, probe: replay.Reducer, diff: replay.Diff, capsule: replay.Capsule, reduced: replay.Capsule, failing: Path, sibling: Path, length: u16, pub fn run(self: *Witness) types.Error!types.Report { const selected = seed_owner.selected(); const derived = try seed_owner.provenance(selected); try self.driver.start(types.defective, selected, derived.seed); const defective = try self.driver.run(); if (defective.settled_failures == 0) return error.FailureMissing; std.debug.assert(defective.conjunction_branches >= defective.settled_failures); try self.primary.init(types.defective, selected); try self.repeat.init(types.defective, selected); try self.alternate.init(types.defective, selected); try self.capture(selected); const reduction = try self.minimize(); const replayed = try self.confirm(); try self.collect(); const branch = try self.fork(); const persistent_only = try self.ablate(types.persistent_only, derived); const delayed_only = try self.ablate(types.delayed_only, derived); try self.driver.start(types.repaired, selected, derived.seed); const repaired = try self.driver.run(); if (repaired.settled_failures != 0) return error.RepairRegressed; std.debug.assert(repaired.exhausted == defective.exhausted); return .{ .provenance = derived, .defective = defective, .repaired = repaired, .persistent_only = persistent_only, .delayed_only = delayed_only, .reduction = reduction, .replay = replayed, .branch = branch, .repeat_matched = true, }; } fn ablate( self: *Witness, config: types.Config, derived: types.Provenance, ) types.Error!types.Exploration { try self.driver.start(config, seed_owner.selected(), derived.seed); const result = try self.driver.run(); if (result.settled_failures != 0) return error.RepairRegressed; std.debug.assert(result.exhausted == .frontier_empty); std.debug.assert(result.conjunction_branches == 0); return result; } fn capture(self: *Witness, selected: profile.Profile) types.Error!void { const failure = try self.firstFailure(); const builds = canon.builds(); self.capsule = switch (try replay.Capsule.capture( &self.driver.search, failure, selected, &builds, self.primary.runner(), )) { .published => |value| value, .rejected => return error.CaptureRejected, }; } fn minimize(self: *Witness) types.Error!types.Reduction { self.reducer = switch (try replay.Reducer.init( self.capsule, self.primary.runner(), )) { .ready => |value| value, .rejected => return error.ReplayRejected, }; const result = try self.reducer.reduce(self.primary.runner()); self.reduced = result.capsule; std.debug.assert(self.reduced.frame_count > 0); std.debug.assert(self.reduced.frame_count <= self.capsule.frame_count); return .{ .original_frames = self.capsule.frame_count, .reduced_frames = self.reduced.frame_count, .statistics = result.statistics, .minimal = try self.minimal(), }; } fn minimal(self: *Witness) types.Error!bool { if (self.reduced.frame_count == 1) return true; var frame: u16 = 0; while (frame < self.reduced.frame_count) : (frame += 1) { self.probe = switch (try replay.Reducer.init( self.reduced, self.repeat.runner(), )) { .ready => |value| value, .rejected => return error.ReplayRejected, }; switch (try self.probe.tryRemove(frame, self.repeat.runner())) { .accepted => return false, .rejected => {}, } } return true; } fn confirm(self: *Witness) types.Error!types.Replay { const first = try require(try self.reduced.replay(self.primary.runner())); const first_identity = try self.primary.seal(); const second = try require(try self.reduced.replay(self.repeat.runner())); const second_identity = try self.repeat.seal(); if (!std.meta.eql(first, second)) return error.ReplayRejected; if (!std.meta.eql(first_identity, second_identity)) return error.ReplayRejected; const compared = try self.diff.run( &self.primary.history, &self.repeat.history, types.diff_work, ); if (compared.divergence != null) return error.ReplayRejected; return .{ .identity = try self.reduced.identity(), .first = first_identity, .second = second_identity, .evaluation = first, .conjunct = self.primary.state.conjunct, }; } fn collect(self: *Witness) types.Error!void { const failure = try self.firstFailure(); const decisions = try self.driver.search.history(failure.branch, &self.failing); self.length = @intCast(decisions.len); std.debug.assert(self.length > 0); std.debug.assert(self.length <= types.depth); @memcpy(self.sibling[0..self.length], decisions); self.sibling[self.length - 1] = try self.siblingDecision(failure); } pub fn fork(self: *Witness) types.Error!types.Signature { _ = try self.primary.path(self.failing[0..self.length]); _ = try self.alternate.path(self.sibling[0..self.length]); const compared = try self.diff.run( &self.primary.history, &self.alternate.history, types.diff_work, ); const divergence = compared.divergence orelse return error.SiblingMissing; const left = divergence.left orelse return error.SiblingMissing; std.debug.assert(left.choice.index < self.length); var reference: explore.query.RefWire = undefined; explore.query.encodeRef(left.choice, &reference); std.debug.assert(std.meta.eql( try explore.query.decodeRef(&reference), left.choice, )); return .{ .frame = left.choice.index, .reason = divergence.reason, .roots = divergence.roots, .reference = reference, }; } fn siblingDecision( self: *const Witness, failure: Failed, ) types.Error!explore.SearchDecision { for (self.driver.search.branches()) |candidate| { const parent = candidate.parent orelse continue; const decision = candidate.decision orelse continue; if (parent != failure.parent) continue; if (!std.meta.eql(decision.site, failure.decision.site)) continue; if (decision.alternative == failure.decision.alternative) continue; return decision; } return error.SiblingMissing; } fn firstFailure(self: *const Witness) types.Error!Failed { for (self.driver.search.branches(), 0..) |candidate, index| { const evaluation = switch (candidate.settlement) { .failed => |value| value, .origin, .completed => continue, }; std.debug.assert(index > 0); return .{ .branch = @intCast(index), .parent = candidate.parent.?, .decision = candidate.decision.?, .evaluation = evaluation, }; } return error.FailureMissing; }};Source: lib/machine/src/explore/distributed/canon.zig:35
pub fn builds() [types.node_count]explore.CapsuleBuildIdentity { var result: [types.node_count]explore.CapsuleBuildIdentity = undefined; for (&result, 0..) |*entry, index| { var node: fabric.NodeId = .{ .bytes = @splat(0) }; node.bytes[node.bytes.len - 1] = @intCast(index + 1); var hasher = Sha256.init(.{}); hasher.update(build_domain); hasher.update(&node.bytes); var digest: [Sha256.digest_length]u8 = undefined; hasher.final(&digest); std.debug.assert(node.bytes[node.bytes.len - 1] != 0); entry.* = .{ .node = node, .execution = .{ .digest = digest } }; } for (result[1..], result[0 .. result.len - 1]) |next, previous| { std.debug.assert(std.mem.lessThan(u8, &previous.node.bytes, &next.node.bytes)); } return result;}Source: lib/machine/src/explore/distributed/canon.zig:14
pub fn initialRoot(contract: profile.ContractFingerprint) fabric.Root { return .{ .digest = seedDigest(state_domain), .dialect = .ordered_effect_fabric_v3, .machine_contract = contract, .entry_frontier = 0, .admission_frontier = 0, .fault_frontier = 0, };}Source: lib/machine/src/explore/distributed/canon.zig:25
pub fn origin(contract: profile.ContractFingerprint) world.Root { return .{ .digest = seedDigest(world_domain), .dialect = .connected_world_v1, .machine_contract = contract, .fabric = initialRoot(contract), .node_count = types.node_count, };}Source: lib/machine/src/explore/distributed/driver.zig:11
pub const Search = explore.Search(types.search_capacity);Source: lib/machine/src/explore/distributed/property.zig:15
pub const declarations = [_]explore.PropertyDeclaration{ .{ .id = quorum_rule.property, .kind = .safety, .bound = null }, .{ .id = freshness_rule.property, .kind = .safety, .bound = null },};Source: lib/machine/src/explore/distributed/property.zig:10
pub const freshness_rule: explore.Safety = .{ .property = types.Property.accept_freshness.id(), .forbidden = .{ .observation = types.Semantic.stale_accept.id() },};Source: lib/machine/src/explore/distributed/property.zig:5
pub const quorum_rule: explore.Safety = .{ .property = types.Property.commit_quorum.id(), .forbidden = .{ .observation = types.Semantic.unbacked_commit.id() },};Source: lib/machine/src/explore/distributed/replay.zig:11
pub const Capsule = explore.Capsule(types.capsule_capacity);Source: lib/machine/src/explore/distributed/replay.zig:14
Source: lib/machine/src/explore/distributed/replay.zig:13
Source: lib/machine/src/explore/distributed/replay.zig:12
pub const Reducer = explore.CapsuleReducer(types.capsule_capacity);Source: lib/machine/src/explore/distributed/seed.zig:15
pub fn provenance(value: profile.Profile) profile.Error!Provenance { try profile.validate(value); const full = try profile.profileFingerprint(value); const contract = try profile.contractFingerprint(value); const determinism = value.contract.determinism; var hasher = Sha256.init(.{}); hasher.update(domain); hasher.update(&full.digest); hasher.update(&contract.digest); hasher.update(&determinism.digest); var encoded: [6]u8 = undefined; std.mem.writeInt(u16, encoded[0..2], @backingInt(determinism.schema), .little); std.mem.writeInt(u16, encoded[2..4], determinism.entry_count, .little); std.mem.writeInt(u16, encoded[4..6], types.dialect_version, .little); hasher.update(&encoded); var bytes: [Sha256.digest_length]u8 = undefined; hasher.final(&bytes); std.debug.assert(!allZero(&bytes)); return .{ .profile = full, .contract = contract, .determinism = determinism, .dialect = types.dialect_version, .seed = .{ .bytes = bytes }, };}Source: lib/machine/src/explore/distributed/seed.zig:11
Source: lib/machine/src/explore/distributed/types.zig:152
pub const Error = explore.GeneratorError || explore.CapsuleError || explore.query.Error || OwnerError;Source: lib/machine/src/explore/distributed/types.zig:115
pub const Events = explore.EventSequence(.{ .events = event_capacity });Source: lib/machine/src/explore/distributed/types.zig:122
pub const budget: explore.SearchBudget = .{ .executions = 4096, .depth = depth };Source: lib/machine/src/explore/distributed/types.zig:124
pub const capsule_capacity: explore.CapsuleCapacity = .{ .frames = 8, .builds = node_count,};Source: lib/machine/src/explore/distributed/types.zig:8
pub const coordinator: u8 = 0;Source: lib/machine/src/explore/distributed/types.zig:52
pub const defective: Config = .{ .variant = .defective };Source: lib/machine/src/explore/distributed/types.zig:58
pub const delayed_only: Config = .{ .variant = .defective, .persistence = .transient,};Source: lib/machine/src/explore/distributed/types.zig:129
Source: lib/machine/src/explore/distributed/types.zig:13
Source: lib/machine/src/explore/distributed/types.zig:7
pub const node_count: u8 = 3;Source: lib/machine/src/explore/distributed/types.zig:54
pub const persistent_only: Config = .{ .variant = .defective, .delivery = .immediate,};Source: lib/machine/src/explore/distributed/types.zig:9
pub const quorum: u8 = 2;Source: lib/machine/src/explore/distributed/types.zig:53
pub const repaired: Config = .{ .variant = .repaired };Source: lib/machine/src/explore/distributed/types.zig:117
pub const search_capacity: explore.SearchCapacity = .{ .frontier = 3072, .retained = 3584,};Source: lib/machine/src/explore/distributed/types.zig:10
pub const suspicion_steps: u8 = 2;Source: lib/machine/src/explore/distributed/types.zig:15
Source: lib/machine/src/explore/distributed/workload.zig:27
pub fn step( config: types.Config, value: *State, decision: explore.SearchDecision, events: *types.Events,) types.Error!void { std.debug.assert(decision.valid()); value.assertValid(); const previous = value.now; const entry_frontier = value.fabric.entry_frontier; value.now = @max(value.now, choiceTick(decision)); std.debug.assert(value.now >= previous); try record(events, value.now, decision); const advanced = switch (decision.choice) { .input => |choice| try applyInput(value, choice, events), .schedule => |choice| try applySchedule(config, value, choice, events), .fault => |choice| try applyFault(config, value, choice, events), .topology => return error.DecisionStreamMismatch, }; if (advanced) value.fabric = canon.advance(value.fabric, decision, value); std.debug.assert(value.fabric.entry_frontier == entry_frontier + @intFromBool(advanced)); value.assertValid();}Source: lib/machine/src/explore/distributed/root.zig
//! A tool that searches for bugs earns trust by finding a real one in a running//! system and replaying it on demand. This namespace runs a small three-node//! commit protocol with a bug planted on purpose, and a gate that has to find//! the bug, shrink it, replay it, and show that the fixed protocol survives the//! same search.//!//! In the protocol, node 0 coordinates rounds. Opening a round sends a proposal//! to the other two nodes, a node that receives the proposal accepts it and//! replies with an acknowledgement, and the coordinator announces a commit once//! its own vote plus the acknowledgements it counted reach two. Two safety//! rules are declared before any search runs: a commit needs two nodes that//! accepted the current round's value, and a node never accepts a proposal from//! an older round. The gate answers by passing or failing the package tests,//! and on success it returns a report for a renderer to show.//!//! A rule or a seed picked after seeing the failure proves nothing, so the//! rules are fixed ahead of the search and the seed comes from the machine//! configuration alone. A bug that one fault can trigger exercises little, so//! the planted bug needs two faults acting together across steps. A search that//! stops at the first failure, or whose result depends on the order it tries//! things in, can miss the bug or misreport it, so the search has to cover one//! fixed tree completely and keep going after each failure. A fix is believable//! only when it survives the same search that found the bug.//!//! The planted bug (the *seeded defect*) lives in how the coordinator counts.//! The defective variant counts every acknowledgement, a late one from a closed//! round and a repeat from the same node alike, so it can announce a commit//! that fewer than two nodes back. The bug needs two facts at once, a *stateful//! conjunction*: a node stays hung across two fault steps, so the coordinator//! suspects it and opens a new round, and an acknowledgement from the closed//! round arrives afterward. Every search walks the same six levels of choices,//! a *frozen choice tree*, in the order input, schedule, fault, fault,//! schedule, schedule, under a budget of 4096 executions. The seed is a SHA-256//! digest over the identities of the machine configuration, its shared//! execution rules, and its list of places where two runs could differ,//! together with the protocol's encoding version.//!//! The defective search covers the tree in 3017 executions with nothing pruned,//! and it settles nine failing branches, each of which carries both facts. Two//! single-fault variants run the same budget and fail nowhere: one delivers//! every acknowledgement the moment it is sent, and the other never lets a hang//! last. The repaired variant counts an acknowledgement only for the current//! round and only once per node, reaches both facts together in 17 branches,//! and violates neither rule.//!//! The first failure shrinks to five steps, and the gate confirms that none of//! them can be removed: a terminal input, a step for a node other than the//! coordinator, an injected crash, the crash held for another step, and a step//! for the coordinator. The shortened record replays twice to one history//! identity and survives a round trip through its byte encoding. A sibling path//! that takes another alternative at the last choice gives the first step where//! two recorded histories differ, as one stable *first divergence* at that//! choice, encoded as a reference, and it records the reason and which fields//! of the two reached positions differ. The report carries where the seed came//! from, the four search summaries, the shrinking statistics, the replay//! identities, and the signature of that first divergence.//!//! - *workload*: the three-node replicated-commit protocol the gate explores,//! in a defective, a repaired, or a single-fault variant.//! - *witness gate*: the run that finds the defect, shrinks its capsule,//! replays it twice, branches from it, and checks the single-fault and//! repaired variants.//! - *capsule*: a fixed-capacity record of one failing run that replays it//! exactly.//! - *seed provenance*: the record of where the gate's seed came from: the//! profile fingerprint, the contract fingerprint, the inventory identity, the//! encoding version, and the seed itself.const canon = @import("canon.zig");const driver = @import("driver.zig");const property = @import("property.zig");const queue = @import("queue.zig");const replay = @import("replay.zig");const seed = @import("seed.zig");const state = @import("state.zig");const types = @import("types.zig");const witness = @import("witness.zig");const workload = @import("workload.zig");pub const Capsule = replay.Capsule;pub const Config = types.Config;pub const Conjunct = types.Conjunct;pub const Cursors = driver.Cursors;pub const Delivery = types.Delivery;pub const Diagnostic = types.Diagnostic;pub const Diff = replay.Diff;pub const Driver = driver.Driver;pub const Error = types.Error;pub const Events = types.Events;pub const Exploration = types.Exploration;pub const History = replay.History;pub const Message = types.Message;pub const MessageKind = types.MessageKind;pub const Node = types.Node;pub const Persistence = types.Persistence;pub const Property = types.Property;pub const Provenance = types.Provenance;pub const Queue = queue.Queue;pub const Reducer = replay.Reducer;pub const Reduction = types.Reduction;pub const Replay = types.Replay;pub const Replayer = replay.Replayer;pub const Report = types.Report;pub const Search = driver.Search;pub const Semantic = types.Semantic;pub const Signature = types.Signature;pub const State = state.State;pub const Variant = types.Variant;pub const Witness = witness.Witness;pub const budget = types.budget;pub const capsule_capacity = types.capsule_capacity;pub const coordinator = types.coordinator;pub const declarations = property.declarations;pub const defective = types.defective;pub const delayed_only = types.delayed_only;pub const depth = types.depth;pub const freshness_rule = property.freshness_rule;pub const history_capacity = types.history_capacity;pub const kinds = types.kinds;pub const node_count = types.node_count;pub const persistent_only = types.persistent_only;pub const quorum = types.quorum;pub const quorum_rule = property.quorum_rule;pub const repaired = types.repaired;pub const search_capacity = types.search_capacity;pub const suspicion_steps = types.suspicion_steps;pub const tree = types.tree;pub const builds = canon.builds;pub const initialRoot = canon.initialRoot;pub const origin = canon.origin;pub const seedProvenance = seed.provenance;pub const selectedProfile = seed.selected;pub const step = workload.step;Source: lib/machine/src/explore/root.zig:82
pub const distributed = @import("distributed/root.zig");Source: lib/machine/src/explore/distributed/types.zig:24
pub const depth: u16 = tree.len;Complete call list for explore.distributed.Witness.run
7 direct calls.
tiny.machine.explore.distributed.Replayer.init[method] atlib/machine/src/explore/distributed/replay.zig:51lib.machine.src.explore.distributed.witness.Witness.ablate[method] — private source atlib/machine/src/explore/distributed/witness.zig:88in nearest public ownerlib.machine.src.explore.distributed.witnesslib.machine.src.explore.distributed.witness.Witness.capture[method] — private source atlib/machine/src/explore/distributed/witness.zig:101in nearest public ownerlib.machine.src.explore.distributed.witnesslib.machine.src.explore.distributed.witness.Witness.collect[method] — private source atlib/machine/src/explore/distributed/witness.zig:177in nearest public ownerlib.machine.src.explore.distributed.witnesslib.machine.src.explore.distributed.witness.Witness.confirm[method] — private source atlib/machine/src/explore/distributed/witness.zig:155in nearest public ownerlib.machine.src.explore.distributed.witnesstiny.machine.explore.distributed.Witness.fork[method] atlib/machine/src/explore/distributed/witness.zig:187lib.machine.src.explore.distributed.witness.Witness.minimize[method] — private source atlib/machine/src/explore/distributed/witness.zig:116in nearest public ownerlib.machine.src.explore.distributed.witness
Audit
| Definitions | 87 |
|---|---|
| Public names | 87 |
| Members | 132 |
| Version | 26.7.0 |
| Revision | daab053ee433 |