Skip to guide

The environment reacts

Reactive Events & Fault Recovery

Model errors, asynchronous interrupts, firmware handshakes, recovery paths, and deliberately hostile conditions as first-class scenarios.

Build reactive stimulus that pre-empts normal traffic, predicts expected exceptions, observes side effects, and proves recovery rather than merely detecting corruption.

4 connected case studiesFailure → evidence → recoveryMechanism diagrams, code, stress, and Q&A
The environment reactsOne reusable contract across 4 cases.
  1. 01TRIGGERInject a fault or observe an asynchronous event
  2. 02DETECTPredict the exact exception or service obligation
  3. 03CONTAINPrevent corrupted state from becoming architectural
  4. 04RECOVERClear, acknowledge, retry, or restore useful work

Negative testing · IRQ Testing · Deadlock Detection · Firmware DV

01 · Reactive and fault

Negative Testing and Exception Scoreboarding

Treat injected faults as first-class transactions so the testbench can prove safe failure, payload suppression, error signaling, and status updates without false scoreboard mismatches.

Mechanism · failure · evidence · recovery
Expected-failure ledgerInject one fault, predict every externally visible consequence.

The testbench records fault intent before injection, then checks payload suppression, error reporting, sticky status, interrupt behavior, and recovery as one correlated outcome.

01Intentcorrupt packet ID 4202Injectflip ECC bit at boundary03Observeerror response · no payload04Accountstatus + counter + IRQ05Recoverclear · retry · resume
PayloadsuppressedmatchResponseSLVERRmatchSticky statusECC_FATAL = 1matchRecoverynext clean packet passespending

injected fault token ↔ expected exception token ↔ observed response; unrelated traffic stays on the normal scoreboard path

Why it matters

  • Server, automotive, and mobile silicon must fail safely when data is corrupted.
  • An unhandled ECC error or poisoned packet can cause silent data corruption or a permanent system hang.
  • The correct outcome of a corrupted transaction may be a dropped payload plus an error flag and status-counter update, not Correct In equals Correct Out.

What is difficult

  • Most ordinary scoreboards assume every input should produce valid output data.
  • The injection path and scoreboard expectation must identify the same transaction.
  • A predicted exception must be consumed exactly once so a later clean transaction with the same ID is not masked.
  • The checker must verify both the absence of corrupted payload and the presence of required error side effects.

Failure signatures

  • An expected CRC or ECC rejection appears as a false data mismatch.
  • The DUT forwards a poisoned payload instead of dropping it.
  • The DUT drops the payload but fails to assert its error flag.
  • Error-status registers or counters do not update.
  • The scoreboard suppresses an unexpected later mismatch because a predicted-error entry was not deleted.
  • A transaction is corrupted without a matching predicted-error notification.

Two viable approaches—and their cost

Transaction poisoning

Flip parity, CRC, ECC, or response bits directly in the transaction or driver path.

Strengths
  • Simple to add for a single agent or packet type.
  • Makes directed corruption easy to understand.
Costs
  • Couples error behavior to the driver.
  • Makes expected-failure coordination with the scoreboard difficult.

Error-injection callback or proxy

Keep corruption policy in a reusable layer that can notify checking before the modified transfer reaches the DUT.

Strengths
  • Decouples fault logic from the normal driver.
  • Reusable across agents and corruption types.
  • Provides one place to coordinate injection and expected exception state.
Costs
  • Requires a clean uvm_callback or proxy ownership model.
  • Must define deterministic ordering between notification and observed DUT response.

Interview answer, built from the mechanism

  1. Treat errors as first-class transactions. An injection proxy can flip ECC or CRC bits, poison an AXI response, or otherwise mangle the transfer without contaminating the normal driver.
  2. Before the corrupted transaction reaches the DUT, notify the scoreboard with the stable transaction ID, for example, expect a failure for ID 50.
  3. The exception scoreboard checks the expected error set, proves that the DUT asserted its error indication, confirms the payload was dropped, and verifies error-status registers or counters.
  4. Delete the predicted-error entry after the expected exception is consumed so later traffic with a reused ID receives ordinary checking.

Component responsibility contract

Component responsibilities and required verification changes for Negative Testing and Exception Scoreboarding
ComponentResponsibilityRequired change
Agent configError knobsAdd enable_crc_err_injection and related fault controls for the driver or proxy.
Driver or proxyCorruption logicMangle parity, CRC, ECC, or response bits on the selected transfer.
Virtual sequenceExpectation notificationSend the corrupted transaction ID to checking before the DUT result arrives.
ScoreboardException handlingMaintain a Predicted Error Set and prevent false data mismatches.
RAL or status monitorSide-effect proofCheck the required error flag, status field, and counter increment.

Implementation patterns

Use an associative Predicted Error Setsystemverilog
// Associative-array membership is the set: key = transaction ID.
bit predicted_error_ids[int unsigned];

virtual function void write_error_notif(mac_item tr);
  predicted_error_ids[tr.id] = 1'b1;
endfunction

virtual function void check_data(mac_item act);
  if (predicted_error_ids.exists(act.id)) begin
    if (!act.error_flag)
      uvm_report_error("ERR", "DUT failed to detect injected error");

    check_payload_was_dropped(act.id);
    check_error_status_updated(act.id);
    predicted_error_ids.delete(act.id);
    return;
  end

  check_normal_data(act);
endfunction

The source mixes queue insertion with associative-array exists/delete calls. This corrected representation is an actual set keyed by transaction ID, so membership and one-time deletion are unambiguous.

Coordinate an injection proxy and the scoreboardtext
select transaction ID 50
notify scoreboard: predicted_error_ids includes 50
corrupt CRC, ECC, parity, or AXI response
drive corrupted transfer
observe DUT result
require: payload dropped
require: error flag asserted
require: status register or counter updated
remove ID 50 from predicted_error_ids

The virtual sequence controls both sides of the experiment: the injected fault and the expected exception contract.

Stress recipe

  1. Flip CRC, parity, or ECC bits on a selected transaction and register its ID in the predicted-error set before driving it.
  2. Poison an AXI response and prove the receiver rejects it without a permanent channel hang.
  3. Use the source example ID 50, then immediately reuse ID 50 for a clean transaction to prove the set entry was deleted.
  4. Check payload suppression, error_flag, error-status registers, and counter increments together.
  5. Inject an error without a notification and a notification without an error so the environment catches both coordination failures.

Follow-up questions

Why should errors be modeled as first-class transactions?

Because corruption changes the expected result. The checker needs the fault type and stable transaction identity to distinguish correct rejection from an ordinary mismatch.

Why use a callback or proxy instead of changing every driver?

A proxy separates reusable corruption policy from normal protocol driving and gives the environment one coordination point for injection plus predicted-error notification.

What must exception scoreboarding verify?

It must prove the corrupted payload was not accepted, the required error indication occurred, architectural status or counters updated, and the predicted exception was consumed exactly once.

02 · Reactive stimulus

IRQ Subsystem Verification

Verify nested, prioritized, level, edge, and coalesced interrupts with reactive sequences, pulse-aware monitoring, event prediction, and hardware-software service flows.

Mechanism · failure · evidence · recovery
Reactive event flowFrom event to service—and back to a cleared source.

The tracker predicts coalescing and priority while the reactive sequence models the software-visible service path.

1DMA done2Timer3ECC4Mailbox
Priority · mask · coalescingEvent trackerthreshold = 10 or safety timeout
IRQ
01grab()02read status03clear source04ungrab()

Why it matters

  • Interrupts are the hardware doorbell. A complex SoC interrupt controller may coordinate thousands of events.
  • A dropped interrupt can hang the operating system. Incorrect prioritization can make latency-sensitive work, such as audio processing, stutter.
  • The verification target includes the pin-level IRQ, the source/status mapping, priority behavior, and the software-visible read-and-clear service flow.

What is difficult

  • Interrupts are asynchronous to normal bus traffic, so a linear UVM sequence does not naturally model a high-priority IRQ interrupting a low-priority ISR.
  • A coalescing controller may wait for 10 packets before firing one IRQ. The checker must accept an interrupt on the 10th packet or when the safety timeout expires, whichever happens first.
  • Level-triggered interrupts persist until software clears them, while a one-clock edge interrupt can disappear before a polling sequence observes it.
  • An interrupt storm requires simultaneous source activity plus exact checking against the programmed priority registers.

Failure signatures

  • An event is dropped and the expected IRQ never arrives.
  • The arbiter services simultaneous sources in an order that disagrees with the priority registers.
  • A coalesced IRQ fires before event 10, after event 10, or after the timeout boundary.
  • A level IRQ deasserts before its clear write, or a one-cycle edge IRQ is never captured.
  • The IRQ pin toggles while the IP status register reports no active source, creating a spurious interrupt.
  • The testbench services an IRQ but never reads status or clears the source, hiding an incorrect hardware-software handshake.

Two viable approaches—and their cost

Reactive virtual sequences

Use wait-for-interrupt tasks to launch the ISR sequence when the DUT asserts an interrupt.

Strengths
  • Naturally models hardware-software interaction.
  • Makes nested-interrupt and pre-emption scenarios direct to express.
Costs
  • Introduces threading and arbitration complexity in the virtual sequencer.
  • Can become difficult to manage when IRQs fire faster than service sequences complete.

IRQ scoreboard and event tracker

Count source events, predict output IRQ pulses, and check event-to-line mapping, coalescing, pulse width, and timing.

Strengths
  • Provides accurate event-to-pulse and threshold checking.
  • Separates asynchronous prediction from bus stimulus.
Costs
  • Does not by itself verify the ISR read-status and clear-register flow.
  • Needs state for every source, line, threshold, timer, and pending event.

Interview answer, built from the mechanism

  1. I verify interrupt subsystems around pre-emption and coalescing. A reactive virtual sequence listens to the interrupt pins instead of assuming interrupts occur at a convenient point in a linear test.
  2. When a high-priority IRQ fires, the ISR sequence takes sequencer ownership, reads the status register, clears the interrupt, and then releases normal traffic. This models the software-visible service path as well as the pin.
  3. A dedicated scoreboard tracks source events, the programmed Event_ID-to-IRQ_Line mapping, pulse width, and the event-to-pulse ratio. It permits a coalesced IRQ only when the threshold is met or the safety timer expires.
  4. I add assertions for level persistence, a capture queue for one-cycle edges, simultaneous-source storm tests for priority, and a spurious-interrupt check that correlates every IRQ with an active status bit.

Component responsibility contract

Component responsibilities and required verification changes for IRQ Subsystem Verification
ComponentResponsibilityRequired change
MonitorPulse detectionCapture IRQ edge, level, and pulse width, and timestamp the trigger.
SequencerPre-emptionUse is_relevant() or an equivalent arbitration policy to pause normal traffic while an IRQ is pending.
ScoreboardEvent mappingMap Event_ID from the IP to the output IRQ_Line and predict coalescing thresholds and timers.

Implementation patterns

Reactive traffic and ISR threadssystemverilog
task body();
  fork
    // Thread 1: Normal Traffic
    forever begin
      `uvm_do_with(req, {type == DATA_PACKET;})
    end

    // Thread 2: Interrupt Handler (The ISR)
    forever begin
      @(posedge vif.irq[0]); // High Priority IRQ
      `uvm_info("IRQ", "High-Prio IRQ Detected! Pre-empting...", UVM_LOW)

      // Stop current bus traffic to simulate SW reading the status reg
      m_sequencer.grab(this);
      `uvm_do(read_status_reg_seq);
      `uvm_do(clear_irq_seq);
      m_sequencer.ungrab(this);
    end
  join_any
endtask

The source slide runs normal traffic and an IRQ listener concurrently. The listener takes sequencer ownership, performs the status-read and clear-register flow, and then releases arbitration.

Stress recipe

  1. Program a coalescing threshold of 10 plus a known safety timeout.
  2. Send nine qualifying packets and prove that no IRQ is emitted.
  3. Send packet 10 and require the IRQ at the specified threshold boundary.
  4. Repeat with fewer than 10 packets, stop event arrival, and require the timeout-driven IRQ.
  5. Begin a low-priority ISR, assert a high-priority IRQ during service, and check pre-emption, status read, clear, and traffic resumption.
  6. Drive every event source simultaneously and compare service order with the priority-register image.
  7. Exercise a held level, a one-cycle edge, and an IRQ with no status source.

Follow-up questions

How do you verify level-triggered versus edge-triggered interrupts?

Use SVA to require a level IRQ to remain asserted until its Clear bit is written through RAL. Capture an edge IRQ in a local event queue even when it is only one clock wide.

How do you verify an interrupt storm?

Saturate all event inputs simultaneously and verify that the arbiter services them in the exact order defined by the priority registers.

What is a spurious interrupt, and how do you find it?

It is an IRQ with no active source. Flag an error if an IRQ pin toggles while the IP status register reports no active event.

03 · Progress and liveness

Deadlock, Livelock & Credit Watchdogs

Distinguish a quiet but healthy design from a circular wait by tracking accepted work, completions, activity, blocked IDs, and flow-control credits.

Mechanism · failure · evidence · recovery
Progress, not activityBusy can still mean stuck.

The watchdog compares accepted work, completed work, dependency progress, and transaction age so livelock is not mistaken for useful motion.

Queue AQueue B
Accepted128
Completed91
Oldest age5,000 cycles

in_flight > 0 ∧ completions unchanged ∧ dependency has no progress → dump pending IDs and credits

Why it matters

  • A deadlock is a silent failure: simulation time advances, but packets stop moving because A waits for B while B waits for A.
  • A generic test timeout reports only that the test failed. A progress watchdog captures the blocked state when the failure forms.
  • The diagnostic must distinguish a testbench sequence stuck in wait(ready) from an RTL arbiter or credit loop that cannot make forward progress.

What is difficult

  • Legitimate reset, initialization, and long-latency operations can look inactive and create false heartbeat failures.
  • Deadlock is a total stall, while livelock can keep state machines toggling without completing useful work.
  • A precise credit watchdog must understand accepted requests, completed responses, internal buffering limits, and when each credit should return.
  • A credit leak can degrade bandwidth slowly enough to appear only in a long soak test.

Failure signatures

  • Items remain in flight while the completion counter does not change for the watchdog window.
  • The testbench itself is blocked in a wait loop even though the RTL could accept work.
  • Internal state changes continue but external completions stop, producing livelock.
  • A response never returns a flow-control credit, reducing available bandwidth over time.
  • The watchdog fires during reset or initialization because its activity window is configured too aggressively.
  • A simple global timeout loses the pending-ID and credit state needed for root-cause analysis.

Two viable approaches—and their cost

UVM heartbeat monitor

Observe component activity, such as monitor write() calls, within a configured heartbeat window.

Strengths
  • Non-intrusive and broadly reusable.
  • Can expose both testbench hangs and RTL inactivity.
Costs
  • The activity window is difficult to tune.
  • Reset, initialization, or legitimate idle periods can cause false positives.

In-flight credit watchdog

Track the delta between accepted requests and completed responses together with a no-progress timer.

Strengths
  • Targets true blocked work instead of generic inactivity.
  • Can identify a specific credit leak or blocked transaction ID.
Costs
  • Couples the checker to buffering and flow-control rules.
  • Requires reliable request, completion, cancel, reset, and credit-return accounting.

Interview answer, built from the mechanism

  1. I implement a progress-based watchdog that tracks the delta between accepted and completed transactions.
  2. If work is in flight but the completion counter does not advance for a defined cycle window, the watchdog reports the failure immediately and dumps every pending ID plus the known credit state.
  3. That is more actionable than a test timeout because it captures the circular dependency at the point of failure.
  4. I distinguish livelock by comparing internal activity with external completions, verify the watchdog with deliberate negative stimulus, and run long soak tests for slow credit loss.

Component responsibility contract

Component responsibilities and required verification changes for Deadlock, Livelock & Credit Watchdogs
ComponentResponsibilityRequired change
ScoreboardProgress trackingMaintain a counter of items_in_flight and the pending-ID set.
EnvironmentTimeout watchdogRun a timer that resets only when a transaction completes.
MonitorActivity heartbeatTrigger an event every time a valid pin-level handshake is observed.

Implementation patterns

5,000-cycle progress watchdogsystemverilog
task run_phase(uvm_phase phase);
  forever begin
    // Check progress every 5000 cycles
    int initial_count = trans_completed_count;
    repeat(5000) @(posedge vif.clk);

    // Logic: If items are in-flight but no completions occurred
    if (in_flight_count > 0 &&
        trans_completed_count == initial_count) begin
      `uvm_error(
        "DEADLOCK",
        $sformatf(
          "No progress in 5000 cycles! In-Flight: %0d",
          in_flight_count
        )
      )
      dump_debug_info();
    end
  end
endtask

function void dump_debug_info();
  foreach (pending_ids[id]) begin
    `uvm_info(
      "DEADLOCK_DEBUG",
      $sformatf("ID %0d is still waiting for response", id),
      UVM_LOW
    )
  end
endfunction

The implementation snapshots the completion count, waits exactly 5,000 clocks, and reports only when work is still in flight and no completion occurred. It then enumerates the blocked IDs.

Stress recipe

  1. Create a valid accepted burst so at least one item and ID are known to be in flight.
  2. Force READY low indefinitely after the burst has started.
  3. Prove that no false failure occurs before the configured boundary and that the watchdog reports at 5,000 cycles.
  4. Check that the debug dump names every blocked ID and captures credit state before cleanup changes it.
  5. Create internal state activity without an external completion to verify livelock classification.
  6. Run a long, saturated soak test and trend available credits and maximum bandwidth to expose a slow leak.
  7. Repeat through reset and initialization with watchdog suppression or a longer window to test false-positive control.

Follow-up questions

How do you distinguish livelock from deadlock?

Deadlock is a total stall. Livelock is a busy state with no forward progress. Compare internal state-machine toggles with externally visible data completions.

How do you verify the watchdog itself?

Use negative stimulus in the driver and force READY low forever during an active burst. Require the watchdog to fire at the exact predicted cycle.

What is a credit leak, and how do you find it?

A credit leak occurs when the DUT does not return a credit. Run a long soak test and watch whether the maximum sustainable bus bandwidth slowly drops.

04 · End-to-end integration

Firmware & Hardware-Software Interface Verification

Model real boot, interrupt, mailbox, doorbell, timeout, and recovery behavior without waiting for a full firmware image to execute in a slow RTL simulator.

Mechanism · failure · evidence · recovery
Hardware–software contractA doorbell is a round trip, not one register write.

The verification sequence mirrors the firmware flow and checks programming order, hardware work, timeout behavior, interrupt status, acknowledgement, and recovery.

01CPU / driverprogram descriptor02MMIO doorbellwrite starts work03Hardware FSMfetch · execute · complete04IRQ + statuspublish completion or error05ISR / ACKread status · clear · recover
Firmware-equivalent UVMfast · delay-randomizable · can drift from the driverDPI-C shared implementationone source of truth · adapter and synchronization cost

Why it matters

  • The real user of modern silicon is often a firmware driver rather than a UVM sequence.
  • A poorly defined Hardware-Software Interface can create interrupt storms or mailbox deadlocks that block-level verification never exercised.
  • End-to-end integration checks that hardware responds correctly to the programming guide, boot flow, interrupt handling, and recovery behavior used by software.

What is difficult

  • A full 50 MB firmware binary can take impractically long to reach main() in a SystemVerilog RTL simulation.
  • A UVM reimplementation is fast but can drift when the software team changes the real driver.
  • Reusing C driver functions through DPI-C requires an adapter from function calls and software data structures to UVM transactions.
  • Mailbox Doorbell protocols combine register ordering, Ready/Acknowledge handshake timing, interrupts, and timeout recovery.

Failure signatures

  • A firmware-equivalent sequence no longer matches the current software driver or header definitions.
  • A doorbell write does not trigger the expected hardware state machine.
  • Hardware never asserts Ready, software never Acknowledges, or either side violates the mailbox timeout.
  • A slow hardware response makes the software sequence hang instead of entering recovery.
  • Interrupt coalescing fires before N packets, misses N packets, or ignores timer expiry.
  • Block-level verification passes even though the complete boot or interrupt-handling order deadlocks.

Two viable approaches—and their cost

Firmware-equivalent UVM sequences

Write SystemVerilog sequences that follow the Software Programming Guide and mirror boot and interrupt flows.

Strengths
  • Fast enough for RTL simulation.
  • Keeps the testbench in SystemVerilog and makes delays easy to randomize.
Costs
  • Can drift out of sync when the software team changes its driver.
  • Duplicates software control flow and constants unless generation or shared artifacts are used.

DPI-C driver reuse

Call the software team's C functions through DPI-C and adapt those operations into UVM agent transactions.

Strengths
  • Creates a shared source for driver behavior, header values, and checksum algorithms.
  • Can expose mismatches in headers and real driver logic.
Costs
  • Needs a complex adapter between C calls and time-consuming UVM transactions.
  • C execution, simulation time, re-entrancy, and software memory assumptions need explicit ownership.

Interview answer, built from the mechanism

  1. I focus firmware integration on the Hardware-Software Interface rather than attempting to boot an entire 50 MB image in RTL simulation.
  2. Firmware-equivalent UVM sequences mirror the exact boot flow and interrupt-handling steps from the programming guide, with randomized hardware delays and negative responses.
  3. Where practical, I use DPI-C to share the real C header definitions, checksum algorithms, and driver functions. A controlled adapter translates those calls into UVM transactions.
  4. The main sign-off scenarios are mailbox doorbell handshakes, Ready/Acknowledge timeout and recovery, interrupt service, and coalescing after N packets or timer expiry.

Component responsibility contract

Component responsibilities and required verification changes for Firmware & Hardware-Software Interface Verification
ComponentResponsibilityRequired change
Firmware-equivalent sequenceProgramming flowMirror the software boot, register, interrupt, mailbox, and recovery order.
DPI-C adapterDriver reuseTranslate C driver calls, headers, and checksum operations into UVM transactions.
HW agentTiming and fault injectionRandomize Ready, Acknowledge, interrupt, and slow-response behavior.
HSI scoreboardEnd-to-end contractCheck doorbell state transitions, timeout recovery, and interrupt coalescing.

Implementation patterns

Mailbox doorbell contracttext
SW writes Doorbell
  -> HW starts the mailbox state machine
  -> HW signals Ready
  -> SW Acknowledges
  -> transaction completes within the specified timeout

Interrupt coalescing:
  interrupt when packet_count == N
  OR when the coalescing timer expires

Pages 62 through 65 contain no source code. This native contract preserves the source's required doorbell, Ready/Acknowledge, timeout, recovery, and coalescing relationships.

Stress recipe

  1. Translate the documented boot flow into a firmware-equivalent sequence and trace every HSI register transition.
  2. Reuse or compare the software team's headers and checksum algorithms through the DPI-C boundary.
  3. Write the Doorbell register and require the expected hardware state-machine transition.
  4. Randomize the latency from Doorbell to Ready and from Ready to software Acknowledge.
  5. Inject a response slower than the specified timeout and require recovery rather than a blocked sequence.
  6. Generate N minus 1 packets and require no coalesced interrupt.
  7. Generate packet N and require the interrupt, then repeat with timer expiry before N.
  8. Run the same mailbox and interrupt order through boot and post-boot service contexts to expose integration-only deadlocks.

Follow-up questions

What is a doorbell register?

It is a register software writes to notify hardware that a task is ready. The write triggers a hardware state machine.

How do you verify software timeout logic?

Inject slow responses from the hardware agent and require the software sequence to enter its recovery flow instead of hanging.

How do you handle interrupt coalescing in software?

Verify that hardware interrupts software only after N packets arrive or the coalescing timer expires, reducing CPU context-switch overhead.

Continue the evidence chain

Connect cases to system strategy.

Move from focused failure modes to a complete plan, executable architecture, coverage model, regression system, and sign-off argument.
System designVerification StrategyFive reference architectures and a Plan → Execute → Sign-Off operating model.Open guide →All case studiesVerification Case-Study LibraryReset, power, clocks, CDC, ordering, QoS, faults, IRQs, firmware, performance, liveness, and infrastructure.View all families →