03 · Progress and liveness
Deadlock, Livelock & Credit Watchdogs
Distinguish a quiet but healthy design from a circular wait by tracking accepted work, completions, activity, blocked IDs, and flow-control credits.
The watchdog compares accepted work, completed work, dependency progress, and transaction age so livelock is not mistaken for useful motion.
in_flight > 0 ∧ completions unchanged ∧ dependency has no progress → dump pending IDs and credits
Why it matters
- A deadlock is a silent failure: simulation time advances, but packets stop moving because A waits for B while B waits for A.
- A generic test timeout reports only that the test failed. A progress watchdog captures the blocked state when the failure forms.
- The diagnostic must distinguish a testbench sequence stuck in wait(ready) from an RTL arbiter or credit loop that cannot make forward progress.
What is difficult
- Legitimate reset, initialization, and long-latency operations can look inactive and create false heartbeat failures.
- Deadlock is a total stall, while livelock can keep state machines toggling without completing useful work.
- A precise credit watchdog must understand accepted requests, completed responses, internal buffering limits, and when each credit should return.
- A credit leak can degrade bandwidth slowly enough to appear only in a long soak test.
Failure signatures
- Items remain in flight while the completion counter does not change for the watchdog window.
- The testbench itself is blocked in a wait loop even though the RTL could accept work.
- Internal state changes continue but external completions stop, producing livelock.
- A response never returns a flow-control credit, reducing available bandwidth over time.
- The watchdog fires during reset or initialization because its activity window is configured too aggressively.
- A simple global timeout loses the pending-ID and credit state needed for root-cause analysis.
Two viable approaches—and their cost
UVM heartbeat monitor
Observe component activity, such as monitor write() calls, within a configured heartbeat window.
- Non-intrusive and broadly reusable.
- Can expose both testbench hangs and RTL inactivity.
- The activity window is difficult to tune.
- Reset, initialization, or legitimate idle periods can cause false positives.
In-flight credit watchdog
Track the delta between accepted requests and completed responses together with a no-progress timer.
- Targets true blocked work instead of generic inactivity.
- Can identify a specific credit leak or blocked transaction ID.
- Couples the checker to buffering and flow-control rules.
- Requires reliable request, completion, cancel, reset, and credit-return accounting.
Interview answer, built from the mechanism
- I implement a progress-based watchdog that tracks the delta between accepted and completed transactions.
- If work is in flight but the completion counter does not advance for a defined cycle window, the watchdog reports the failure immediately and dumps every pending ID plus the known credit state.
- That is more actionable than a test timeout because it captures the circular dependency at the point of failure.
- I distinguish livelock by comparing internal activity with external completions, verify the watchdog with deliberate negative stimulus, and run long soak tests for slow credit loss.
Component responsibility contract
| Component | Responsibility | Required change |
|---|---|---|
| Scoreboard | Progress tracking | Maintain a counter of items_in_flight and the pending-ID set. |
| Environment | Timeout watchdog | Run a timer that resets only when a transaction completes. |
| Monitor | Activity heartbeat | Trigger an event every time a valid pin-level handshake is observed. |
Implementation patterns
task run_phase(uvm_phase phase);
forever begin
// Check progress every 5000 cycles
int initial_count = trans_completed_count;
repeat(5000) @(posedge vif.clk);
// Logic: If items are in-flight but no completions occurred
if (in_flight_count > 0 &&
trans_completed_count == initial_count) begin
`uvm_error(
"DEADLOCK",
$sformatf(
"No progress in 5000 cycles! In-Flight: %0d",
in_flight_count
)
)
dump_debug_info();
end
end
endtask
function void dump_debug_info();
foreach (pending_ids[id]) begin
`uvm_info(
"DEADLOCK_DEBUG",
$sformatf("ID %0d is still waiting for response", id),
UVM_LOW
)
end
endfunctionThe implementation snapshots the completion count, waits exactly 5,000 clocks, and reports only when work is still in flight and no completion occurred. It then enumerates the blocked IDs.
Stress recipe
- Create a valid accepted burst so at least one item and ID are known to be in flight.
- Force READY low indefinitely after the burst has started.
- Prove that no false failure occurs before the configured boundary and that the watchdog reports at 5,000 cycles.
- Check that the debug dump names every blocked ID and captures credit state before cleanup changes it.
- Create internal state activity without an external completion to verify livelock classification.
- Run a long, saturated soak test and trend available credits and maximum bandwidth to expose a slow leak.
- Repeat through reset and initialization with watchdog suppression or a longer window to test false-positive control.
Follow-up questions
How do you distinguish livelock from deadlock?
Deadlock is a total stall. Livelock is a busy state with no forward progress. Compare internal state-machine toggles with externally visible data completions.
How do you verify the watchdog itself?
Use negative stimulus in the driver and force READY low forever during an active burst. Require the watchdog to fire at the exact predicted cycle.
What is a credit leak, and how do you find it?
A credit leak occurs when the DUT does not return a credit. Run a long soak test and watch whether the maximum sustainable bus bandwidth slowly drops.