Skip to guide

Progress and liveness

Deadlock, Livelock & Credit Watchdogs

Distinguish a quiet but healthy design from a circular wait by tracking accepted work, completions, activity, blocked IDs, and flow-control credits.

Watercolor of a verification workstation with waveform traces and a development board.
Trace the mechanism from failure signature through independent evidence, component ownership, stress, and recovery.

Mechanism and evidence

Guided verification lab

Activity is not evidence of progress.

Inspect the watchdog boundary, then use dependencies and credits to distinguish circular wait from motion without completion.

STEP THROUGH
STATE & OBSERVATIONCircular wait / 1 of 4
OUTSTANDING WORK37
COMPLETION COUNTER91
ELAPSED / 5,0000

Check outstanding work + completion delta + elapsed window together.

OUTSTANDING WORK37COMPLETION COUNTER91ELAPSED / 5,0000Check outstanding work + completion delta + elapsed window together.

Checkpoint 1 · Snapshot the start of the window.

ELAPSED
0
OUTSTANDING
37
COMPLETED
91
EVIDENCE
snapshot
Compare all 4 checkpoints

  1. 1 · Snapshot the start of the window.

    ELAPSED
    0
    OUTSTANDING
    37
    COMPLETED
    91
    EVIDENCE
    snapshot

  2. 2 · At cycle 4,999, the window is incomplete.

    ELAPSED
    4999
    OUTSTANDING
    37
    COMPLETED
    91
    EVIDENCE
    waiting

  3. 3 · The window closes without completion.

    ELAPSED
    5000
    OUTSTANDING
    37
    COMPLETED
    91
    EVIDENCE
    timeout

  4. 4 · Inspect the closed dependency loop.

    ELAPSED
    5000
    OUTSTANDING
    37
    COMPLETED
    91
    EVIDENCE
    A ↔ B
1234ELAPSED0499950005000OUTSTANDING37373737COMPLETED91919191EVIDENCEsnapshotwaitingtimeoutA ↔ B
Inspect the contract

Snapshot the start of the window.

Outstanding
37
Completed
91

Record accepted work, completed work, and the outstanding set before starting the observation window.

Evidence to inspect

Preserve transaction IDs and dependency state.

Checkpoint 1 / 4
Inspect the checker Selected checkpoint pseudocode
Conceptual checker / selected checkpoint
start_completions = completed;
window_start = cycle;
Example contract & limitations

Example contract. The example observes a 5,000-cycle window with 128 accepted and 91 completed transactions (91 accepted in the idle scenario). Monitoring is enabled after initialization. A progress timeout is a trigger for diagnosis, not automatically proof of deadlock. Checkpoints show selected state changes, not equally spaced simulation cycles.

Read the complete walkthrough

Circular wait

  1. Snapshot the start of the window.. Record accepted work, completed work, and the outstanding set before starting the observation window. Evidence: Preserve transaction IDs and dependency state.
  2. At cycle 4,999, the window is incomplete.. The completion count has not advanced yet. Do not report the 5,000-cycle condition one cycle early. Evidence: Compare elapsed cycles against the declared observation window.
  3. The window closes without completion.. Outstanding work remains and the completion counter is unchanged. Capture a progress-timeout report before cleanup. Evidence: Check elapsed window, outstanding work, and completion delta together.
  4. Inspect the closed dependency loop.. Queue A waits for a credit held behind a response in queue B; B waits for a resource A will release only after that credit. This fixture contains circular wait. Evidence: Preserve pending IDs, credit ownership, oldest age, and the wait-for graph before any flush.

Activity without completion

  1. Snapshot the start of the window.. Record accepted work, completed work, and the outstanding set before starting the observation window. Evidence: Preserve transaction IDs and dependency state.
  2. At cycle 4,999, the window is incomplete.. The completion count has not advanced yet. Do not report the 5,000-cycle condition one cycle early. Evidence: Compare elapsed cycles against the declared observation window.
  3. The window closes without completion.. Outstanding work remains and the completion counter is unchanged. Capture a progress-timeout report before cleanup. Evidence: Check elapsed window, outstanding work, and completion delta together.
  4. Retries can move while useful work stalls.. The state machines retry and toggle, but no transaction completes. Record the repeating state and missing completion evidence. Evidence: Preserve pending IDs, credit ownership, oldest age, and the wait-for graph before any flush.

Completion in window

  1. Snapshot the start of the window.. Record accepted work, completed work, and the outstanding set before starting the observation window. Evidence: Preserve transaction IDs and dependency state.
  2. At cycle 4,999, the window is incomplete.. The completion count has not advanced yet. Do not report the 5,000-cycle condition one cycle early. Evidence: Compare elapsed cycles against the declared observation window.
  3. A completion proves window progress.. The completion count increased to 92 inside the window. That suppresses this progress-timeout condition. Evidence: Check elapsed window, outstanding work, and completion delta together.
  4. Keep the evidence and continue.. There is no progress timeout in this selected window. Continue monitoring later windows; this sample does not prove all future progress. Evidence: Preserve pending IDs, credit ownership, oldest age, and the wait-for graph before any flush.

No outstanding work

  1. Snapshot the start of the window.. Record accepted work, completed work, and the outstanding set before starting the observation window. Evidence: Preserve transaction IDs and dependency state.
  2. At cycle 4,999, the window is incomplete.. The completion count has not advanced yet. Do not report the 5,000-cycle condition one cycle early. Evidence: Compare elapsed cycles against the declared observation window.
  3. Idle is not a liveness failure.. No work is waiting. The outstanding-work guard prevents an idle system from being called deadlocked. Evidence: Check elapsed window, outstanding work, and completion delta together.
  4. Keep the evidence and continue.. There is no progress timeout in this selected window. Continue monitoring later windows; this sample does not prove all future progress. Evidence: Preserve pending IDs, credit ownership, oldest age, and the wait-for graph before any flush.
Related implementation: 5,000-cycle progress watchdog
task run_phase(uvm_phase phase);
  forever begin
    // Check progress every 5000 cycles
    int initial_count = trans_completed_count;
    repeat(5000) @(posedge vif.clk);

    // Logic: If items are in-flight but no completions occurred
    if (in_flight_count > 0 &&
        trans_completed_count == initial_count) begin
      `uvm_error(
        "DEADLOCK",
        $sformatf(
          "No progress in 5000 cycles! In-Flight: %0d",
          in_flight_count
        )
      )
      dump_debug_info();
    end
  end
endtask

function void dump_debug_info();
  foreach (pending_ids[id]) begin
    `uvm_info(
      "DEADLOCK_DEBUG",
      $sformatf("ID %0d is still waiting for response", id),
      UVM_LOW
    )
  end
endfunction

The implementation snapshots the completion count, waits exactly 5,000 clocks, and reports only when work is still in flight and no completion occurred. It then enumerates the blocked IDs.

Circular wait, checkpoint 1: Snapshot the start of the window.. Inspect the contract.
Understand the failure
Progress, not activityBusy can still mean stuck.

The watchdog compares accepted work, completed work, dependency progress, and transaction age so livelock is not mistaken for useful motion.

Queue AQueue B
Accepted128
Completed91
Oldest age5,000 cycles

in_flight > 0 ∧ completions unchanged ∧ dependency has no progress → dump pending IDs and credits

Forward-progress watchdog

Progress is judged from accepted work, completed work, and outstanding ownership rather than elapsed simulation time alone.

Accepted requests
Increment in-flight work and record the owning transaction ID.
Pending-ID tracker
Preserves the work and credit context needed for diagnosis.
Completion counter
Advances only when useful external progress is observed.
5,000-cycle window
Compares the completion snapshot after the configured clock interval.
Progress timeout
Fires when outstanding work exceeds the documented no-progress bound; dependency evidence then classifies the cause.
  1. Accepted handshake -> in-flight count and pending ID
  2. Completed response -> completion count and credit return
  3. No completion for 5,000 cycles + in-flight work -> progress-timeout report
  4. Internal toggles + no external completion -> livelock evidence
  5. Falling credits across a soak test -> credit-leak evidence

Why it matters

  • A deadlock is a silent failure: simulation time advances, but packets stop moving because A waits for B while B waits for A.
  • A generic test timeout reports only that the test failed. A progress watchdog captures the blocked state when the failure forms.
  • The diagnostic must distinguish a testbench sequence stuck in wait(ready) from an RTL arbiter or credit loop that cannot make forward progress.

What is difficult

  • Legitimate reset, initialization, and long-latency operations can look inactive and create false heartbeat failures.
  • Deadlock is a total stall, while livelock can keep state machines toggling without completing useful work.
  • A precise credit watchdog must understand accepted requests, completed responses, internal buffering limits, and when each credit should return.
  • A credit leak can degrade bandwidth slowly enough to appear only in a long soak test.

Failure signatures

  • Items remain in flight while the completion counter does not change for the watchdog window.
  • The testbench itself is blocked in a wait loop even though the RTL could accept work.
  • Internal state changes continue but external completions stop, producing livelock.
  • A response never returns a flow-control credit, reducing available bandwidth over time.
  • The watchdog fires during reset or initialization because its activity window is configured too aggressively.
  • A simple global timeout loses the pending-ID and credit state needed for root-cause analysis.
Compare approaches

Two viable approaches—and their cost

UVM heartbeat monitor

Observe component activity, such as monitor write() calls, within a configured heartbeat window.

Strengths
  • Non-intrusive and broadly reusable.
  • Can expose both testbench hangs and RTL inactivity.
Costs
  • The activity window is difficult to tune.
  • Reset, initialization, or legitimate idle periods can cause false positives.

In-flight credit watchdog

Track the delta between accepted requests and completed responses together with a no-progress timer.

Strengths
  • Targets true blocked work instead of generic inactivity.
  • Can identify a specific credit leak or blocked transaction ID.
Costs
  • Couples the checker to buffering and flow-control rules.
  • Requires reliable request, completion, cancel, reset, and credit-return accounting.
Explain it in an interview

Interview answer, built from the mechanism

  1. I implement a progress-based watchdog that tracks the delta between accepted and completed transactions.
  2. If work is in flight but the completion counter does not advance for a defined cycle window, the watchdog reports the failure immediately and dumps every pending ID plus the known credit state.
  3. That is more actionable than a test timeout because it captures the circular dependency at the point of failure.
  4. I distinguish livelock by comparing internal activity with external completions, verify the watchdog with deliberate negative stimulus, and run long soak tests for slow credit loss.
Assign responsibilities

Component responsibility contract

Component responsibilities and required verification changes for Deadlock, Livelock & Credit Watchdogs
ComponentResponsibilityRequired change
ScoreboardProgress trackingMaintain a counter of items_in_flight and the pending-ID set.
EnvironmentTimeout watchdogRun a timer that resets only when a transaction completes.
MonitorActivity heartbeatTrigger an event every time a valid pin-level handshake is observed.
Build the checker

Implementation patterns

5,000-cycle progress watchdogsystemverilog
task run_phase(uvm_phase phase);
  forever begin
    // Check progress every 5000 cycles
    int initial_count = trans_completed_count;
    repeat(5000) @(posedge vif.clk);

    // Logic: If items are in-flight but no completions occurred
    if (in_flight_count > 0 &&
        trans_completed_count == initial_count) begin
      `uvm_error(
        "DEADLOCK",
        $sformatf(
          "No progress in 5000 cycles! In-Flight: %0d",
          in_flight_count
        )
      )
      dump_debug_info();
    end
  end
endtask

function void dump_debug_info();
  foreach (pending_ids[id]) begin
    `uvm_info(
      "DEADLOCK_DEBUG",
      $sformatf("ID %0d is still waiting for response", id),
      UVM_LOW
    )
  end
endfunction

The implementation snapshots the completion count, waits exactly 5,000 clocks, and reports only when work is still in flight and no completion occurred. It then enumerates the blocked IDs.

Stress the design

Stress recipe

  1. Create a valid accepted burst so at least one item and ID are known to be in flight.
  2. Force READY low indefinitely after the burst has started.
  3. Prove that no false failure occurs before the configured boundary and that the watchdog reports at 5,000 cycles.
  4. Check that the debug dump names every blocked ID and captures credit state before cleanup changes it.
  5. Create internal state activity without an external completion to verify livelock classification.
  6. Run a long, saturated soak test and trend available credits and maximum bandwidth to expose a slow leak.
  7. Repeat through reset and initialization with watchdog suppression or a longer window to test false-positive control.

Follow-up questions

How do you distinguish livelock from deadlock?

Deadlock is a total stall. Livelock is a busy state with no forward progress. Compare internal state-machine toggles with externally visible data completions.

How do you verify the watchdog itself?

Use negative stimulus in the driver and force READY low forever during an active burst. Require the watchdog to fire at the exact predicted cycle.

What is a credit leak, and how do you find it?

A credit leak occurs when the DUT does not return a credit. Run a long soak test and watch whether the maximum sustainable bus bandwidth slowly drops.

Engineering qualifications