Renewal Reward Rate

Concept & Intuition

Why does the long-run reward per unit time equal the ratio of expectations E[R] / E[L] rather than the expectation of the single-cycle ratio E[R / L]? Longer cycles occupy a larger share of total elapsed time than shorter cycles.

Specifically, a cycle of type i with selection probability pi and duration Li accounts for an asymptotic fraction wi = (pi Li) / E[L] of total time. Taking the time-weighted average of the individual cycle rates Ri / Li recovers ∑i [(pi Li) / E[L]] (Ri / Li) = E[R] / E[L].

Mathematical Formulation

Given nonnegative integer weights WA, WB, WC with positive sum W = WA + WB + WC, the cycle probabilities are pi = Wi / W for the three cycle types (LA, RA) = (1, 0), (LB, RB) = (2, 1), and (LC, RC) = (4, 3):

E[L] = ∑i pi Li, E[R] = ∑i pi Ri, Long-Run Rate = E[R] / E[L], and Mean Cycle Rate = E[R / L] = ∑i pi (Ri / Li).

Worked Example & Boundary Behavior

Consider a system with three types of cycles, chosen with equal probability (weights 1, 1, 1). The cycle types are:

1. Type A: Length L = 1, Reward R = 0. The rate for this individual cycle is R/L = 0/1 = 0.
2. Type B: Length L = 2, Reward R = 1. The rate for this cycle is R/L = 1/2 = 0.5.
3. Type C: Length L = 4, Reward R = 3. The rate for this cycle is R/L = 3/4 = 0.75.

Since the weights are equal, the probabilities pi are each 1/3. We calculate the expected values:

1. E[L] = (1/3)×1 + (1/3)×2 + (1/3)×4 = 7/3 ≈ 2.333.
2. E[R] = (1/3)×0 + (1/3)×1 + (1/3)×3 = 4/3 ≈ 1.333.

The true long-run reward rate is given by the ratio of expected values: E[R]/E[L] = (4/3) / (7/3) = 4/7 ≈ 0.571, whereas the unweighted average of individual cycle rates is E[R/L] = (0 + 1/2 + 3/4)/3 = 5/12 ≈ 0.417.

Interactive Experiment

Open the Renewal Reward Rate experiment to observe the difference between these rates in real time.

1. Ensure all weights (Weight A, Weight B, Weight C) are set to 1.
2. Observe that the E[R]/E[L] result shows 4/7 (0.571), while the E[R/L] result shows 5/12 (0.417).
3. Change Weight A to 0 and Weight B to 0, leaving Weight C at 1.
4. Notice that both E[R]/E[L] and E[R/L] now equal 3/4 (0.75) because only a single cycle length is active.

Interpretation & Limitations

The renewal reward theorem is an asymptotic statement about the limit as time goes to infinity: the cumulative reward divided by elapsed time converges to E[R]/E[L] almost surely for IID cycles with finite expectations and E[L] > 0. When all weights are set to zero, no probability distribution is defined and the module explicitly reports an invalid law.

References

The fundamental results on renewal reward processes can be found in:

• Random Services: Renewal Reward Processes — Formalizes the renewal reward theorem and explains why the long-run rate is E[R]/E[L] rather than E[R/L].