Wednesday 12 August 2026

1. Two first-step recursions

Track only what a pattern needs next, which makes each chain two states.

For HH, let $a=𝔼[N_{HH}]$ from a fresh start and $b$ the expected further wait after a head. From a fresh start the next toss is a head (move to the $b$ state) or a tail (start over):

$$a=1+\frac{1}{2}b+\frac{1}{2}a$$

From the $b$ state a head finishes and a tail sends you back to the start:

$$b=1+\frac{1}{2}·0+\frac{1}{2}a$$

Substituting gives $a=2+b$ and $b=1+a/2$, hence $a=6$ and $b=4$.

For TH, let $a^{\prime }$ be the wait from a fresh start and $b^{\prime }$ the wait once a tail has appeared. The difference is in the $b^{\prime }$ state: a head finishes, and a tail leaves you exactly where you were.

$$a^{\prime }=1+\frac{1}{2}{b}^{\prime }+\frac{1}{2}{a}^{\prime },b^{\prime }=1+\frac{1}{2}·0+\frac{1}{2}{b}^{\prime }$$

So $b^{\prime }=2$ and $a^{\prime }=4$.

$$𝔼[N_{HH}]=6,𝔼[N_{TH}]=4$$

2. Where the asymmetry comes from

HH overlaps itself: after a head, a tail destroys the progress you had made and returns you to the start. TH cannot overlap itself in that way — once a tail has appeared you are permanently one head away, and a further tail keeps you one head away. Progress toward TH is never lost, progress toward HH is, so HH waits longer. This is exactly the correlation term in Conway's leading-number algorithm: a pattern's expected waiting time grows with its self-overlap.

3. Which comes first

HH appears before TH if and only if the first two tosses are both heads, so the probability is $1/4$.

The forward direction is immediate. For the converse, suppose the first two tosses are not HH. If the first toss is a tail, then TH completes at the very first head that appears, and no HH can have occurred earlier because every toss before that first head is a tail. If the first toss is a head and the second a tail, the same argument applies from the second toss onward. Either way TH comes first.

$$\mathbb{P}(\text{HH before TH})=\frac{1}{4}$$

Note. Compare with HT, where the answer is $1/2$: after the first head, the next toss decides between HH and HT immediately. Pattern races are decided by overlap structure, not by the individual pattern probabilities, and swapping one letter changes the answer.

1. The estimator and its asymptotics

The log-likelihood is $n\log \lambda -\lambda {\sum }_i{X}_i$, with derivative $n/\lambda -{\sum }_i{X}_i$, so

$${\widehat{\lambda }}_n=\frac{1}{\overline{X}}.$$

For one observation, $\log f=\log \lambda -\lambda x$ has ${\partial }_{\lambda }^2\log f=-1/{\lambda }^2$, a constant, so

$$I(\lambda )=-𝔼\left[{\partial }_{\lambda }^2\log f\right]=\frac{1}{{\lambda }^2}.$$

Standard MLE asymptotics then give

$$\sqrt{n}({\widehat{\lambda }}_n-\lambda )\Rightarrow N(0,I(\lambda {)}^{-1})=N(0,{\lambda }^2).$$

2. Exact moments

Let $S={\sum }_i{X}_i~\mathrm{G}\mathrm{a}\mathrm{m}\mathrm{m}\mathrm{a}(n,\lambda )$ (shape $n$, rate $\lambda$). For $k<n$,

$$𝔼[S^{-k}]={\lambda }^k\frac{\Gamma (n-k)}{\Gamma (n)},$$

so $𝔼[S^{-1}]=\lambda /(n-1)$ for $n\ge 2$ and $𝔼[S^{-2}]={\lambda }^2/((n-1)(n-2))$ for $n\ge 3$. Since ${\widehat{\lambda }}_n=n/S$,

$$𝔼[{\widehat{\lambda }}_n]=\frac{n\lambda }{n-1},\mathrm{B}\mathrm{i}\mathrm{a}\mathrm{s}=\frac{\lambda }{n-1}.$$

The MLE overestimates the rate, for the reason Jensen's inequality predicts: $1/\overline{X}$ is a strictly convex function of $\overline{X}$, so its mean exceeds $1/𝔼[\overline{X}]=\lambda$. It is nonetheless consistent — the bias and the variance both vanish as $n\to \infty$.

3. Debiasing, and then doing better

Write $\tilde{\lambda }=\frac{n-1}{n}{\widehat{\lambda }}_n=(n-1)/S$, which is exactly unbiased. Using the moments above,

$$\mathrm{M}\mathrm{S}\mathrm{E}({\widehat{\lambda }}_n)=\frac{n^2{\lambda }^2}{(n-1)(n-2)}-\frac{2n{\lambda }^2}{n-1}+{\lambda }^2=\frac{(n+2){\lambda }^2}{(n-1)(n-2)}$$

$$\mathrm{M}\mathrm{S}\mathrm{E}(\tilde{\lambda })=\mathrm{V}\mathrm{a}\mathrm{r}(\tilde{\lambda })=\frac{(n-1){\lambda }^2}{n-2}-{\lambda }^2=\frac{{\lambda }^2}{n-2}=\frac{(n-1){\lambda }^2}{(n-1)(n-2)}$$

Comparing numerators, $n+2$ against $n-1$, the unbiased version wins for every $n\ge 3$, by

$$\mathrm{M}\mathrm{S}\mathrm{E}({\widehat{\lambda }}_n)-\mathrm{M}\mathrm{S}\mathrm{E}(\tilde{\lambda })=\frac{3{\lambda }^2}{(n-1)(n-2)}>0.$$

Now optimize over the whole family $c/S$. Its MSE is

$$\frac{c^2{\lambda }^2}{(n-1)(n-2)}-\frac{2c{\lambda }^2}{n-1}+{\lambda }^2,$$

a convex quadratic in $c$ minimized at $c=n-2$, giving

$$\mathrm{M}\mathrm{S}\mathrm{E}\left(\frac{n-2}{S}\right)=\frac{{\lambda }^2}{n-1},$$

smaller still. So the ranking is $(n-2)/S$, then $(n-1)/S$, then the MLE $n/S$ — and note that the optimal $c$ does not depend on the unknown $\lambda$, so this estimator is actually usable.

Note. Two lessons sit on top of each other here. The MLE's optimality is asymptotic, and at finite $n$ a deterministic rescaling beats it; and unbiasedness is not the objective, since the MSE-optimal member of the family is biased low on purpose. The tempting wrong answer is that $\tilde{\lambda }$ must be best because it is the unbiased one.