How well can you tell two stimuli apart from a noisy firing rate alone? Following Maneesh Sahani’s Theoretical Neuroscience slides on rate codes, suppose a neuron must support a binary choice — present / absent, up / down, horizontal / vertical. Call the two stimuli $\mathtt{s_0}$ and $\mathtt{s_1}$. On any trial the response (say a spike count) is $n$, drawn from one of two distributions:
$$ P(n\mid \mathtt{s_0})\quad\text{and}\quad P(n\mid \mathtt{s_1}). $$A decision rule sets a criterion $c$ and reports “$\mathtt{s_1}$” whenever $n > c$. Two numbers summarise it, exactly as on the slides — the hit rate, in Wikipedia’s terms the true-positive rate (TPR), and the false-alarm rate, the false-positive rate (FPR):
$$ \begin{aligned} \text{TPR}(c)\ \ (\text{hit rate}) &= P(n>c \mid \mathtt{s_1}) = \int_{c}^{\infty}\! P(n\mid \mathtt{s_1})\,dn,\\[3pt] \text{FPR}(c)\ \ (\text{false-alarm rate}) &= P(n>c \mid \mathtt{s_0}) = \int_{c}^{\infty}\! P(n\mid \mathtt{s_0})\,dn. \end{aligned} $$Both depend on the criterion $c$ — that is the whole point — so we carry the $(c)$ everywhere.
There is no single best $c$: lowering it catches more real $\mathtt{s_1}$ trials but also raises false alarms. Sweep $c$ across the whole axis and plot $\big(\text{FPR}(c),\,\text{TPR}(c)\big)$ — the receiver operating characteristic. Press play to sweep the criterion, drag the purple criterion line, or grab the operating point on the ROC plot directly; change $d'$ — how far apart the two distributions sit — to see the curve bow toward the perfect corner.
Reading the picture
The decision rule cuts the response axis at the criterion $c$: everything to the right is predicted “$\mathtt{s_1}$” (positive), everything to the left “$\mathtt{s_0}$” (negative). That splits each distribution into the four cells of the Wikipedia confusion matrix — true/false × positive/negative:
| predicted $\mathtt{s_1}$: $n>c$ | predicted $\mathtt{s_0}$: $n\le c$ | |
|---|---|---|
| condition $\mathtt{s_1}$ — positives (P) | true positive · TP (hit) | false negative · FN (miss) |
| condition $\mathtt{s_0}$ — negatives (N) | false positive · FP (false alarm) | true negative · TN (correct rejection) |
Counting each cell as the area under the relevant distribution, and since the decision is “positive when $n>c$”, each ROC axis is a tail probability — written in Wikipedia’s canonical form:
$$ \begin{aligned} \text{TPR}(c) &= P(n>c \mid \mathtt{s_1}) = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} = \int_{c}^{\infty}\! P(n\mid \mathtt{s_1})\,dn,\\[4pt] \text{FPR}(c) &= P(n>c \mid \mathtt{s_0}) = \frac{\mathrm{FP}}{\mathrm{FP}+\mathrm{TN}} = \int_{c}^{\infty}\! P(n\mid \mathtt{s_0})\,dn. \end{aligned} $$(TPR is also called sensitivity or recall; FPR is fall-out, $1-\text{specificity}$.) So the teal tail under $P(n\mid \mathtt{s_1})$ is $\text{TPR}(c)$ and the terracotta tail under $P(n\mid \mathtt{s_0})$ is $\text{FPR}(c)$; drop both onto the ROC axes and you get one point.
The operating point is the criterion
Each value of $c$ produces exactly one pair $\big(\mathrm{FPR}(c),\,\mathrm{TPR}(c)\big)$ — a single dot in ROC space, the operating point, drawn here in the same purple as the criterion line because it is the same thing seen twice. The ROC curve is nothing more than the image of the whole criterion axis under that map,
$$ c\ \longmapsto\ \big(\mathrm{FPR}(c),\,\mathrm{TPR}(c)\big), \qquad c:\ +\infty\to-\infty \ \Longrightarrow\ (0,0)\to(1,1), $$so the two panels share a single degree of freedom: sliding the purple line and dragging the purple dot are the same knob. Raise the threshold (push $c$ right) and the point slides down the curve toward the origin — few hits but few false alarms, a conservative “rarely say $\mathtt{s_1}$”. Lower it (pull $c$ left) and the point climbs toward $(1,1)$ — a liberal “almost always say $\mathtt{s_1}$”. The map is monotone, so the operating point can only travel along the curve, never off it: the criterion chooses where on the curve you sit, while the separation $d'$ below sets which curve you are on.
What AUC measures
The area under the ROC curve is a single, threshold-free summary of how separable the two distributions are. Equivalently, it is the probability that a randomly drawn $\mathtt{s_1}$ response outranks a randomly drawn $\mathtt{s_0}$ response,
$$ \mathrm{AUC}=P\!\big(n_1 > n_0\big),\qquad n_1\sim P(n\mid \mathtt{s_1}),\; n_0\sim P(n\mid \mathtt{s_0}). $$Show the derivation — why AUC = P(n₁ > n₀)
Write the ROC curve parametrically in the criterion $c$, keeping the rule “report $\mathtt{s_1}$ when $n>c$”. With $F_0,F_1$ the cumulative distribution functions of the two responses and $f_0=P(\,\cdot\mid \mathtt{s_0})$ the $\mathtt{s_0}$ density,
$$ \mathrm{FPR}(c)=P(n_0>c)=1-F_0(c),\qquad \mathrm{TPR}(c)=P(n_1>c)=1-F_1(c). $$The AUC is the area under the curve — $\mathrm{TPR}(c)$ integrated against $\mathrm{FPR}(c)$ as $c$ sweeps the axis. Substituting $d\,\mathrm{FPR}=-f_0(c)\,dc$ and noting that $c:+\infty\to-\infty$ drives $\mathrm{FPR}:0\to1$, the two sign flips cancel:
$$ \mathrm{AUC}=\int_0^1 \mathrm{TPR}\;d\,\mathrm{FPR} =\int_{+\infty}^{-\infty}\!\big(1-F_1(c)\big)\big(-f_0(c)\big)\,dc =\int_{-\infty}^{\infty}P(n_1>c)\,f_0(c)\,dc. $$Now read $f_0(c)\,dc$ as the chance the $\mathtt{s_0}$ draw lands at $n_0=c$. The integrand is then $P(n_1>c\mid n_0=c)\,P(n_0=c)$, and integrating over $c$ is exactly the law of total probability (using that $n_0,n_1$ are independent):
$$ \mathrm{AUC}=\int_{-\infty}^{\infty}P\!\big(n_1>n_0\mid n_0=c\big)\,P(n_0=c)\,dc =P\!\big(n_1>n_0\big). $$So the area under the ROC curve is the probability that a random response to $\mathtt{s_1}$ outranks a random response to $\mathtt{s_0}$ — a single, threshold-free measure of separability. (Ties split evenly, contributing $\tfrac12$; for continuous distributions they have probability zero.)
For two unit-variance Gaussians a distance $d'=\mu_1-\mu_0$ apart — the equal-variance case drawn here — this has a closed form,
$$ \mathrm{AUC}=\Phi\!\left(\frac{d'}{\sqrt2}\right), $$so $d'=0$ gives AUC $=\tfrac12$ (the dashed no-discrimination diagonal, pure chance) and larger $d'$ pushes the curve toward the perfect corner $(0,1)$ with AUC $\to 1$. Pull the $d'$ slider to watch the distributions separate and the area fill in.
The whole confusion matrix, and where precision sits
The ROC above uses two numbers, $\mathrm{TPR}(c)$ and $\mathrm{FPR}(c)$. They are two of eight ratios you can build from the four cells of a confusion matrix, and it is worth seeing why those two. In the figure the dashed line splits relevant elements from irrelevant ones — the actual class — and the circle is what the classifier selected, the predicted class. Those two cuts partition a population of $\textcolor{#4f463c}{\mathrm{tot}}$ elements into four cells, and everything else is bookkeeping on them. Each cell keeps its colour wherever it appears below.
There are three ways to cut the same total, and which one you divide by is what distinguishes the metrics:
$$ \textcolor{#4f463c}{\mathrm{tot}} \;=\; \textcolor{#1f6765}{\mathrm{TP}}+\textcolor{#b4532a}{\mathrm{FP}}+\textcolor{#57908d}{\mathrm{FN}}+\textcolor{#c08356}{\mathrm{TN}} \;=\; \textcolor{#1a5b59}{\mathrm{P}}+\textcolor{#9c4a26}{\mathrm{N}} \;=\; \textcolor{#6d3fd4}{\mathrm{PP}}+\textcolor{#9b7fd9}{\mathrm{PN}}. $$| whole population tot = 100 P + N = PP + PN | Actual condition | row-normalised divided by a row total | ||
|---|---|---|---|---|
| Positive P | Negative N | |||
| Predicted | Positive PP | TPtrue positive · hit | FPfalse positive · type I | Precision = PPV positive predictive value TPTP + FP=TPPP TP + FP = PP — all predicted positive FDR false discovery rate FPTP + FP=FPPP TP + FP = PP — so FDR = 1 − precision |
| Negative PN | FNfalse negative · type II | TNtrue negative | FOR false omission rate FNFN + TN=FNPN FN + TN = PN — all predicted negative NPV negative predictive value TNFN + TN=TNPN FN + TN = PN — so NPV = 1 − FOR | |
| column-normalised divided by a column total → the ROC axes | Recall = TPR sensitivity, hit rate TPTP + FN=TPP TP + FN = P — all actual positive FNR miss rate FNTP + FN=FNP TP + FN = P — so FNR = 1 − recall | FPR fall-out, false-alarm rate FPFP + TN=FPN FP + TN = N — all actual negative TNR specificity, selectivity TNFP + TN=TNN FP + TN = N — so TNR = 1 − FPR | Prevalence PP + N=Ptot P + N = tot — normalised over the whole population Accuracy TP + TNP + N=TP + TNtot P + N = PP + PN = tot F₁ harmonic mean of precision and recall 2TP2TP + FP + FN=2TPPP + P 2TP + FP + FN = PP + P — note: not over tot Youden's J informedness TPR − FPR height of the ROC above the diagonal Markedness PPV + NPV − 1 the row-normalised mirror of J MCC √( J · markedness ) geometric mean of the two, so it is symmetric under swapping prediction and truth | |
The two the subject is named after, written out over both denominators:
$$ \text{Precision}\;=\;\frac{\textcolor{#1f6765}{\mathrm{TP}}}{\textcolor{#1f6765}{\mathrm{TP}}+\textcolor{#b4532a}{\mathrm{FP}}}\;=\;\frac{\textcolor{#1f6765}{\mathrm{TP}}}{\textcolor{#6d3fd4}{\mathrm{PP}}}, \qquad\qquad \text{Recall}\;=\;\frac{\textcolor{#1f6765}{\mathrm{TP}}}{\textcolor{#1f6765}{\mathrm{TP}}+\textcolor{#57908d}{\mathrm{FN}}}\;=\;\frac{\textcolor{#1f6765}{\mathrm{TP}}}{\textcolor{#1a5b59}{\mathrm{P}}}. $$Same numerator, different denominator, and the denominators are the whole story. Precision divides by everything inside the circle: of what the classifier picked, how much should it have picked? Recall divides by everything left of the dashed line: of what it should have picked, how much did it get? Neither divides by $\textcolor{#4f463c}{\mathrm{tot}}$ — only prevalence and accuracy do.
That difference decides which pair belongs on an ROC. Recall and FPR are column-normalised, each conditioned on the actual class, so neither depends on how many positives exist. Precision is row-normalised, conditioned on the prediction, and its denominator $\textcolor{#6d3fd4}{\mathrm{PP}} = \textcolor{#1f6765}{\mathrm{TP}}+\textcolor{#b4532a}{\mathrm{FP}}$ mixes the two classes — so it moves when prevalence moves, with the classifier untouched. Make positives rarer, holding $\textcolor{#4f463c}{\mathrm{tot}}$ fixed, and precision falls while recall, FPR and the entire ROC stay exactly where they were. That is why AUC is a property of the classifier alone, and why a precision-recall curve, often the more useful thing to look at when positives are rare, is not comparable across datasets with different base rates.
When accuracy is the AUC
Accuracy and AUC usually measure different things: accuracy is read off a single operating point, AUC integrates over all of them. But there are two situations where the two coincide exactly, and they are worth separating because only one of them is a coincidence.
1. A hard binary classifier, with balanced classes. Suppose the score takes only two values, so there is no threshold left to sweep. Its ROC is not a curve but a single interior point $(\mathrm{FPR}, \mathrm{TPR})$, joined to $(0,0)$ and $(1,1)$ by straight chords. The area under that polygon is
$$ \mathrm{AUC}\;=\;\tfrac{1}{2}\bigl(\mathrm{TPR}+\mathrm{TNR}\bigr)\;=\;\text{balanced accuracy}. $$Balanced accuracy is accuracy weighted so each class contributes half. When $\textcolor{#1a5b59}{\mathrm{P}} = \textcolor{#9c4a26}{\mathrm{N}}$ that reweighting does nothing, so plain accuracy already equals it, and
$$ \text{accuracy}\;=\;\frac{\textcolor{#1f6765}{\mathrm{TP}}+\textcolor{#c08356}{\mathrm{TN}}}{\textcolor{#4f463c}{\mathrm{tot}}}\;=\;\mathrm{AUC}. $$Both conditions are needed. Drop the binary scores and AUC exceeds balanced accuracy, because sweeping a threshold can only add area. Drop the balance and accuracy drifts toward whichever class is larger while AUC does not move at all — the same prevalence-independence that put recall and FPR on the ROC axes in the first place.
2. Two-alternative forced choice. Here the equality is not a coincidence but a restatement. By the Mann–Whitney identity,
$$ \mathrm{AUC}\;=\;\Pr\bigl(s(x^{+}) > s(x^{-})\bigr), $$the probability that a randomly drawn positive outscores a randomly drawn negative. If your task is to show a judge one positive and one negative and ask which is which, then the fraction of trials it gets right is a direct estimate of that probability. The accuracy you measure is the AUC — not approximately, but definitionally, with chance at $0.5$ rather than at $0$.
This is why a paired protocol needs no threshold and no calibration: pairing supplies the comparison that a threshold would otherwise have to stand in for. It is also why such an accuracy is not comparable to an unpaired one. A method scoring $0.44$ in a forced choice is below chance, while $0.44$ on a task with a floor at $0$ is a mediocre positive. Same number, opposite sign.
Two entries connect back to what came earlier. Youden’s J, $\mathrm{TPR}-\mathrm{FPR}$, is the vertical distance from the ROC curve to the diagonal, so maximising it picks the operating point furthest above chance — the tangent-slope-1 point. Markedness, $\mathrm{PPV}+\mathrm{NPV}-1$, is its row-normalised mirror, and the Matthews correlation coefficient is the geometric mean of the two, which is what makes MCC symmetric under swapping the roles of prediction and truth.
Psychometric curves: the same sweep, seen from the stimulus side
Everything above holds the stimulus fixed and slides the criterion $c$. Psychophysics does the opposite, holding the observer fixed and moving the stimulus. The resulting plot, the probability of responding “$\mathtt{s_1}$” against stimulus intensity, is the psychometric function. It is the same construction seen along a different axis, and this section makes that exact — every formula below reduces to its counterpart above in the right limit.
The decision variable, now with a stimulus in it
The first half had exactly one response distribution per stimulus. Here we need many, because there are many items — individual neurons, subjects, or models — indexed by $i$. Write $\mu_i$ for item $i$’s mean decision variable, its latent score. Everything else carries over untouched: $n$ is still the decision variable, $c$ is still the criterion, and the rule is still “report $\mathtt{s_1}$ when $n>c$”.
What the experimenter varies is the stimulus level $x$, and $\Delta(x)$ denotes how much stimulus level $x$ adds to the decision variable. One trial on item $i$ at level $x$ then produces
$$ n \;=\; \mu_i + \Delta(x) + \varepsilon, \qquad \varepsilon\sim\mathcal N(0,\sigma^2), $$where $\sigma$ is the within-item noise, the trial-to-trial spread at fixed $i$. The first half’s unit-variance distributions are the case $\sigma=1$. Applying the unchanged decision rule,
$$ P_i(x)\;=\;P(n>c)\;=\;\Phi\!\left(\frac{\mu_i+\Delta(x)-c}{\sigma}\right). $$That is the psychometric function: a sigmoid in $\Delta(x)$, centred where $\Delta(x)=c-\mu_i$, with slope set by $1/\sigma$. The noise that smeared the two distributions above is the same $\sigma$ that sets how gently this curve rises.
The threshold is the latent, measured against the criterion
Define item $i$’s threshold $x^{*}_i$ by $P_i(x^{*}_i)=\tfrac12$. Then
$$ \Delta(x^{*}_i)\;=\;c-\mu_i . $$Where the curve crosses 50% returns $\mu_i$, read off against the criterion. Fitting a psychometric threshold is therefore a measurement of the latent, not merely a curve fit, and one threshold per item gives a per-item score that can be ranked.
Sweeping the stimulus sweeps the criterion
Rearrange the decision rule. Reporting “$\mathtt{s_1}$” when $\mu_i+\Delta(x)+\varepsilon>c$ is the same as reporting it when $\mu_i+\varepsilon>c-\Delta(x)$. So define the effective criterion
$$ c_{\text{eff}}(x)\;=\;c-\Delta(x), $$and raising the stimulus lowers the effective criterion one for one. The stimulus axis and the criterion axis of the first half are the same axis, traversed in opposite directions. Each stimulus level therefore yields one pair $\big(\mathrm{FPR},\mathrm{TPR}\big)$ and the sweep traces an ROC, exactly as sliding $c$ did above. The psychometric plot keeps the stimulus axis and shows one curve per item; the ROC eliminates it and shows the trade-off it generates.
Two AUCs, and why the swept one is smaller
Here the two halves genuinely differ, because a second variance appears. Let the item latents be $\mu_i\sim\mathcal N(\mu_1,s^2)$ for $\mathtt{s_1}$ items and $\mathcal N(\mu_0,s^2)$ for $\mathtt{s_0}$ items. That introduces $s$, the between-item spread of $\mu_i$ within one class, and the class means $\mu_0,\mu_1$, so that $d'=\mu_1-\mu_0$ exactly as in the first half — which is now visibly the case $s=0$, one distribution per class with no item-to-item variation. Ranking items by their true latent gives
$$ \mathrm{AUC}_{\text{latent}}\;=\;P(\mu_i^{(1)}>\mu_j^{(0)})\;=\;\Phi\!\left(\frac{d'}{s\sqrt2}\right). $$Ranking instead by observed binary responses swept over $x$ uses $\mathbb E_\mu\big[\Phi((\mu-c_{\text{eff}})/\sigma)\big] =\Phi\big((\bar\mu-c_{\text{eff}})/\sqrt{\sigma^2+s^2}\big)$, so
$$ \mathrm{TPR}(x)=\Phi\!\left(\frac{\mu_1-c_{\text{eff}}(x)}{\sqrt{\sigma^2+s^2}}\right),\qquad \mathrm{FPR}(x)=\Phi\!\left(\frac{\mu_0-c_{\text{eff}}(x)}{\sqrt{\sigma^2+s^2}}\right), $$a unit-slope zROC, and therefore
$$ \mathrm{AUC}_{\text{swept}}\;=\;\Phi\!\left(\frac{d'}{\sqrt2\,\sqrt{\sigma^2+s^2}}\right) \;\le\;\mathrm{AUC}_{\text{latent}} . $$Set $s=0$ and $\sigma=1$ and this is $\Phi(d'/\sqrt2)$, the first half’s result, as it must be.
Within-item noise attenuates the swept AUC, however finely the stimulus is sampled. Sweeping buys the operating point, not the noise. If the latent is directly readable — a firing rate, a log-odds, a logit — the unattenuated number is recovered; if only binary responses are observable, the $\sigma$ penalty is unavoidable. The gap between the two readouts in the widget below is exactly this term.
When the correspondence breaks
All of it rests on $\Delta(x)$ shifting every item equally. Let the stimulus instead couple to item $i$ through its own sensitivity $\beta_i$, a per-item multiplier that has silently been $1$ everywhere above:
$$ n \;=\; \mu_i + \beta_i\,\Delta(x) + \varepsilon . $$The ordering of items by $P_i(x)$ now depends on $x$, and the psychometric curves cross. Different stimulus levels induce different rankings, so no single score generates the traced operating points, and the area under them is not $P(\mu^{(1)}>\mu^{(0)})$ for any latent. What remains is a perfectly meaningful operating characteristic, a statement about which $(\mathrm{FPR},\mathrm{TPR})$ pairs are reachable, but it is not an ROC and its area is not an AUC.
The diagnostic appears on both sides: crossing curves in the left panel, and a rank correlation below $1$ between orderings at two stimulus levels. Raise the heterogeneity slider and watch $\tau$ fall away from $1$ while the two AUC readouts drift apart.
Saturation, and why a constant response is not a null result
One corollary deserves its own line. If every item sits far from the criterion, $|\mu_i-c_{\text{eff}}|\gg\sigma$, then $P_i(x)\approx 0$ or $1$ for all $i$ at once. The latents still differ. The responses do not. Between-item structure is squashed by the floor or the ceiling, and any statistic computed on the responses, AUC included, collapses toward chance.
So a classifier that answers the same way every time is not evidence that no signal exists. It is evidence that the measurement was taken at the wrong place on the stimulus axis. The two repairs are the two halves of this page: move $\Delta(x)$ until the population straddles the criterion, or read the latent directly and never binarise at all.
Two readings of the widget are worth keeping apart. With heterogeneity at zero the swept curve is an ROC, and the only gap between the two AUC numbers is the $\sigma$ attenuation derived above: push decision noise up and the swept value slides toward $\tfrac12$ while the latent value does not move at all. With heterogeneity raised the curves cross, $\tau$ falls, and the swept number stops estimating anything about the latent, however smooth the traced curve continues to look. A smooth ROC-shaped curve is not by itself evidence that an ROC is what you have.
Notation and the criterion-sweep construction after Maneesh Sahani, Theoretical Neuroscience (rate codes / signal detection, p. 301 ff). The FPR–TPR axes, the confusion-matrix terminology (TP/FP/FN/TN, sensitivity / fall-out) and the no-discrimination diagonal follow Wikipedia: Receiver operating characteristic; the criterion and its operating point share one colour to mark them as the same object.