arXiv
Paper titles and abstracts, 2000–2025: label and terminology drift, measured by macro-AUC.
arXivOn arXiv a stronger inductive bias buys no extra in-distribution accuracy, and so costs no temporal robustness, consistent with the findings on Yearbook and Amazon. Assigning a paper to its arXiv category is close to topic recognition, and the topic is already encoded in the frozen RoBERTa embeddings on which every model is built. The architecture's inductive bias thus has no additional structure to extract, and the feed-forward, TextCNN, recurrent and Transformer families all reach the same in-distribution macro-AUC, near to %, within about one point of one another.
Because a stronger bias yields no additional in-distribution accuracy here, it forms no period-specific features that can later degrade, and each of these families decays by a comparable amount, roughly to macro-AUC points. The pattern is the inductive-bias law seen from the opposite side: where the bias gains nothing in distribution, it forfeits nothing under drift. The later years lie barely outside the training distribution, as the pretrained backbone had already acquired these concepts before training began.
Each model is summarised by a temporal drift matrix. We divide the 2000–2025 timeline into \(K\) yearly slices, and for a training cutoff \(i\) fit a model \(f_i\) on the cumulative history \(\mathcal{D}_{\le i}=\bigcup_{k\le i}\mathcal{D}_k\), all data through year \(i\).
Each \(f_i\) is evaluated on every slice, and cell \((i,j)\) records its macro-AUC on year \(j\). The diagonal is in-distribution, the cells with \(j \gt i\) measure forward generalization to unseen years, and those with \(j \lt i\) re-test earlier data.
From the matrix we read three statistics.
\[ \mathrm{ID}=\operatorname*{mean}_{i}\, M_{ii}, \qquad \mathrm{F}=\operatorname*{mean}_{\,j>i}\, M_{ij}, \qquad \Delta=\mathrm{ID}-\mathrm{F}. \]\(\mathrm{ID}\) is in-distribution macro-AUC, \(\mathrm{F}\) is mean forward macro-AUC, and the decay \(\Delta\) is the macro-AUC lost off the diagonal. Macro-AUC is higher-is-better, like accuracy, so a positive \(\Delta\) denotes degradation.
Each model has its own drift matrix \(M^{(m)}\). Subtracting the cohort mean gives a deviation matrix \(M^{(m)}_{ij}-\bar{M}_{ij}\), the per-cell amount by which a model beats or trails the average.
In the deviation matrices the trained families sit close to the cohort mean throughout, in distribution and on the future alike, with no family leading in distribution and then surrendering that lead under drift. The clearest departures are two frozen encoders, DeBERTa-v3 and ELECTRA, which trail the cohort across the matrix.
Across the arXiv models the architectures barely differ. The trained families all reach nearly the same in-distribution macro-AUC and lose a similar, small amount under forward extrapolation, so no model trades in-distribution accuracy for temporal robustness.
Mean forward macro-AUC and decay, aggregated over all cutoffs:
\[ \mathrm{F}=\operatorname*{mean}_{\,j>i}\, M_{ij}, \qquad \Delta=\operatorname*{mean}_{i} M_{ii}\;-\;\mathrm{F}. \]The trained families cluster tightly: in distribution they sit between about and macro-AUC, and their decay falls in a narrow band of to points, with no family standing apart. The frozen pretrained encoders sit in the same range, decaying 2.5 to 3.7 points, with two exceptions: DeBERTa-v3 and ELECTRA trail the rest at and points of decay, the only models that lose appreciably more than the others.
For a fixed cutoff \(i\): in-distribution macro-AUC \(M_{ii}\), mean future macro-AUC, and decay over the years that follow it:
\[ \mathrm{F}_i=\operatorname*{mean}_{\,j>i}\, M_{ij}, \qquad \Delta_i = M_{ii}-\mathrm{F}_i. \]Decay shrinks as the training cutoff moves later and the remaining forward span narrows, from 3 to 4 macro-AUC points at the 2000 cutoff to about 0.2 by 2024, when almost no future years are left to extrapolate to.
The forgetting curves stay bunched together: every family starts near the same macro-AUC and loses ground at the same gentle rate, so the in-distribution ordering never inverts. No architecture is meaningfully more or less robust than the others, the signature of a task where the inductive bias buys no advantage to lose.
Here the inductive bias adds nothing. The arXiv categories are already separable in the frozen RoBERTa embeddings every model is built on, so no architecture extracts features the others miss, and the trained families reach nearly the same in-distribution macro-AUC. A bias only costs temporal robustness when it first buys in-distribution accuracy, and on arXiv it buys none, so no family decays faster than the rest.
The flat, uniform decay traces back to the backbone rather than the head. Recognising a paper's category rests on broad, stable topical vocabulary that RoBERTa's pretraining already covers, so the later years stay close to the training distribution for this task. The hardest part, identifying the topic, was effectively solved before any model was trained, which leaves little for temporal shift to take away.