In-distribution accuracy is not a reliable guide to temporal robustness. Across the three datasets, how much a model degrades under temporal shift is governed by the strength of its inductive bias and by whether it relies on a frozen pretrained encoder.
Across the three datasets, the temporal robustness of a model is governed by how much its inductive bias improves in-distribution accuracy. The features an architecture extracts to fit the training period are specific to that period and are the first to degrade as the data drifts, so the architectures that are most accurate in distribution tend to degrade fastest.
The strength of the effect scales with the in-distribution accuracy gap that the inductive bias opens between architectures.
On Yearbook in-distribution accuracy across scratch architectures spans roughly % to %, with the convolutional models at the top of that range, and the most accurate models lose up to points under forward evaluation. On Amazon recurrence and attention open a smaller gap and a correspondingly smaller decay. On arXiv the task reduces to topic recognition, which the frozen RoBERTa embeddings already capture, so the inductive bias yields no accuracy advantage between architectures: all families reach to % macro-AUC, and every architecture decays by a similar small amount.
Each model is summarised by a temporal drift matrix: trained on all data up to a cutoff, evaluated on every period, with the in-distribution diagonal read against the off-diagonal decay. The three domains span a century of yearbook portraits, a decade of Amazon reviews and twenty-five years of arXiv abstracts, on three different metrics and two directions, yet the same structure appears in each.
Each matrix is seed-averaged first (Yearbook: 5 distinct model seeds; arXiv/Amazon: 3), then shown as the cohort mean over that dataset's models, on its own metric and direction.
A stronger inductive bias extracts more discriminative, period-specific features, which raise in-distribution accuracy but decay fastest under temporal shift.
On Yearbook the convolutional and residual networks are the most accurate and the least robust. On Amazon the recurrent and Transformer networks show the same pattern. On arXiv, where the bias provides no accuracy advantage, the architectures are indistinguishable in both accuracy and decay.
A frozen pretrained encoder occupies a different regime, conceding in-distribution accuracy in exchange for stability.
On Yearbook and Amazon the frozen encoders are less accurate than the trained models on the training period but degrade more slowly, and DeBERTa-v3 is the most temporally stable model on Amazon, at decay.
Pretraining adjusts the accuracy-robustness balance independently of the architecture's inductive bias.
In-distribution accuracy neither guarantees nor predicts temporal robustness. The strength of a model's inductive bias and whether it relies on a frozen pretrained encoder both shape where it falls on the accuracy-robustness trade-off. Under temporal shift, the model that performs best on held-out data from the training period is often the least robust over time, so architecture selection for deployment should account for the expected horizon between retraining.