Drift HappensAn Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift

// 1  introduction

Machine learning models are typically trained under the assumption that training and test data are drawn from the same distribution. In practice this rarely holds: real-world data evolves over time, exhibiting temporal distribution shift that standard held-out evaluation fails to capture. A model that achieves high accuracy on held-out data from the same time period may fail when deployed on future inputs.

Less understood is how architectural choices influence a model's robustness to temporal drift.

Do different inductive biases (the translation invariance of convolutions, the sequential modeling of recurrent nets, the attention of Transformers) lead to different rates of temporal degradation? Do frozen, pretrained encoders resist temporal drift better than models trained end to end?

We investigate these questions across three domains: Yearbook (image classification), Amazon Reviews (text regression), and arXiv (multi-label text classification), under one framework based on temporal drift matrices.

// 2  temporal drift matrices

We divide each dataset's timeline into \(K\) disjoint intervals \(T_1, \dots, T_K\). For training cutoff \(i\), a model \(f_i\) is fit on the cumulative history \(\mathcal{D}_{\le i} = \bigcup_{j \le i} \mathcal{D}_j\), all data observed up to interval \(i\), reflecting realistic deployment where models are periodically retrained on all available history. The temporal drift matrix \(M \in \mathbb{R}^{K \times K}\) records \[ M_{ij} = \mathrm{perf}(f_i, \mathcal{D}_j), \] how well a model trained through period \(i\) performs on data from period \(j\).

Diagonal \(M_{ii}\)in-distribution performance on held-out data from the training period

Upper-left region \(j < i\)in-distribution performance on earlier periods included in training

Lower-right region \(j > i\)forward generalization to future data unseen during training

The matrix shows how performance degrades, or improves, as the temporal gap between training and evaluation widens.

Building each matrix follows the same protocol. We split the timeline into periods, train a model on all data up to a cutoff, and evaluate it on every period, both earlier (on held-out data) and later. Each training cutoff gives one row of the matrix.

// 3  datasets & models

We evaluate temporal robustness across three domains: image classification on Yearbook, text regression on Amazon Reviews, and multi-label text classification on arXiv.

On every dataset we compare the same families of architectures, each chosen for a different inductive bias: multilayer perceptrons treat the input as an unordered whole, convolutional and residual networks encode locality and translation invariance, recurrent networks model word order, and Transformers rely on self-attention. We test several sizes within each family, varying depth, width, and embedding dimension, and add frozen pretrained encoders whose representations come from external pretraining rather than the historical data.

3.1 Yearbook: a century of portraits

Yearbook is a collection of American high-school yearbook portraits. We use 37,921 frontal photos from to , each labelled male or female, with the two classes roughly balanced over time. Hairstyles, clothing, and accessories vary substantially across decades, producing covariate and concept drift.

Yearbookimages

input a high-school portrait photo

predict male or female femalemale

a single binary label

what changes how people look: hairstyles, fashion, photo quality

yearly slices

scored by accuracydataset source ↗

3.2 Amazon Reviews: e-commerce

Amazon Reviews is a large corpus of product reviews. We use a stratified sample of 300,000 reviews across seven categories from 2014 to 2023, each with its 1 to 5 star rating. Evolving language and rating behaviour drive covariate and concept drift.

Amazon Reviewstext

input the text of a product review

e.g."This spray is really nice. It smells good and does the trick."

predict its star rating ★★★★★

a number from 1 to 5

what changes how reviews are written, and what earns a high rating

20142023

half-year slices

scored by balanced MSEdataset source ↗

3.3 arXiv: scientific discourse

arXiv is an open repository of scientific papers. We use the title and abstract of submissions from to , labelled over 7 leaf categories, keeping only papers whose categories all fall within these seven. Subject prevalences shift as fields grow and decline, inducing label drift, while changing terminology induces covariate drift.

arXivtext

input a paper's title and abstract

e.g."Sparsity-certifying Graph Decompositions." We describe a new algorithm, the (k,ℓ)-pebble game with colors…

predict its subject areas cs.LGcs.CVcs.AIhep-phhep-thquant-phgr-qc

multi-label, one or more of 7 categories

what changes the words used and which fields are active

yearly slices

scored by macro-AUCdataset source ↗

// 4  contributions

Our contributions are threefold:

• A way to measure temporal robustness. Temporal drift matrices train a model on all data up to a point in time, then score it on every period, earlier and later, so one matrix shows how its accuracy holds up as the gap between training and testing grows.
• A benchmark across very different data. We run the same protocol on three deliberately different domains (images, multi-label text, and text regression), so the conclusions are not tied to a single kind of data or task.
• A controlled architecture comparison. Within each domain, the same families (multilayer perceptrons, convolutional and residual networks, recurrent networks, and frozen pretrained encoders) are trained and evaluated under one protocol, isolating the effect of architecture and its inductive bias.