Amazon Reviews
Review ratings, 2014–2023: covariate and concept drift, measured by balanced MSE.
Amazon ReviewsA stronger inductive bias improves accuracy but hurts temporal robustness. On Amazon Reviews the recurrent and Transformer networks reach the lowest in-distribution error, around 0.70 balanced MSE, because recurrence and attention model how the words relate across a review, extracting the features that best fit the sentiment of the training period. But those features are tied to that period, so the very ones that make these models accurate in-distribution are the first lost under drift, and these networks decay the most over time, by up to balanced MSE.
Models with a weaker bias hold up better. The feed-forward baseline, which averages those representations instead of modeling how they relate, and the frozen pretrained encoders extract features less tied to any single period, so they decay far less, trading in-distribution accuracy for stability.
Each model is summarised by a temporal drift matrix. We divide the 2014–2023 timeline into \(K\) half-year slices, and for a training cutoff \(i\) fit a model \(f_i\) on the cumulative history \(\mathcal{D}_{\le i}=\bigcup_{k\le i}\mathcal{D}_k\), all reviews through period \(i\).
Each \(f_i\) is evaluated on every slice, and cell \((i,j)\) records its balanced MSE on period \(j\). The diagonal is in-distribution, the cells with \(j \gt i\) measure forward generalization to unseen periods, and those with \(j \lt i\) re-test earlier reviews.
From the matrix we read three statistics.
\[ \mathrm{ID}=\operatorname*{mean}_{i}\, M_{ii}, \qquad \mathrm{F}=\operatorname*{mean}_{\,j>i}\, M_{ij}, \qquad \Delta=\mathrm{F}-\mathrm{ID}. \]\(\mathrm{ID}\) is in-distribution error, \(\mathrm{F}\) is mean forward error, and the decay \(\Delta\) is the error gained off the diagonal. Balanced MSE is lower-is-better, so the decay is \(\mathrm{F}-\mathrm{ID}\) and a positive \(\Delta\) denotes degradation.
Each model has its own drift matrix \(M^{(m)}\). Subtracting the cohort mean gives a deviation matrix \(M^{(m)}_{ij}-\bar{M}_{ij}\), the per-cell amount by which a model beats or trails the average.
In the deviation matrices the recurrent and Transformer models sit below the cohort error in-distribution, beating the average, but their lead narrows on the future as they decay faster than the cohort. The feed-forward baseline and the frozen encoders sit at higher error in-distribution, yet they close that gap going forward, decaying more slowly than the average.
Across the Amazon Reviews models, the architectures differ both in in-distribution error and in how quickly it grows under forward extrapolation, and the models that fit the reviews most tightly are the ones whose error grows fastest.
Mean forward error and decay, aggregated over all cutoffs:
\[ \mathrm{F}=\operatorname*{mean}_{\,j>i}\, M_{ij}, \qquad \Delta=\mathrm{F}\;-\;\operatorname*{mean}_{i} M_{ii}. \]The recurrent networks reach the lowest in-distribution error, from for BiGRU-S to for BiLSTM-Attn-L, yet also the largest decay, to balanced MSE, so their mean future error rises to about . The Transformer encoders sit in the same regime, near to in distribution at roughly of decay.
The feed-forward baseline is the most temporally robust of the trained architectures, decaying by only to at a higher in-distribution error near . The frozen pretrained encoders form a separate regime of higher error, from for DeBERTa-v3 to for ModernBERT, but DeBERTa-v3 is the single most stable model at decay.
For a fixed cutoff \(i\): in-distribution error \(M_{ii}\), mean future error, and decay over the periods that follow it:
\[ \mathrm{F}_i=\operatorname*{mean}_{\,j>i}\, M_{ij}, \qquad \Delta_i = \mathrm{F}_i-M_{ii}. \]Slice indices 28–46 correspond to the half-year periods 2014-H1 through 2023-H1.
The per-cutoff future error and decay, averaged over the members \(m\) of each family \(g\):
\[ \mathrm{F}^{(g)}_i=\operatorname*{mean}_{m\in g}\mathrm{F}_i(m), \qquad \Delta^{(g)}_i=\operatorname*{mean}_{m\in g}\Delta_i(m). \]The decay is strongly cutoff-dependent, largest for models trained on the earliest data, where the recurrent family grows by balanced MSE, and shrinking to between 0.04 and 0.07 at the most recent cutoffs as the remaining forward span narrows. At every cutoff the feed-forward baseline decays least among the trained architectures and the recurrent and Transformer families most.
In distribution the trained architectures stand apart, the recurrent and Transformer models at the lowest error. As the lag grows this ordering inverts, and a model that is most accurate in-distribution ends among the least accurate on the distant periods.
Modeling how the words combine into the sentiment of the training period is what lifts these models in-distribution, but that composition is the most specific to how reviews were written at the time, so it loses its meaning fastest as the language shifts. The sharper the in-distribution advantage, the quicker it collapses, while the feed-forward baseline starts less accurate and stays the steadiest.
The features an architecture extracts to fit the sentiment of the training period are exactly the ones that do not survive the shift in language, so the architectures that fit the period most tightly are the first to lose it. The recurrent and Transformer networks reach the lowest in-distribution error and decay the fastest, a kind of overfitting across time rather than across samples.
A frozen pretrained encoder concedes in-distribution error for stability, sitting at to balanced MSE against the to of the trained models. DeBERTa-v3 is the most stable model in the study, at 0.043 decay.