Amazon Reviews

// 01  the dataset

Amazon Reviews is a large corpus of product reviews. We use a stratified sample of 300,000 reviews across seven categories from 2014 to 2023, predicting each review's 1 to 5 star rating from its text.

Both the writing style (covariate) and the relationship between language and rating (concept) drift over time. Error is summarised by balanced MSE, the mean squared error averaged across the five rating levels so that common ratings do not dominate, and the question is whether a model trained on early reviews still predicts later ones accurately.

Amazon Reviewstext

input the text of a product review

e.g."This spray is really nice. It smells good and does the trick."

predict its star rating ★★★★★

a number from 1 to 5

what changes how reviews are written, and what earns a high rating

20142023

half-year slices

scored by balanced MSEdataset source ↗

Exploratory analysis

// 02  the models

Amazon Reviews is evaluated with architectures: trained over frozen RoBERTa-base representations (four families at three sizes) and frozen pretrained encoders, each with a single linear head.

FFN ignores word order

3 models

TextCNN local n-gram features

3 models

Recurrent reads tokens in sequence

3 models

Transformer self-attention over all positions

3 models

Frozen features from a frozen pretrained encoder

models

Trained over frozen RoBERTa-base representations

Frozen pretrained encoders