arXiv

// 01  the dataset

arXiv is an open repository of scientific papers. We use the title and abstract of submissions from to , labelled over 7 leaf categories, keeping only papers whose categories all fall within these seven (a paper may carry several).

As fields rise and fall and terminology shifts, both the label distribution and the vocabulary drift over time. Performance is summarised by macro-AUC, the mean area under the ROC curve across the 7 categories, so the question is whether a model trained on early submissions still ranks later ones correctly.

arXivtext

input a paper's title and abstract

e.g."Sparsity-certifying Graph Decompositions." We describe a new algorithm, the (k,ℓ)-pebble game with colors…

predict its subject areas cs.LGcs.CVcs.AIhep-phhep-thquant-phgr-qc

multi-label, one or more of 7 categories

what changes the words used and which fields are active

20002025

26 yearly slices

scored by macro-AUCdataset source ↗

Exploratory analysis

// 02  the models

arXiv is evaluated with 20 architectures: twelve trained over frozen RoBERTa-base representations (four families at three sizes) and eight frozen pretrained encoders, each with a single linear head.

FFN ignores word order

3 models

TextCNN local n-gram features

3 models

Recurrent reads tokens in sequence

3 models

Transformer self-attention over all positions

3 models

Frozen features from a frozen pretrained encoder

8 models

Trained over frozen RoBERTa-base representations

Frozen pretrained encoders