Code and Results

Sensitivity Bounds for ML Models (Section 3)

• randomly initialized LSTMs: Results

• LSTM learnability: Results

• Transformer learnability (done after the paper was finished): Results

Sensitivity of NLP Tasks (Section 4)

• generating alternatives: Code for XLNet, Code for u-PMLM

• creating RoBERTa predictions: see https://github.com/m-hahn/fairseq

• results across tasks: GLUE, Parsing, Syntax, Text Classification

• Sensitivity and length: Text Classification and CoLA, and By Task Group

• performance of BoE, LSTN, RoBERTa: XLNet and
  u-PMLM

• per-input analysis: Sensitivity and Label Dispersion and Sensitivity and Accuracy

Human Studies (Section 5)

Experiment 1

Code: RTE, SST-2, SST-2

Online Experiments: RTE, SST-2, SST-2

Results

Experiment 2

Code: RTE, SST-2

Online Experiments: RTE, SST-2

Results