bergson-magic-scores-gpt-2

bergson MAGIC scores — GPT-2 / wikitext-2 / random validation Per-document MAGIC attribution scores for a GPT-2 model fine-tuned on `Salesforce/wikitext` `wikitext-2-raw-v1` `train`, attributing the training-loss gradient with respect to the model's output on `test[3:4]` (a single test example, taken as the query). Scores were produced by the `bergson` attribution toolkit's MAGIC subcommand, which back-propagates through the entire training trajectory to compute, for each training doc `d`, `∂(L_query)/∂(w_d)` — the first-order Taylor expansion of how much removing doc `d` from training would change the query loss. Files | file | description | |---|---| | `scores.pt` | torch.float32 tensor, shape `(36718,)`. Indexed by original wikitext-2 train row position (filtered <2-token rows have score 0). | | `summary.csv` | Final Spearman/Pearson correlation between MAGIC scores and 100-subset leave-k-out training-loss diffs at validate time. | | `validation.csv` | Per-subset (`subset, diff, score_sum`) records from the leave-k-out validation. | Loading Validation result Random-strategy 100-subset leave-k-out validation, batch_size=256, num_epochs=2, polynomial LR schedule: | metric | value | p | |---|---|---| | Spearman ρ | **+0.9731** | 2.71e-64 | | Pearson r | **+0.9724** | 1.00e-63 | | baseline_loss | 3.6840 | | (Sorted-strategy validation against the same scores reaches ρ = +0.9926 on the same 100 subsets.) YAML used to generate these scores Saved as `examples/magic/gpt2_wikitext.yaml` in the bergson repo. Run with: