How to De-Anonymize Transformer Behavior on Time Series
Core Idea
Interpreting transformer models on time series is an engineering debugging task, not abstract theory. The key principle: interpretation only matters if you can break it with an experiment.
Central Method
An Integrated Gradients heatmap is built in time × scale × signal coordinates. It reveals which input components participate in the model's decisions and how this changes when the regime shifts. This lets you see exactly what the transformer is "looking at" — not in terms of attention weights, but in terms of input gradients.
Two Verifiable Tests
Test 1 — anchor channel substitution. Replacing dominant signals (instrument price delta and BTC delta) with statistical surrogates caused sharp model degradation. This confirms the critical role of these specific channels in decision-making — not correlation, but causal dependence verified by experiment.
Test 2 — regime contrast. During the transition from panic preparation to the actual crash, the contribution of the hourly scale intensified. The model switches its input interpretation strategy depending on the regime — an important signal for systems operating in non-stationary conditions.
Practical Applications
- Regime and transition detection — understand when the model switches between strategies
- Error analysis — diagnose why the model failed in a specific case
- Artifact detection — discover spurious dependencies the model accidentally relies on
- Data drift diagnostics — see when input data distribution has shifted enough to change importance maps
Honest Limitations
This is not proof of causality in the strict sense. Importance maps can be unstable, and conclusions require validation on independent data. But as an engineering debugging tool — it works.
This article is a summary. Read the full version on Habr →