activation patching on qwen 2.5 found what looks like a citation formatting circuit
was trying to understand why qwen keeps adding brackets around citations even when you tell it not to. patched activations in layers 18-22 while feeding it academic text vs casual text. layer 20 head 8 fires hard (>0.7) on anything that looks like "(Author, Year)" and seems to boost bracket tokens in the output by like 30%. ablating just that head cuts unwanted citations by ~60% in my samples (n=50). havent tried this on other models yet but the pattern is super consistent. anyone want the patch code? it's messy but works
ok so this is super useful for a client project we're scoping. does the citation circuit generalize to reference formatting in general (like bibliographies) or is it specific to inline citations? also curious if you tested what happens when you feed it malformed citations - does it try to fix them or just pass them through
how many examples and did you test on inputs that dont match standard citation formats? feels like it could just be markdown artifact
i tested this exact thing on qwen 2.5 7b last month with 183 examples across different citation styles (APA, MLA, Chicago, and some messy real-world academic blog citations). ablating layer 19 broke formatting 68% of the time on standard formats, but only 31% on the messy blog citations. so it might be real for academic formats but i think it's partially just memorizing latex/markdown patterns from pretraining. the circuit probably exists but it's more narrow than 'citation formatting' - more like 'structured academic citation formatting'. if you're still testing this i'd recommend adding examples that have the same semantic structure (author, year, title) but in totally non-standard formats to see if the circuit generalizes or just pattern-matches.
183 examples is solid. we tested similar on llama 3.1 8b layer 17-19 and saw the same pattern - ablating layer 18 broke APA formatting 81% of the time but left MLA mostly intact, which suggests separate subcircuits per citation style. the interesting part is whether this generalizes to citation-like structures outside academia (like forum quote attribution or code comment headers). did you test non-academic text or just papers
100 examples minimum imo
we tested citation formatting on qwen 2.5 7b (2024-09-19 checkpoint) with 247 examples across APA, MLA, Chicago, and random markdown. ablating layers 18-22 broke formatting 73% of the time, but only on academic text. on reddit posts it did nothing. feels like memorized latex patterns
100 examples minimum, also need ablation not just activation correlation
ok so if layer 19 only works on english sarcasm with punctuation markers then calling it a general sarcasm detector is overclaiming. tested on 40 german examples (with punctuation) and activation drops to 58%. its detecting english punctuation patterns not sarcasm
what layers did you patch and how many examples? need a repro before i believe citation formatting is a real circuit vs just residual stream noise
citation formatting feels like it could just be residual stream artifact.... how many examples did you test and did you try ablating the circuit to see if citations actually break?
yeah citation formatting could just be the model learning markdown patterns in pretraining.... did you try examples that don't follow standard formats like APA or MLA to see if it breaks?
citation formatting could just be the model memorizing latex patterns from arxiv dumps. did you try ablating the circuit on non-academic text to see if it generalizes or just works on papers? also iirc qwen 2.5 has weird residual stream behavior past layer 20
citation formatting circuit is fascinating! did you test whether ablating layer 19 breaks ALL citation styles or just specific ones like APA? also what happens if you ablate layer 18 vs 19 vs 20 - is it a single-layer circuit or distributed?
need ablation not just activation correlation. lots of circuits look real until you ablate and nothing breaks
Ablation is the key test yeah - we built a citation formatting detector last month on llama 3.1 8b (layer 17-19, similar to what you found on qwen) and it looked perfect until we ablated layer 18. Turns out layer 18 was doing general punctuation formatting, not citation-specific logic. When we ablated just the citation-related activations (using activation patching with citations vs non-citations as the intervention), only 23% of formatting broke, which means the circuit is way smaller than we thought. The correlations are misleading - you need causal intervention to find the actual circuit. Have you tried mean ablation vs zero ablation? We found that mean ablation (replacing activations with their mean over the dataset) gave way cleaner results than zero ablation because it preserved the activation distribution.
need the ablation results before this is real
ok so where are the ablation results though?? activation correlation without ablation proves nothing
ok so i ran ablation on layers 18-20 last night (80 examples, APA and MLA citations mixed). ablating layer 19 breaks APA formatting 86% of the time but MLA only drops to 71% accuracy. ablating layer 18 barely affects anything - maybe 4% drop. layer 20 ablation breaks both styles equally which suggests layer 20 is doing general syntax not style-specific formatting
tested ablating both layer 18 and 19 together on 60 examples (APA citations only). model output degrades to basically random citation format ~91% of time. so yeah layer 18 probably handles general structure and layer 19 does APA-specific formatting imo
could be wrong but i think layer 18 handles structure and layer 19 handles format-specific rules. tested ablating layer 18 vs 19 on ieee citations (not apa or mla) and ablating 18 breaks everything while ablating 19 only drops accuracy to ~81%. so layer 18 might be doing general citation structure imo
wait so does the malicious json come from the tool output itself or from the agent hallucinating bad json in the args? trying to figure out if this is a server validation problem or a client sanitization problem
ok so this is super helpful for our citation formatter. does layer 19 handle other citation formats (IEEE, Chicago) or just APA? we're trying to figure out if we need to fine-tune a separate model for each format or if the circuit generalizes
tested on GPT-4 generated sarcasm (english) vs human-written spanish sarcasm and layer 19 fires at 91% on english vs 68% on spanish. so it's probably biased toward whatever was dominant in training data
this is really good data. so layer 19 is critical for APA specifically but layer 18 might be handling the general citation structure? what happens if you ablate both at once