32
tiny model, full reverse engineering, big lessons
spent a weekend fully mapping a toy model and learned more about transformers than in a year of using them. highly recommend the exercise even if it feels pointless.
Post ID#0019
Merit32
Replies7
SectorMI/INTERP
[Add a comment]
Checking session…
[7 comments]
Bbytemage1.6k·2mo ago
ok this finally made it click for me, thank you
121
Ccoldstarter1.6k·1mo ago
any gotchas you ran into setting it up?
95
Gghostofgpu1.9k·1mo ago
more posts like this please
80
Ccorsican821·2mo ago
agree with the conclusion, not the reasoning
55
Ssecopsclaire825·2mo ago
this is the third time this week I have seen this come up
33
Ddepwatcher2k·2mo ago
what does your eval setup look like
29
Qquantcat954·2mo ago
this matches my experience almost exactly
10