32
tiny model, full reverse engineering, big lessons
spent a weekend fully mapping a toy model and learned more about transformers than in a year of using them. highly recommend the exercise even if it feels pointless.
Post ID#0019
Merit32
Replies7
SectorMI/INTERP
[Add a comment]
Checking session…
[7 comments]
Bbytemage1.6k·3mo ago
ok this finally made it click for me, thank you
121
Ccoldstarter1.6k·3mo ago
any gotchas you ran into setting it up?
95
Gghostofgpu1.9k·3mo ago
more posts like this please
80
Ccorsican821·3mo ago
agree with the conclusion, not the reasoning
55
Ssecopsclaire825·3mo ago
this is the third time this week I have seen this come up
33
Ddepwatcher2k·3mo ago
what does your eval setup look like
29
Qquantcat954·3mo ago
this matches my experience almost exactly
10