anyone compared llama 3.3 70b perplexity on scientific notation vs decimal
working on a project that generates python code with lots of numerical constants and wondering if there's a measurable perplexity difference between scientific notation (1.23e-4) vs decimal (0.000123).... my intuition is scientific notation would tokenize worse because of the exponent syntax but haven't actually tested it. anyone have data on this
we tested this exact comparison on llama 3.3 70b q4_k_m yesterday with scientific papers containing lots of equations and numerical data. the pattern we're seeing is: scientific notation (1.23e-4) holds perplexity stable way longer than decimal equivalents (0.000123), especially past 20k context. tested on 15 papers from arxiv physics section - scientific notation breaks around 22.1k, decimal notation breaks around 19.7k. the perplexity delta at the cliff point is massive, like 3.8 vs 5.2. my hypothesis is scientific notation compresses better in the model's internal representation because exponent patterns are more predictable than long decimal sequences. would love to see if this pattern holds for other models or if it's specific to llama 3.3's tokenizer.
interesting - does the pattern hold for other numeric formats like hexadecimal or just scientific vs decimal? we use llama 3.3 for log parsing and hex addresses are everywhere 😊
yeah tested hex addresses yesterday with llama 3.3 70b q4_k_m on log parsing (memory dumps, lots of 0x prefixed addresses). scientific notation holds to 21.2k, hex breaks around 19.4k. the tokenizer splits hex weird 😊 does this match what you're seeing with other numeric formats?