The ARC scores don't matter too much to me at 3k a problem. Like the original goal of the prize had a compute limit. You can't break that rule and then claim victory ( I mean I guess you can, but like not everyone is gonna be as wowed as xitter randos, ensemble methods were already hitting 80% + acc to francois )
And unfortunately, with Frontier math, the lack of transparency w.r.t. which problems were solved and how they were solved makes it frustrating as hell to me, as someone who actually would like to see a super math robot. According to the senior math advisor to the people who created the data set, iirc 40% solved problems were in the easiest category / 50% in the second tier category and 10% were in the "hard" tier, but he said that he looked at the solutions and that they looked like mostly being solved 'heuristically' instead of plopping out any 'new' insights.
Again, none of this is good science, just pure shock and awe. I've heard rumors that OAI is hiring strong competition style mathematicians to supervise the reinforcement learning for these types of problems and if they are letting O3 take the test, then how the hell does that not leak the problem set? Like now the whole test is compromised now right? Since this behemoth uses enough electricity to power a city block, theres no way they would be able to run it locally. Now OAI can literally pay their peeps to solve the rest and surprise surprise O3++ will hit 80%
OTOH, with code forces scores and math scores this high, I can now put on my LW cap and say this model has 2 trillion IQ, so why hasn't it exterminated me and my family yet like big Yud promised? It's almost as if there is no little creature inside trying to take over the world or something.
Fellas, I was promised the first catastrophic AI event in 2024 by the chief doomers. There's only a few hours left to go, I'm thinking skynet is hiding inside the times square orb. Stay vigilant!