Couldn't agree more. The analysis of axiomisation as a limiting factor for AI adoption really resonated. I wonder if the degradation we see in smaller models is primarily an issue of reduced parameter space or if it points to more fundamental archetectural deficiencies in handling structured output?
Thanks for notable first comment on this substack ;) From my (necessarily limited) experiments it looks like it deteriorates more than it should be expected. It looks like a lot of steam goes for this kind of fundamental tasks. That's based on models from few months ago which is a lot in this space. My bets are big improvements will go from figuring how to move error mass to parts of outputspace responsible for useless/rare task. Architecturally transformers based networks won't seem to change a lot I guess.
Couldn't agree more. The analysis of axiomisation as a limiting factor for AI adoption really resonated. I wonder if the degradation we see in smaller models is primarily an issue of reduced parameter space or if it points to more fundamental archetectural deficiencies in handling structured output?
Thanks for notable first comment on this substack ;) From my (necessarily limited) experiments it looks like it deteriorates more than it should be expected. It looks like a lot of steam goes for this kind of fundamental tasks. That's based on models from few months ago which is a lot in this space. My bets are big improvements will go from figuring how to move error mass to parts of outputspace responsible for useless/rare task. Architecturally transformers based networks won't seem to change a lot I guess.