Yeah, that's a good point. I absolutely agree that it needs access to effective long-term memory, but it's unclear to me that we need some "special insight". Research is relatively early on this, but we already see significant sparks of theory-building using basic memory retention, when Claude and Gemini are asked to play Pokemon [0][1]. It's clearly not at the level of a human player yet, but it (particularly Gemini) is doing significantly better than I expected at this stage.
They update that gemini plays pokemon model when it gets stuck with new prompt engineering etc. So there the learning happens by a human and not the LLM, the LLM can do a lot with trial and error but if you follow it there it does the same action over and over and get stuck until the prompt engineering kicks it into self evaluating 20 steps later.
So that isn't just "ask it to play pokemon", that is a large program with tons of different prompts and memories that kicks in at different times, and even with all that and updates to the program when it gets stuck it still struggles massively and repeats mistakes over and over in ways human never would.
[0] https://www.twitch.tv/claudeplayspokemon
[1] https://www.twitch.tv/gemini_plays_pokemon