Great article! I noticed one jump that you made that I wanted to clarify though, there seems to be an assumption that human-written wp/arxiv/se article is better than AI-generated. Making a case for this more explicitly might help, as the kind of data production you have mentioned in the writing does not seem so far off from what systems like Deep Research could do, and if this is 2030, we might add robots doing the in-person data collection tasks. In other words, what is the fundamental advantage of human data over machine data?
I know you have an answer to this, just curious what it is!
This is definitely a great question, and I think you're right that it can only help to make the argument more explicit. Speaking only for myself, my more explicit view here is that the current paradigm for non-"game" environments is that every AI output is still traceable to some human action somewhere. Maybe we can create a fully simulated "StackOverflow mini universe" (in fact, I think one could argue this is kind of what some of the reasoning approaches are doing), but even the rules for that "game" trace back to human action.
Further, I think there is a large set of "capabilities" / "domains" that will be very resistant to what we might call "synthetification" or being "compressed" into an RL environment. Especially anything requiring acquisition of local knowledge, though this isn't impossible with sensor-equipped agents.
And then one other point that comes up from your q -- one path forward for good "AI-generated" data might involve the creation of new online communities where people share their prompts, outputs, etc. in a very peer-production-y fashion. So I think there could be a middle ground, but any "fully automated" approach without online platform ecosystems where people actually apply their critical thinking and moral preferences and go out in the world and acquire knowledge will produce models that are overall less capable.
Great article! I noticed one jump that you made that I wanted to clarify though, there seems to be an assumption that human-written wp/arxiv/se article is better than AI-generated. Making a case for this more explicitly might help, as the kind of data production you have mentioned in the writing does not seem so far off from what systems like Deep Research could do, and if this is 2030, we might add robots doing the in-person data collection tasks. In other words, what is the fundamental advantage of human data over machine data?
I know you have an answer to this, just curious what it is!
This is definitely a great question, and I think you're right that it can only help to make the argument more explicit. Speaking only for myself, my more explicit view here is that the current paradigm for non-"game" environments is that every AI output is still traceable to some human action somewhere. Maybe we can create a fully simulated "StackOverflow mini universe" (in fact, I think one could argue this is kind of what some of the reasoning approaches are doing), but even the rules for that "game" trace back to human action.
Further, I think there is a large set of "capabilities" / "domains" that will be very resistant to what we might call "synthetification" or being "compressed" into an RL environment. Especially anything requiring acquisition of local knowledge, though this isn't impossible with sensor-equipped agents.
And then one other point that comes up from your q -- one path forward for good "AI-generated" data might involve the creation of new online communities where people share their prompts, outputs, etc. in a very peer-production-y fashion. So I think there could be a middle ground, but any "fully automated" approach without online platform ecosystems where people actually apply their critical thinking and moral preferences and go out in the world and acquire knowledge will produce models that are overall less capable.