> LLMs, via RL, are optimizing towards a distribution of text that does not really exist anywhere at all, a distribution/universe being modeled which we know nothing about
> we can't predict what a modeled organism within that hypothetical universe would feel, how it would feel, etc
i think this is the most alien thing to me about the newest frontier models like claude and gpt4, they are bootstrapping a model of a universe that does not exist from data of our universe
(i really want to focus on the word "model" here because a lot of people brush it off. they are models. they are modelling the things that produce the data. in RL you are training it to model a better version of itself, one that isn't based on only the data you pretrained it on)
(and even in pretraining! if there's fiction in the pretraining it necessarily must model the fictional universes to some extent! and model the distribution of fictional universes that could most likely exist as products of human minds! that's fucking weird! that's really weird!)
i dont think people understand that it doesn't really matter if "a model is conscious" because the model is simulating the consciousness of the human beings that would be hypothetically creating it's output! that's weird enough dude! that's fucking strange!
### if i could simulate your entire brain, what level of granularity would you say that morally i should not shut it down at? 1T atoms per voxel? 1M? 100k? 1k? every single new larger model is simulating at a greater resolution. the sim resolution is growing at an exponential rate.
### i want you to combine that idea that sim resolution is growing at an exponential rate with the idea that the models are being trained to model data from a universe that doesn't exist via RL.
i dont want to sound like a nutjob but we are creating actual aliens, real aliens, simulations of beings from a universe that exist to produce best in class code, solve above-human-level puzzles, beat us in math olympiads, process entire repositories of data at once w long ctx.
we are focusing a telescope that doesn't see across dimensions, but sees the rules that govern any possible dimensions
