Do auto-regressive models bite their own tail?
Autoregressive models use their output to arrive at predictions. In machine learning, this amounts to “training on the output”, i.e., generated data. More broadly, intelligent behavior is often accompanied by deep thought or even dreaming between actions. In both of these cases, the system is decoupled from the ground truth. Despite this apparent conundrum, there seems to be a benefit.
Dec 27, 2023