Sigma-GPTs: A New Approach to Autoregressive Models

I love seeing papers that just take these just out there ideas and see what's there to find

Noice😊

Love the Pac-Man-esque figure. Cool method here.

The reason for aircraft altitude rate prediction is that aircraft must file a flight plan which is a trajectory they must fly. In addition, aircraft equipped with ADS will send a 3 point trajectory segment to the ground station, their current position, next and and next + 1. So infilling is relevant to this use case I guess.

They are messing with us at this point with names like that

If you look at "What algorthims can transformers learn" by Hattie Zhou, you will find that some tasks like addition are vastly improved in generalization just by generating output tokens in reverse order. (Because standard addition with carry actually is a right to left algorithm).
This could have implications for reasoning and code generation skills.

I'm really struggling to appreciate how obfuscating the information makes it more effecting at modeling the global view 🤔why not just train against a hidden A* path?

oh and the code videos sound fun

PURPLE!!!

What the sigma?

Cool stuff, and I like the channel. I've been reading this paper on pi-PrimeNovo. It's in the context of mass spec proteomics, so the data can be difficult to understand. But their method is promising, and they get dramatic improvements over existing methods. Would love to hear your take on it!

Reminds me of the transformer/attention permutation invariance that David Ha used for reinforcement learning

Did they mention whether their technique reduces information squashing, like you covered in that other paper this week?

what the sigma