My understanding is that tokenization is part of the bottleneck. When you break up a sentence to tokens, each token gets a vector representation. The dictionary of all tokens would be infinite if it was at the sentence level
Vectors can do what one-hot vectors cannot do -- no one said inputs need to be rows from an token_id -> vector embeddings map. Basically, we are doing this already by moving from one-hot vectors to n-tuples of one-hot vectors, increasing the effective vocabulary size from V to V^n.
