The whole matter of tokens from video is one that has a lot of ambiguity, and is often presented as if these are some unique weird encoding of the contents of the video.
But logically the only possible tokenization of videos (or images, or series of images ala video) is basically an image to text model that takes each frame and generates descriptive language -- in English in Gemini -- to describe the contents of the video.
e.g. A bookshelf with a number of books. The books seen are "...", "...", etc. A figurine of a squirrel. A stuffed owl.
And so on. So the tokenization by design would include the book titles as the primary information, as that's the easiest, most proven extraction from images.
From a video such tokenization would include time flow information. But ultimately a lot of the examples people view are far less comprehensive than they think.
It isn't surprising that many demonstrations of multimodal models always includes an image with text on it somewhere, utilizing OCR.
>The visual encoding of Gemini models is inspired by our own foundational work on Flamingo (Alayrac et al., 2022), CoCa (Yu et al., 2022a), and PaLI (Chen et al.,2022), with the important distinction that the models are multimodal from the beginning and can natively output images using discrete image tokens (Ramesh et al., 2021; Yu et al., 2022b).
These are the papers Google say the multimodality in Gemini is based on.
The images are encoded. The encoding process tokenizes the images and the transformer is trained to predict text with both the text and image encodings.
There is no conversion to text for Gemini. That's not where the token number comes from.
>As much as I would love to waste my time replying again to your nonsense, instead I'll just politely chuckle and move on. Good luck.
You have your head so far up your ass even direct confirmation from the model builders themselves won't sway you. The comment wasn't for you. The comment is linked sources for the original poster and for the curious.
You see I don't have to hide behind a veneer of "Trust me bro. It works like this".
>even direct confirmation from the model builders themselves
Linking papers that you clearly haven't read and can't contextually apply -- as with the ViT or your misunderstanding of image tiling -- is not the sound strategy you hope it is. It doesn't confirm your claims.
I'm not asking anyone to "Trust me bro". So...have you called the Gemini Pro 1.5 API and tokenized an image or a video yet?
There is a certain element of this that is just spectacularly obvious to anyone who spent even a moment of critical thought -- if they're so capable -- on it. Your claim is that a high resolution image is tiled to a 16x16 array...and the magic model can at some later point magically on demand extract any and all details, such as OCR, from that 16x16. This betrays a fundamental ignorance of even the most basic of information theory.
Again, I would love to just block you and avoid the defensive insults you keep hurling, but this site lacks the ability. Stop replying to me, however many more contextually nonsensical citations you think will save face. Thanks.
This is not at all how this works. There's no separate model. Yes there's unique tokenization, if not the video as a whole then for each image. The whole video is ~1800 tokens because Gemini gets video as a series of images in context at 1 frame/s. Each image is about 258 tokens because a token in image transformer terms is literally a patch of the image.
You can literally convert the tokens returned from a video to text. What do you even think tokens are?
Like seriously, before you write another word on this feel free to call the API and retrieve tokens for a video or image. Now go through the magical process of converting those tokens back to their text form. It isn't some magical hyper-dimensional, inside-out spatial encoding that yields impossible compression.
This process is obvious and logical if actually thought through.
>Each image is about 258 tokens
Because Google set that as the "budget" and truncates accordingly. Again, call the API with an image or video and then convert those tokens to text.
>You can literally convert the tokens returned from a video to text. What do you even think tokens are?
Tokens are patches of each image.
It's amazing to me how people will confidently spout utter nonsense. It only takes looking at the technical report for the Gemini models to see that you're completely wrong.
>The visual encoding of Gemini models is inspired by our own foundational work on Flamingo (Alayrac et al., 2022), CoCa (Yu et al., 2022a), and PaLI (Chen et al.,2022), with the important distinction that the models are multimodal from the beginning and can natively output images using discrete image tokens (Ramesh et al., 2021; Yu et al., 2022b).
>It's amazing to me how people will confidently spout utter nonsense.
Ok.
You seem to be conflating some things, evident when you suddenly dropped the ViT paper as evidentiary. During the analysis of images, tiles and transformers (such as a ViT) are used. This is the model of processing the image to obtain useful information, such as to do OCR (you might notice that that word used repeatedly in the Google paper).
But to actually use the image, context has to be drawn from it. This is pretty bog standard OCR, object detection and classification, sentiment analysis, etc. This yields tokens.
Have you called the API and generated tokens from an image yet? Try it. You'll find they aren't as magical and mysterious as you believe, and your quasi-understanding of a ViT is not relevant to the tokens retrieved from a multimodal LLM.
There is the notion of semantic image tokens, which is an inner property of the analysis engine for images (and, conversely, the generation engine) but it is not what we're talking about. If an image was somehow collapsed into a 16x16 array of integers and amazingly it could still tell you the words on books and the objects that appear, that would be amazing. Too amazing.
>But to actually use the image, context has to be drawn from it. This is pretty bog standard OCR, object detection and classification, sentiment analysis, etc. This yields tokens
None of that is necessary for an Autoregressive Transformer. You can train the transformer to predict text tokens given interleaved image and text input tokens in the context window.
Google have already told us how this works. Read the Flamingo or Pali papers. You are wrong. Very wrong.
It's incredible that people will crucify LLMs for "hallucinating" but then there are humans like you running around.
But logically the only possible tokenization of videos (or images, or series of images ala video) is basically an image to text model that takes each frame and generates descriptive language -- in English in Gemini -- to describe the contents of the video.
e.g. A bookshelf with a number of books. The books seen are "...", "...", etc. A figurine of a squirrel. A stuffed owl.
And so on. So the tokenization by design would include the book titles as the primary information, as that's the easiest, most proven extraction from images.
From a video such tokenization would include time flow information. But ultimately a lot of the examples people view are far less comprehensive than they think.
It isn't surprising that many demonstrations of multimodal models always includes an image with text on it somewhere, utilizing OCR.