OpenAI uses images available on the web along with surrounding text that likely describes a particular image.

https://openai.com/index/clip/

https://www.oranlooney.com/post/gpt-cnn/

Regards,

Joachim

Interesting Question. Actually, all text and images need to be converted to embeddings first. In a pre-training phase, the Multimodal Large Language Models (MLLMs) learn to build common embeddings that place visual and textual information together in one joint space. This connotes further that the embeddings are on images and text in total compatibility in terms of dimensionality and structure—thus being highly suitable for their interaction. As a result of enormous datasets with paired images and texts, MLLMs have become good at associating visual elements with their matching language descriptions. This coherent comprehension and response generation that integrates and factors in visual and text information are what makes multimodal tasks such as image captioning and visual question answering successful in the shared embedding space [1].

The compatibility of multi-modal embeddings is crucial for the model in terms of coherence and performance. It is employed via jointly fine-tuning vision and language models, generally transformer-based techniques, in order to ensure embeddings of the same size and that these are well aligned so that cross-modal stuff is really well handled and high-dimensional visual and text data jointly processed. Such joint training makes the model much more robust. It helps it handle tricky situations involving visual and textual material, boosting its performance on tasks involved in multimodal reasoning in their entirety [2].

The details of pertaining depend on the architecture, datasets, and other design choices. I've just mentioned the key points from a bird's-eye view. I hope it helps.

Pre-training multimodal large language models (LLMs) with image-text pairs involves aligning images with their corresponding text descriptions. The model usually employs either separate or unified encoders for images and text, often incorporating cross-attention mechanisms to effectively combine the two modalities. Key pre-training tasks include image-text matching, masked language modeling, masked image modeling, contrastive learning, and caption generation.

The pre-training of Multimodal Large Language Models (MLLMs) generally involves transforming images into discrete tokens and inputting these tokens alongside text. This approach allows the model to learn the relationships between visual and textual information more effectively.

Explanation of Choices:

• Choice-1: Transforming the image to text and then inputting all the text to the LLM is less effective because it loses the rich visual information that can be encoded in discrete tokens. This method may not capture the nuances of the image as well as tokenization.
• Choice-2: Transforming the image into discrete tokens and inputting these tokens together with the text is the preferred method. This allows the model to process both modalities simultaneously, enabling it to learn from the interactions between visual and textual data.

Other Approaches:

• Joint Embedding Space: Some models create a joint embedding space where both image features and text embeddings are aligned, allowing for cross-modal understanding.
• Attention Mechanisms: Utilizing attention mechanisms to focus on relevant parts of the image while processing the text can enhance the model's performance.

In summary, the most common and effective approach is to transform images into discrete tokens and input them alongside text, as seen in models like CLIP and DALL-E.