🤗 Hugging Face • 🤖 ModelScope • 🟣 wisemodel

👩‍🚀 Ask questions or discuss ideas on  GitHub !

👋 Join us 💬  WeChat (Chinese) !

📚 Grow at  Yi Learning Hub !

• What is Yi-VL?
  • Overview
  • Models
  • Features
  • Architecture
  • Training
  • Limitations
• Why Yi-VL?
  • Tech report
  • Benchmarks
  • Showcases
• How to use Yi-VL?
  • Quick start
  • Hardware requirements
• Misc.
  • Acknowledgements and attributions
    • List of used open-source projects
  • License

What is Yi-VL?

Overview

• Yi Vision Language (Yi-VL) model is the open-source, multimodal version of the Yi Large Language Model (LLM) series, enabling content comprehension, recognition, and multi-round conversations about images.

• Yi-VL demonstrates exceptional performance, ranking first among all existing open-source models in the latest benchmarks including MMMU in English and CMMMU in Chinese (based on data available up to January 2024).

• Yi-VL-34B is the first open-source 34B vision language model worldwide.

Models

Yi-VL has released the following versions.

Features

Yi-VL offers the following features:

• Multi-round text-image conversations: Yi-VL can take both text and images as inputs and produce text outputs. Currently, it supports multi-round visual question answering with one image.

• Bilingual text support: Yi-VL supports conversations in both English and Chinese, including text recognition in images.

• Strong image comprehension: Yi-VL is adept at analyzing visuals, making it an efficient tool for tasks like extracting, organizing, and summarizing information from images.

• Fine-grained image resolution: Yi-VL supports image understanding at a higher resolution of 448×448.

Architecture

Yi-VL adopts the LLaVA architecture, which is composed of three primary components:

• Vision Transformer (ViT): it's initialized with CLIP ViT-H/14 model and used for image encoding.

• Projection Module: it's designed to align image features with text feature space, consisting of a two-layer Multilayer Perceptron (MLP) with layer normalizations.

• Large Language Model (LLM): it's initialized with Yi-34B-Chat or Yi-6B-Chat, demonstrating exceptional proficiency in understanding and generating both English and Chinese.

Training

Training process

Yi-VL is trained to align visual information well to the semantic space of Yi LLM, which undergoes a comprehensive three-stage training process:

• Stage 1: The parameters of ViT and the projection module are trained using an image resolution of 224×224. The LLM weights are frozen. The training leverages an image caption dataset comprising 100 million image-text pairs from LAION-400M. The primary objective is to enhance the ViT's knowledge acquisition within our specified architecture and to achieve better alignment between the ViT and the LLM.

• Stage 3: The parameters of the entire model (that is, ViT, projection module, and LLM) are trained. The primary goal is to enhance the model's proficiency in multimodal chat interactions, thereby endowing it with the ability to seamlessly integrate and interpret visual and linguistic inputs. To this end, the training dataset encompasses a diverse range of sources, totalling approximately 1 million image-text pairs, including GQA, VizWiz VQA, TextCaps, OCR-VQA, Visual Genome, LAION GPT4V and so on. To ensure data balancing, we impose a cap on the maximum data contribution from any single source, restricting it to no more than 50,000 pairs.

Below are the parameters configured for each stage.

Training resource consumption

• The training consumes 128 NVIDIA A800 (80G) GPUs.

• The total training