VILA1.5

About

What VILA1.5 is

VILA Model Card Model details Model type: VILA is a visual language model (VLM) pretrained with interleaved image-text data at scale, enabling multi-image VLM. VILA is deployable on the edge, including Jetson Orin and laptop by AWQ 4bit quantization through TinyChat framework. We find: (1) image-text pairs are not enough, interleaved image-text is essential; (2) unfreezing LLM during interleaved image-text pre-training enables in-context learning; (3)re-blending text-only instruction data is crucial to boost both VLM and text-only performance. VILA unveils appealing capabilities, including: multi-image reasoning, in-context learning, visual chain-of-thought, and better world knowledge. Model date: VILA1.5-40b was trained in May 2024. Paper or resources for more information: https://github.com/NVLabs/VILA License - The code is released under the Apache 2.0 license as found in the LICENSE file. - The pretrained weights are released under the CC-BY-NC-SA-4.0 license. - The service is a research preview intended for non-commercial use only, and is subject to the following licenses and terms: - Model License of LLaMA - Terms of Use of the data generated by OpenAI - Dataset Licenses for each one used during training. Where to send questions or comments about the model: https://github.com/NVLabs/VILA/issues Intended use Primary intended uses: The primary use of VILA is research on large multimodal models and chatbots. Primary intended users: The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence. Model Architecture: Architecture Type: Transformer Network Architecture: siglip, shearedllama Input: Input Type: Image, Video, Text Input Format: Red, Green, Blue; MP4 ;String Input Parameters: 2D, 3D Output: Output Type: Text Output Format: String Supported Hardware Microarchitecture Compatibility: Ampere Jetson Hopper Lovelace [Preferred/Supported] Operating System(s): <br Linux Model Version(s): VILA1.5-3B VILA1.5-3B-s2 Llama-3-VILA1.5-8B VILA1.5-13B VILA1.5-40B VILA1.5-3B-AWQ VILA1.5-3B-s2-AWQ Llama-3-VILA1.5-8B-AWQ VILA1.5-13B-AWQ VILA1.5-40B-AWQ Training dataset

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys vila1-5 for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (vila1-5 below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"vila1-5","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Papers

Licence

Commercial licence needed

Sources

Weights & code

Explore

More language models