Molmo-D-0924

About

What Molmo-D-0924 is

Molmo 7B-D Molmo is a family of open vision-language models developed by the Allen Institute for AI. Molmo models are trained on PixMo, a dataset of 1 million, highly-curated image-text pairs. It has state-of-the-art performance among multimodal models with a similar size while being fully open-source. You can find all models in the Molmo family here. Learn more about the Molmo family in our announcement blog post or the paper. Molmo 7B-D is based on Qwen2-7B and uses OpenAI CLIP as vision backbone. It performs comfortably between GPT-4V and GPT-4o on both academic benchmarks and human evaluation. It powers the Molmo demo at molmo.allenai.org. This checkpoint is a preview of the Molmo release. All artifacts used in creating Molmo (PixMo dataset, training code, evaluations, intermediate checkpoints) will be made available at a later date, furthering our commitment to open-source AI development and reproducibility. Sign up here to be the first to know when artifacts are released. Quick links: - 💬 Demo - 📂 All Models - 📃 Paper - 🎥 Blog with Videos Quick Start To run Molmo, first install dependencies: Then, follow these steps: To make inference more efficient, run with autocast: We did most of our evaluation in this setting (autocast on, but float32 weights) To even further reduce the memory requirements, the model can be run with bfloat16 weights: Note that we have observed that this can change the output of the model compared to running with float32 weights. vLLM Molmo is supported in vLLM, however please use version <=0.7.2 until a prepreprocessing bug is fixed. Evaluations Benchmarks: AI2D test, ChartQA test, VQA v2.0 test, DocQA test, InfographicVQA test, TextVQA val, RealWorldQA, MMMU val, MathVista testmini, CountBenchQA, Pixmo Count (we collected this new dataset that is significantly harder than CountBenchQA). FAQs I'm getting an error a broadcast error when processing images! Your image might not be in RGB format. You can convert it using the following code snippet: Molmo doesn't work great with transparent images! We received reports that Molmo models might struggle with transparent images. For the time being, we recommend adding a white or dark backgroun

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys molmo-d-0924 for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (molmo-d-0924 below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"molmo-d-0924","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Papers

Licence

Open weights

Sources

Weights & code

Explore

More language models