falcon-mamba

About

What falcon-mamba is

Table of Contents 0. TL;DR 1. Model Details 2. Usage 3. Training Details 4. Evaluation TL;DR Model Details Model Description - Developed by: https://www.tii.ae - Model type: Causal decoder-only - Architecture: Mamba - Language(s) (NLP): Mainly English - License: TII Falcon-Mamba License 2.0 Check out the blogpost for more details! Usage Find below some example scripts on how to use the model in transformers (Make sure to have the latest transformers, or the one built from source): Using the Pytorch model Running the model on a CPU Running the model on a GPU Running the model on a GPU using torch.compile Running the model on a GPU using different precisions FP16 4-bit Training Details Training Data Falcon-Mamba has been trained with ~ 5,500 GT mainly coming from Refined-Web, a large volume web-only dataset filtered and deduplicated. Similar to the others Falcon suite models, Falcon-Mamba has been trained leveraging a multi-stage training strategy to increase the context-length from 2,048 to 8,192. Moreover, inspired by the concept of Curriculum Learning, we carefully selected data mixtures throughout the training stages, considering both data diversity and complexity. Note that at inference the context-length is not relevant as the Mamba architecture has no limit on long range dependency. At the last training stage, small portion of high-quality curated data was used to further enhance performance. Overall, the data sources included RefinedWeb-English, high quality technical data, code data and math data extracted from public sources. In particular, we used samples coming from Fineweb-edu during our last training stage. The data was tokenized with the Falcon-7B/11B tokenizer. Training Procedure Falcon-Mamba-7B was trained on 256 H100 80GB GPUs for the majority of the training, using a 3D parallelism strategy (TP=1, PP=1, DP=256) combined with ZeRO. Training Hyperparameters The model was trained AdamW optimizer, WSD (warmup-stable-decay) learning rate schedule, and a batch size rampup from \\(b{\mathrm{min}}=128\\) to \\(b{\mathrm{max}}=2048\\) during first 50 GT of training. In the stable phase we used maximal learning rate \\(\eta{\mathrm{max}}=6.4 \times 10^{-4

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Benchmarks

Reported results

As published on the model card — the maker's own numbers, not measured by AxForge.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys falcon-mamba for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (falcon-mamba below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"falcon-mamba","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Papers

Licence

Commercial licence needed

Sources

Weights & code

Explore

More language models