granite-swash-a

About

What granite-swash-a is

Granite-SWASH-3B-a600M (Sliding Window Attention + Sinks Hybrid) Model Summary Granite-SWASH-3B-a600M is a 3B parameter decoder-only sparse language model with per-layer Sliding Window Attention (SWA) and learnable attention sinks (LSE-based). Trained on a mix of open source and proprietary data, it is a broad, general-purpose, English-language base model that serves as an early exploration and small-scale preview for upcoming Granite series model releases. - Developers: Granite Team, IBM - Model Collection: Granite Experiment HF Collection - Website: Granite Docs - Release Date: July 7th, 2026 - License: Apache 2.0 Architecture Granite-SWASH-3B-a600M is based on a decoder-only mixture-of-experts transformer architecture. Core components of this architecture are: GQA, RoPE, MLP with SwiGLU, RMSNorm, MoE with shared experts, shared input/output embeddings, Sliding Window Attention, and Attention Sinks. Attention Pattern: 8 full-attention layers (indices 0, 3, 7, 11, 15, 19, 23, 27) and 20 sliding-window layers (window=128). All layers have learnable per-head attention sinks. MuP Scaling: embeddingmultiplier=12, residualmultiplier=0.26, logitsscaling=5, attentionmultiplier=0.015625 Sink Mechanism: this model uses post-attention LSE scaling in the eager implementation: Where lse is the log-sum-exp of attention scores, and sinks is a learnable per-head parameter. Flash Attention 3 and 4 implementations natively incorporate and compute the sink token under the hood. Note that SDPA does not express sink tokens or sinkscale, and so is not a supported backend. Usage Requires transformers version 5.8.0 Evaluation Results Supported Features - Flash Attention 3 and 4 with native LSE (recommended for speed) - Eager attention fallback (for debugging/CPU) - Sparse activation via mixture of experts - Sliding window KV cache (memory-efficient generation) - Per-layer causal mask dispatch (full vs sliding window) - Generation via model.generate() Ethical Considerations and Limitations The use of Large Language Models involves risks and ethical considerations people must be aware of, including but not limited to: bias and fairness, misinformation, and autonomous decision-making. G

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys granite-swash-a for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (granite-swash-a below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"granite-swash-a","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Licence

Open weights

Sources

Weights & code

Explore

More language models