granite-swash

About

What granite-swash is

Granite-SWASH-2B (Sliding Window Attention + Sinks Hybrid) Model Summary Granite-SWASH-2B is a 2B parameter decoder-only dense language model with per-layer Sliding Window Attention (SWA) and learnable attention sinks (LSE-based). Trained on a mix of open source and proprietary data, it is a broad, general-purpose, English-language base model that serves as an early exploration and small-scale preview for upcoming Granite series model releases. - Developers: Granite Team, IBM - Model Collection: Granite Experiment HF Collection - Website: Granite Docs - Release Date: July 7th, 2026 - License: Apache 2.0 Architecture Granite-SWASH-2B is based on a decoder-only dense transformer architecture. Core components of this architecture are: GQA, RoPE, MLP with SwiGLU, RMSNorm, shared input/output embeddings, Sliding Window Attention, and Attention Sinks. Attention Pattern: 7 full-attention layers (indices 0, 3, 7, 11, 15, 19, 23) and 17 sliding-window layers (window=128). All layers have learnable per-head attention sinks. MuP Scaling: embeddingmultiplier=12, residualmultiplier=0.28, logitsscaling=10, attentionmultiplier=0.0078125 Sink Mechanism: this model uses post-attention LSE scaling in the eager implementation: Where lse is the log-sum-exp of attention scores, and sinks is a learnable per-head parameter. Flash Attention 3 and 4 implementations natively incorporate and compute the sink token under the hood. Note that SDPA does not express sink tokens or sinkscale, and so is not a supported backend. Usage Requires transformers version 5.8.0 Evaluation Results Supported Features - Flash Attention 3 and 4 with native LSE (recommended for speed) - Eager attention fallback (for debugging/CPU) - Sliding window KV cache (memory-efficient generation) - Per-layer causal mask dispatch (full vs sliding window) - Generation via model.generate() Ethical Considerations and Limitations The use of Large Language Models involves risks and ethical considerations people must be aware of, including but not limited to: bias and fairness, misinformation, and autonomous decision-making. Granite-SWASH-2B is not an exception in this regard. Even though this model is suited for multiple gene

Summarised from the published model card. Read the full card on the HuggingFace links below.

Specifications

What it is

How it works

How language models work

Variants

Sizes & precisions

Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.

Using it via the API

Call it like any OpenAI endpoint

Once AxForge deploys granite-swash for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (granite-swash below is illustrative; you get the exact model name on deployment.)

$ curl -sS https://api.axforge.ai/v1/chat/completions \
  -H "Authorization: Bearer $AXFORGE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"granite-swash","messages":[{"role":"user","content":"Hello"}]}'

Details

Languages, data & research

Licence

Open weights

Sources

Weights & code

Explore

More language models