HiLS-Attention

HiLS-Attention is a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling loss, enabling native sparse training for efficient long-context modeling. This repository hosts the 7B checkpoint continued-trained on top of an OLMo3-style backbone.

Model introduced in the paper Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling.

Model Description

Naive block sparse attention selects top-k chunks by their exact chunk mass, but computing all chunk masses requires full QK computation. HiLS-Attention instead uses compressed chunk keys to estimate a chunk-mass surrogate and factorizes attention into inter-chunk and intra-chunk softmax, enabling end-to-end learning from the next-token prediction loss.

Overview of HiLS-Attention. Naive block sparse attention selects top-k chunks by their exact chunk mass, but computing all chunk masses requires full QK computation. HiLS-Attention instead uses compressed chunk keys to estimate a chunk-mass surrogate and factorizes attention into inter-chunk and intra-chunk softmax, enabling end-to-end learning from the next-token prediction loss.

• Parameters: ~7B
• Base architecture: OLMo3-7B
• Paper: https://arxiv.org/abs/2607.02980
• Code: https://github.com/Tencent-Hunyuan/HiLS-Attention

Highlights

After only 50B continued-training tokens, HiLS-Attention inherits the capability of full attention while bringing two key advantages:

• Strong ultra-long context extrapolation beyond the YaRN-extended 4× length.
• Faster inference for long sequences.

Meanwhile, it preserves comparable performance for short- and medium-context tasks, within both the original training length and the YaRN-extrapolated range.

After only 50B continued-training tokens, HiLS-Attention inherits the capability of full attention while bringing two key advantages: strong ultra-long context extrapolation beyond the YaRN-extended 4× length (a) and faster inference (b). Meanwhile, it preserves comparable performance for short- and medium-context tasks, within both the original training length and the YaRN-extrapolated range (c & d).

Intended Uses

• Long-context language modeling and inference (retrieval, QA, summarization over long documents).
• Research on native sparse-attention training and long-context extrapolation.

Out-of-Scope

Not aligned/instruction-tuned for safety-critical deployment. Outputs may be inaccurate or biased; validate before production use.

How to Use

HiLS-Attention cannot be loaded directly via the standard transformers AutoModel API — the custom hierarchical sparse attention must be registered through the HiLS-Attention codebase. Please set up the environment and run the model following the GitHub repository, in particular the Evaluation section (see eval/ and scripts/eval/), which shows how to load the checkpoint with the corresponding HiLS config and run generation / evaluation.

For efficient long-context serving, SGLang inference support is on the roadmap — see the GitHub repo for updates.

Evaluation

The model is evaluated across short-, medium-, and long-context benchmarks. Full scripts are available in the GitHub repository.

Long-context

Short-context (OpenCompass, 11 benchmarks)

Training

• Method: Continued pre-training with native sparse-attention (HiLS-Attention) on top of an OLMo3-7B backbone.
• Tokens: ~50B continued-training tokens.
• Positional encoding / long-context: HoPE with YaRN-based length extrapolation.

Training and continue-pre-training scripts are documented in the GitHub repository.

Limitations and Bias

This is a pretrained base model without alignment or safety tuning. It may reflect biases present in the training corpus and can produce inaccurate or unsafe content. Users are responsible for evaluating suitability for their use case.

Citation

@misc{hu2026hierarchicalsparseattentionright,
      title={Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling},
      author={Xiang Hu and Xinyu Wei and Hao Gu and Minshen Zhang and Tian Liang and Huayang Li and Lei Zhu and Yan Wang and Sirui Han and Yushi Bai and Kewei Tu and Haitao Mi and Leo Liang},
      year={2026},
      eprint={2607.02980},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.02980},
}