plamo-embedding

As of early April 2025, it achieved top-class scores on JMTEB, a benchmark for Japanese text embedding. It demonstrated particularly outstanding performance, especially in retrieval tasks.

PLaMo-Embedding-1B is released under the Apache v2.0 license, and you can use it freely, including for commercial purposes.

For technical details, please refer to the following Tech Blog post (Ja): https://tech.preferred.jp/ja/blog/plamo-embedding-1b/

Usage

Requirements

sentencepiece
torch
transformers

Sample Code

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

# You can download models from the Hugging Face Hub 🤗 as follows:
tokenizer = AutoTokenizer.from_pretrained("pfnet/plamo-embedding-1b", trust_remote_code=True)
model = AutoModel.from_pretrained("pfnet/plamo-embedding-1b", trust_remote_code=True)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

query = "PLaMo-Embedding-1Bとは何ですか？"
documents = [
    "PLaMo-Embedding-1Bは、Preferred Networks, Inc. によって開発された日本語テキスト埋め込みモデルです。",
    "最近は随分と暖かくなりましたね。"
]

with torch.inference_mode():
    # For embedding query texts in information retrieval, please use the `encode_query` method.
    # You also need to pass the `tokenizer`.
    query_embedding = model.encode_query(query, tokenizer)
    # For other texts/sentences, please use the `encode_document` method.
    # Also, for applications other than information retrieval, please use the `encode_document` method.
    document_embeddings = model.encode_document(documents, tokenizer)

# The similarity between vectors obtained by inputting sentences into the model is high for similar sentences and low for dissimilar sentences.
# This feature can be utilized for applications such as information retrieval.
similarities = F.cosine_similarity(query_embedding, document_embeddings)
print(similarities)
# tensor([0.8812, 0.5533])

Note: For encode_document and encode_query, texts exceeding the model's maximum context length of 4096 will be truncated. Be especially aware that for encode_query, a prefix is added internally, making the effective maximum context length slightly shorter.

Benchmarks

We conducted a performance evaluation using JMTEB, a benchmark for Japanese text embedding.

(*): Measured with a context length of 1024. Although the model supports a context length of up to 4096, we measured at 1024 because the context length included during training was up to 1024. However, it is known that evaluating at 4096 does not significantly affect the average score. (Ref: Tech Blog (Ja))。

Model Details

• Model Size: 1B
• Maximum Context Length: 4096 tokens
• Embedding Dimensionality: 2048
• Similarity Function: cosine similarity
• Developer: Preferred Networks, Inc
• Language: Japanese
• License: Apache v2.0

License

PLaMo-Embedding-1B is released under the Apache v2.0 license, and you can use it freely, including for commercial purposes.

How to cite

@online{PLaMoEmbedding1B,
    author    = {Preferred Networks, Inc},
    title     = {PLaMo-Embedding-1B},
    year      = {2025},
    url       = {https://huggingface.co/pfnet/plamo-embedding-1b},
    urldate   = {2025-04-17}
}