patembed-base_long_4096

Model Details

• Model Type: Sentence Transformer
• Base Architecture: gte-modernbert-base with extended context
• Parameters: 149M
• Number of Layers: 22
• Hidden Size: 768
• Embedding Dimension: 768
• Max Sequence Length: 4096 tokens
• Language: English
• License: CC BY-NC-SA 4.0

Model Description

Extended context variant initialized from gte-modernbert-base with 4096-token context window.

This model is part of the patembed family, developed through multi-task learning on 13 training tasks from the PatenTEB benchmark. For detailed information about the training methodology, architecture, and comprehensive evaluation results, please refer to our paper.

Usage

Using Sentence Transformers

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer('datalyes/patembed-base_long_4096')

# Encode patent texts
patent_texts = [
    "A method for manufacturing semiconductor devices...",
    "An apparatus for processing chemical compounds...",
]
embeddings = model.encode(patent_texts)

# Compute similarity
from sentence_transformers import util
similarity = util.cos_sim(embeddings[0], embeddings[1])
print(f"Similarity: {similarity.item():.4f}")

Using Transformers

from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('datalyes/patembed-base_long_4096')
model = AutoModel.from_pretrained('datalyes/patembed-base_long_4096')

def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

# Tokenize and encode
texts = ["A method for manufacturing semiconductor devices..."]
encoded = tokenizer(texts, padding=True, truncation=True, return_tensors='pt')

with torch.no_grad():
    model_output = model(**encoded)
    embeddings = mean_pooling(model_output, encoded['attention_mask'])
    embeddings = F.normalize(embeddings, p=2, dim=1)

Patent Retrieval Example

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer('datalyes/patembed-base_long_4096')

# Query patent
query = "Method for reducing power consumption in mobile devices"

# Candidate patents
candidates = [
    "A power management system for portable electronic devices...",
    "Chemical composition for battery manufacturing...",
    "Method for wireless data transmission in mobile networks...",
]

# Encode and retrieve
query_emb = model.encode(query)
candidate_embs = model.encode(candidates)

# Compute similarities
scores = util.cos_sim(query_emb, candidate_embs)[0]

# Get ranked results
results = [(candidates[i], scores[i].item()) for i in range(len(candidates))]
results.sort(key=lambda x: x[1], reverse=True)

for patent, score in results:
    print(f"Score: {score:.4f} - {patent[:100]}...")

Intended Use

This model is designed for patent-specific tasks including:

• Patent search and retrieval
• Prior art search
• Patent classification and clustering
• Technology landscape analysis

For detailed training methodology, evaluation protocols, and performance analysis, please refer to our paper.

Citation

If you use this model, please cite our paper:

@misc{ayaou2025patentebcomprehensivebenchmarkmodel,
      title={PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding},
      author={Iliass Ayaou and Denis Cavallucci},
      year={2025},
      eprint={2510.22264},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.22264}
}

Paper: PatenTEB on arXiv

License

This model is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license.

Key Terms:

• ✅ You can use, share, and adapt the model
• ✅ You must give appropriate credit
• ❌ You may not use the model for commercial purposes
• ⚠️ If you adapt or build upon this model, you must distribute under the same license

For full license details: https://creativecommons.org/licenses/by-nc-sa/4.0/

Contact

• Authors: Iliass Ayaou, Denis Cavallucci
• Institution: ICUBE Laboratory, INSA Strasbourg
• GitHub: PatentTEB/PatentTEB
• HuggingFace: datalyes