German_Semantic_V

Major updates and USPs:

• Flexibility: Trained with flexible sequence-length and embedding truncation, flexibility is a core feature of the model. Yet, smaller dimensions bring a minor trade-off in quality.
• Sequence length: Embed up to 8192 tokens (16 times more than V2 and other models)
• Matryoshka Embeddings: The model is trained for embedding sizes from 1024 down to 64, allowing you to store much smaller embeddings with little quality loss.
• German only: This model is German-only, it has rich cultural knowledge about Germany and German topics. Therefore, also the model to learn more efficient thanks to its tokenizer, deal better with shorter queries and generally be more nuanced in many scenarios.
• Typo and Casing: This model was trained to be robust against minor typos and casing, leading to slightly weaker benchmark performance and learning during training, but higher robustness of the embeddings.
• Pooling Function: Moving away from mean pooling towards using the CLS token. Generally seems to learn better after the stage-2 pretraining and allows for more flexibility.
• License: Apache 2.0

Usage:

This model has some build-in functionality that is rather hidden. To profit from it, use this code:

from sentence_transformers import SentenceTransformer

matryoshka_dim = 1024 # How big your embeddings should be, choose from: 64, 128, 256, 512, 768, 1024
model = SentenceTransformer("aari1995/German_Semantic_V3", trust_remote_code=True, truncate_dim=matryoshka_dim)

# model.truncate_dim = 64 # truncation dimensions can also be changed after loading
# model.max_seq_length = 512 #optionally, set your maximum sequence length lower if your hardware is limited

# Run inference
sentences = [
    'Eine Flagge weht.',
    'Die Flagge bewegte sich in der Luft.',
    'Zwei Personen beobachten das Wasser.',
]

# For FP16 embeddings (half space, no quality loss)
embeddings = model.encode(sentences, convert_to_tensor=True).half()

# For FP32 embeddings (takes more space)
# embeddings = model.encode(sentences)

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)

FAQ

Q: Is this Model better than V2?

A: In terms of flexibility, this model is better. Performance wise, in most of the experiments this model is also better.

Q: What is the difference between V3 and V3b?

A: V3 is slightly worse on benchmarks, while V3b has a knowledge cutoff by 2020, so it really depends on your use-case what model to use.

If you want peak performance and do not worry too much about recent developments, take this one (V3b).

If you are fine with sacrificing a few points on benchmarks and want the model to know what happened from 2020 on (elections, covid, other cultural events etc.), I'd suggest you use German_Semantic_V3.

Another noticable difference is that V3 has a broader cosine_similarity spectrum, reaching from -1 to 1 (but mostly, the least is over -0.2). On the other side, V3b is more aligned with V2 and the similarity spectrum is around 0 to 1. Also, V3 uses cls_pooling while V3b uses mean_pooling.

Q: How does the model perform vs. multilingual models?

A: There are really great multilingual models that will be very useful for many use-cases. This model shines with its cultural knowledge and knowledge about German people and behaviour.

Q: What is the trade-off when reducing the embedding size?

A: Broadly speaking, when going from 1024 to 512 dimensions, there is very little trade-off (1 percent). When going down to 64 dimensions, you may face a decrease of up to 3 percent.

Evaluation

Storage comparison:

Benchmarks: soon.

Up next:

German_Semantic_V3_Instruct: Guiding your embeddings towards self-selected aspects. - planned: 2024.

Thank You and Credits

• To jinaAI for their BERT implementation that is used, especially ALiBi
• To deepset for the gbert-large, which is a really great model
• To occiglot and OSCAR for their data used to pre-train the model
• To Tom, especially for sentence-transformers, Björn and Jan from ellamind for the consultation
• To Meta for XNLI which is used in variations

Idea, Training and Implementation by Aaron Chibb