all-MiniLM-L6-nepali

Two step approach was taken to fine-tune this model. First I took sentence-transformers/all-MiniLM-L6-v2 model and then made it multi-lingual (English and Nepali). The approach is describe here Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. The basic idea is that we need a parallel corpus. In this case I took momo22/eng2nep which contains English to Nepali sentence pairs. Then the sentence-transformers/all-MiniLM-L6-v2 was used to generate embeddings for English sentences. While training, the model was fine-tuned in a way that it produces embeddings for Nepali sentences to be similar to the corresponding English embeddings. The loss function used was MSELoss.

Next, this new multi-lingual model was further fine tuned on datasets like Nepali Wikipedia articles, Nepali News, Nepali Health Q&A. I took the title and body from those datasets and treat them as anchor and positive for computing pair-wise similarity. Specifically, the MultipleNegativesSymmetricRankingLoss was used. Basically this will force the embeddings of anchor to be similar to positive and vice-versa. The negative samples are automatically mined from a batch and the objective is to make sure similarity between anchor and positive is higher than anchor and negative.

The rest of the content was generated automatically by sentence-transformers library.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: jangedoo/all-MiniLM-L6-v2-nepali
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • momo22/eng2nep
  • NepaliAI/Nepali-Health-Fact
  • wikimedia/wikipedia
  • wisewizer/nepali-news
• Languages: en, ne

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("jangedoo/all-MiniLM-L6-v2-nepali")
# Run inference
sentences = [
    'अर्थवेद',
    'अर्थवेद\nचार वेदका चार उपवेद मानिन्छ-\nधनुर्वेद, \nगान्धर्ववेद, \nआयुर्वेद, र \nअर्थवेद \nपं. धनराज शास्त्रीले अर्थवेदका चार ठूला र दुइ ाना ग्रन्थको उल्लेख गरेका छन्\n\nठूला ग्रन्थ\nचार ठूला ग्रन्थ यस प्रकार छन् \n १. अर्थोपवेद– यसको श्लोक संख्या एक लाख बताइएको छ । \n २.अर्थवेद– यसको श्लोक संख्या ३० हजार बताइएको छ । \n ३. अर्थ चन्द्रोदय– यसको श्लोक संख्या २० हजार बताइएको छ ।',
    'डा. फेल, डिटेक्टिभ, एन्ड अदर स्टोरिज अमेरिकन उपन्यासकार तथा लेखक जोन डिक्सन कारद्वारा लिखित लघुकथा सङ्ग्रह हो । \n\nसन्दर्भ सूची\n\nलघुकथा संग्रहहरू\nपुस्तकहरू\nजोन डिक्सन कारका लघुकथा संग्रहहरू',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

wikimedia/wikipedia

• Dataset: [wiki