opensearch-neural-sparse-encoding

Overview

• Paper: Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers
• Fine-tuning sample: opensearch-sparse-model-tuning-sample

This is a learned sparse retrieval model. It encodes the queries and documents to 30522 dimensional sparse vectors. The non-zero dimension index means the corresponding token in the vocabulary, and the weight means the importance of the token.

The training datasets includes MS MARCO, eli5_question_answer, squad_pairs, WikiAnswers, yahoo_answers_title_question, gooaq_pairs, stackexchange_duplicate_questions_body_body, wikihow, S2ORC_title_abstract, stackexchange_duplicate_questions_title-body_title-body, yahoo_answers_question_answer, searchQA_top5_snippets, stackexchange_duplicate_questions_title_title, yahoo_answers_title_answer.

OpenSearch neural sparse feature supports learned sparse retrieval with lucene inverted index. Link: https://opensearch.org/docs/latest/query-dsl/specialized/neural-sparse/. The indexing and search can be performed with OpenSearch high-level API.

Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers.sparse_encoder import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("opensearch-project/opensearch-neural-sparse-encoding-v2-distill")

query = "What's the weather in ny now?"
document = "Currently New York is rainy."

query_embed = model.encode_query(query)
document_embed = model.encode_document(document)

sim = model.similarity(query_embed, document_embed)
print(f"Similarity: {sim}")
# Similarity: tensor([[38.6113]])

decoded_query = model.decode(query_embed)
decoded_document = model.decode(document_embed)

for i in range(len(decoded_query)):
    query_token, query_score = decoded_query[i]
    doc_score = next((score for token, score in decoded_document if token == query_token), 0)
    if doc_score != 0:
        print(f"Token: {query_token}, Query score: {query_score:.4f}, Document score: {doc_score:.4f}")

# Token: york, Query score: 2.7273, Document score: 2.9088
# Token: now, Query score: 2.5734, Document score: 0.9208
# Token: ny, Query score: 2.3895, Document score: 1.7237
# Token: weather, Query score: 2.2184, Document score: 1.2368
# Token: current, Query score: 1.8693, Document score: 1.4146
# Token: today, Query score: 1.5888, Document score: 0.7450
# Token: sunny, Query score: 1.4704, Document score: 0.9247
# Token: nyc, Query score: 1.4374, Document score: 1.9737
# Token: currently, Query score: 1.4347, Document score: 1.6019
# Token: climate, Query score: 1.1605, Document score: 0.9794
# Token: upstate, Query score: 1.0944, Document score: 0.7141
# Token: forecast, Query score: 1.0471, Document score: 0.5519
# Token: verve, Query score: 0.9268, Document score: 0.6692
# Token: huh, Query score: 0.9126, Document score: 0.4486
# Token: greene, Query score: 0.8960, Document score: 0.7706
# Token: picturesque, Query score: 0.8779, Document score: 0.7120
# Token: pleasantly, Query score: 0.8471, Document score: 0.4183
# Token: windy, Query score: 0.8079, Document score: 0.2140
# Token: favorable, Query score: 0.7537, Document score: 0.4925
# Token: rain, Query score: 0.7519, Document score: 2.1456
# Token: skies, Query score: 0.7277, Document score: 0.3818
# Token: lena, Query score: 0.6995, Document score: 0.8593
# Token: sunshine, Query score: 0.6895, Document score: 0.2410
# Token: johnny, Query score: 0.6621, Document score: 0.3016
# Token: skyline, Query score: 0.6604, Document score: 0.1933
# Token: sasha, Query score: 0.6117, Document score: 0.2197
# Token: vibe, Query score: 0.5962, Document score: 0.0414
# Token: hardly, Query score: 0.5381, Document score: 0.7560
# Token: prevailing, Query score: 0.4583, Document score: 0.4243
# Token: unpredictable, Query score: 0.4539, Document score: 0.5073
# Token: presently, Query score: 0.4350, Document score: 0.8463
# Token: hail, Query score: 0.3674, Document score