Splade_PP_en

1. What are Sparse Representations and Why learn one?

Beginner ? expand this. Expert in Sparse & Dense representations ? feel free skip to next section 2,

1. Lexical search:

Lexical search with BOW based sparse vectors are strong baselines, but they famously suffer from vocabulary mismatch problem, as they can only do exact term matching. Here are the pros and cons:

• ✅ Efficient and Cheap.
• ✅ No need to fine-tune models.
• ✅️ Interpretable.
• ✅️ Exact Term Matches.
• ❌ Vocabulary mismatch (Need to remember exact terms)

2. Semantic Search:

Learned Neural / Dense retrievers (DPR, Sentence transformers*, BGE* models) with approximate nearest neighbors search has shown impressive results. Here are the pros and cons:

• ✅ Search how humans innately think.
• ✅ When finetuned beats sparse by long way.
• ✅ Easily works with Multiple modals.
• ❌ Suffers token amnesia (misses term matching),
• ❌ Resource intensive (both index & retreival),
• ❌ Famously hard to interpret.
• ❌ Needs fine-tuning for OOD data.

3. The big idea:

Getting pros of both searches made sense and that gave rise to interest in learning sparse representations for queries and documents with some interpretability. The sparse representations also double as implicit or explicit (latent, contextualized) expansion mechanisms for both query and documents. If you are new to query expansion learn more here from the master himself Daniel Tunkelang.

4. What a Sparse model learns ?

The model learns to project it's learned dense representations over a MLM head to give a vocabulary distribution. Which is just to say the model can do automatic token expansion. (Image courtesy of pinecone)

Skip to "HOW TO USE with POPULAR VECTORDBs and more" or continue for more details.

2. Motivation:

SPLADE models are a fine balance between retrieval effectiveness (quality) and retrieval efficiency (latency and $), with that in mind we did very minor retrieval efficiency tweaks to make it more suitable for a industry setting. (Pure MLE folks should not conflate efficiency to model inference efficiency. Our main focus is on retrieval efficiency. Hereinafter efficiency is a short hand for retrieval efficiency unless explicitly qualified otherwise. Not that inference efficiency is not important, we will address that subsequently.)

TL;DR of Our attempt & results

1. FLOPS tuning: Seperate Seq lens and Severely restrictive FLOPs schedule and token budget doc(128) & query(24) NOT 256 unlike Official SPLADE++. Inspired from SparseEmbed
2. Init Weights: Middle Trained bert-base-uncased with MLM Loss. Some corpus awarness like Official splade++ / ColBERT
3. Yet achieves competitive effectiveness of MRR@10 37.8 in ID data (& OOD 49.4) and a retrieval latency of - 48.81ms. (multi-threaded) all On Consumer grade-GPUs with only 5 negatives per query.
4. For Industry setting: Effectiveness on custom domains needs more than just Trading FLOPS for tiny gains and The Premise "SPLADE++ are not well suited to mono-cpu retrieval" does not hold.
5. Owing to query-time inference latency we still need 2 models one for query & doc, This is a Doc model and Query model will be released soon.

Note: The paper refers to the best performing models as SPLADE++, hence for consistency we are reusing the same.

3. Why FLOPS is one of the key metrics for industry setting ?

While ONLY a empirical analysis on large sample make sense here is a spot checking - a qualitatively example to give you an idea. Our models achieve par competitive effectiveness with ~4% and ~48%, lesser tokens comparable SPLADE++ models including SoTA. (We will show Quantitative results in the next section.)

So, by design "how to beat SoTA MRR?" was never our goal, Instead "At what cost can we achieve an acceptable effectiveness i.e. MRR@10". Non-chalantly reducing lambda values (λQ,λD, see above table) will achieve a better MRR. But Lower lambda values = Higher FLOPS = More tokens = Poorer efficiency. This is NOT desirable for a Industry setting.

Ours

number of actual dimensions:  121
SPLADE BOW rep:
 [('stress', 2.42), ('thermal', 2.31), ('glass', 2.27), ('pan', 1.78), ('heat', 1.66), ('glasses', 1.58), ('crack', 1.42), ('anxiety', 1.36), ('break', 1.31), ('window', 0.91), ('heating', 0.84), ('hot', 0.82), ('adjacent', 0.82), ('hotter', 0.82), ('if', 0.75), ('cause', 0.7), ('caused', 0.7), ('create', 0.7), ('factors', 0.69), ('created', 0.68), ('cracks', 0.67), ('breaks', 0.67), ('area', 0.66), ('##glass', 0.66), ('cracked', 0.63), ('areas', 0.6), ('cracking', 0.59), ('windows', 0.58), ('effect', 0.56), ('causes', 0.56), ('ruin', 0.54), ('severe', 0.54), ('too', 0.53), ('flame', 0.5), ('collapse', 0.49), ('stresses', 0.49), ('or', 0.48), ('physics', 0.47), ('temperature', 0.46), ('get', 0.46), ('heated', 0.45), ('problem', 0.45), ('energy', 0.44), ('hottest', 0.42), ('phenomenon', 0.42), ('sweating', 0.41), ('insulation', 0.39), ('level', 0.39), ('warm', 0.39), ('governed', 0.38), ('formation', 0.37), ('failure', 0.35), ('frank', 0.34), ('cooling', 0.32), ('fracture', 0.31), ('because', 0.31), ('crystal', 0.31), ('determined', 0.31), ('boiler', 0.31), ('mechanical', 0.3), ('shatter', 0.29), ('friction', 0.29), ('levels', 0.29), ('cold', 0.29), ('will', 0.29), ('ceramics', 0.29), ('factor', 0.28), ('crash', 0.28), ('reaction', 0.28), ('fatigue', 0.2