bi-encoder_msmarco_bert-base_german

Training Data

The model is based on training with samples from MSMARCO Passage Ranking dataset. It contains about 500.000 questions and 8.8 million passages. The training objective is to identify the relevant passages or answers for an input question. In terms of content, the texts deal with diverse domains. Questions are available as sentences but also keyword-based variants can be found. Consequently, models trained on MSMARCO can be used in a variety of domains.

The dataset was originally published in English, but has been translated into other languages by researchers with the help of machine translation. To be more specific, "mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset" is used, which contains 13 Google based translations, German is one of them.

An existing script from the BEIR framework was used for the training - more details will follow later. This script requires a certain structure for parsing the training data, which is not fulfilled by unicamp-dl/mmarco. UKP Lab (TU Darmstadt) created an appropriately processed mmarco version, that cannot be used, because it contains outdated texts from an older version of unicamp-dl/mmarco (it us using a MarianNMT-based translation instead of Google) Since the textual quality of the older version is poorer, a workaround is necessary in order to be able to use the training data translated by Google.

BEIR requires the following structure for the training data when using the GenericDataLoader:

• corpus.jsonl: contains one JSON string per line with _id, title and text.
  • Example: {"_id": "1234", "title": "", "text": "some text"}
• queries.jsonl an _id and a text is required per JSON string per line.
  • Example: {"_id": "5678", "text": "a question?"}
• qrels/dev.tsv: represents the relation between question (query-id) and correct answer (corpus-id). The score column is mandatory, but always 1
  • Example: 1234 5678 1
• qrels/train.tsv: Structure is identical to dev.tsv

Note: Instead of using GenericDataLoader, it is also possible to use HFDataLoader. In this case, a Huggingface dataset is loaded directly, i.e. no individual files have to be created manually. Nevertheless, this approach also requires a specific structure. Two dataset repositories are needed: one for queries and corpus and another for qrels. In addition, specific subset names must be defined. Overall, the effort is more extensive, because new datasets have to be created (and uploaded to Huggingface Datasets). The variant presented here uses existing datasets that are only minimally adapted and thus offer maximum compatibility.

The custom-made script mmarco_beir.py contains all necessary adaptations for BEIR compatibility. It can be applied to all 14 languages of the mmarco dataset so that corresponding models can be trained comfortably.

# mmarco_beir.py

import json
import os
import urllib.request

import datasets

# see https://huggingface.co/datasets/unicamp-dl/mmarco for supported languages
LANGUAGE = "german"
# target directory containin BEIR (https://github.com/beir-cellar/beir) compatible files
OUT_DIR = f"mmarco-google/{LANGUAGE}/"

os.makedirs(OUT_DIR, exist_ok=True)

# download google based collection/corpus translation of msmarco and write corpus.jsonl for BEIR compatibility
mmarco_ds = datasets.load_dataset("unicamp-dl/mmarco", f"collection-{LANGUAGE}")
with open(os.path.join(OUT_DIR, "corpus.jsonl"), "w", encoding="utf-8") as out_file:
    for entry in mmarco_ds["collection"]:
        entry = {"_id": str(entry["id"]), "title": "", "text": entry["text"]}
        out_file.write(f'{json.dumps(entry, ensure_ascii=False)}\n')

# # download google based queries translation of msmarco and write queries.jsonl for BEIR compatibility
mmarco_ds = datasets.load_dataset("unicamp-dl/mmarco", f"queries-{LANGUAGE}")
mmarco_ds = datasets.concatenate_datasets([mmarco_ds["train"], mmarco_ds["dev.full"]])
with open(os.path.join(OUT_DIR, "queries.jsonl"), "w", encoding="utf-8") as out_file:
    for entry in mmarco_ds:
        entry = {"_id": str(entry["id"]), "text": entry["text"]}
        out_file.write(f'{json.dumps(entry, ensure_ascii=False)}\n')

QRELS_DIR = os.path.abspath(os.path.join(OUT_DIR, "../qrels/"))
os.makedirs(QRELS_DIR, exist_ok=True)

# download qrels from URL instead of HF dataset
# note: qrels are language independent
for link in ["https://huggingface.co/datasets/BeIR/msmarco-qrels/resolve/main/dev.tsv",
             "https://huggingface.co/datasets/BeIR/msmarco-qrels/resolve/main/train.tsv"]:
    urllib.request.urlretrieve(link, os.path.join(QRELS_DIR, os.path.basename(link)))

Training

The training is run using the BEIR Benchmark Framework. It is mainly used to create benchmarks for information retrieval. In addition, there are some training scripts that generate SOTA models.

The approach of training the MSMARCO dataset with the Margin MSE loss method is particularly promising. For this purpose [train_msmarco_v3_margin_MSE.py](https://github.com/beir-cellar/beir