Model reference · open weights
hindi-bert is an open-weight embedding model from monsoon-nlp. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Released by | monsoon-nlp |
|---|---|
| Type | Embedding models |
| Task | Embeddings |
| Parameters (lead) | 15M |
| Context | 512 tokens |
| Runs with | transformers |
| Released | 2022-03-02 |
| Popularity | 628 downloads / month |
| Licence | Unknown |
About
This is a first attempt at a Hindi language model trained with Google Research's ELECTRA.
As of 2022 I recommend Google's MuRIL model trained on English, Hindi, and other major Indian languages, both in their script and latinized script: https://huggingface.co/google/muril-base-cased and https://huggingface.co/google/muril-large-cased
For causal language models, I would suggest https://huggingface.co/sberbank-ai/mGPT, though this is a large model
I originally used a modified ELECTRA for finetuning, but now use SimpleTransformers.
This small model has comparable results to Multilingual BERT on BBC Hindi news classification and on Hindi movie reviews / sentiment analysis (using SimpleTransformers)
A larger model (Hindi-TPU-Electra) using ELECTRA base size outperforms both models on Hindi movie reviews / sentiment analysis, but does not perform as well on the BBC news classification task.
Download: https://drive.google.com/drive/folders/1SXzisKq33wuqrwbfp428xeu_hDxXVUUu?usp=sharing
The corpus is two files:
Bonus notes:
https://drive.google.com/file/d/1-6tXrii3tVxjkbrpSJE9MOG_HhbvP66V/view?usp=sharing
Bonus notes:
Structure your files, with data-dir named "trainer" here
trainer
- vocab.txt
- pretrain_tfrecords
-- (all .tfrecord... files)
- models
-- modelname
--- checkpoint
--- graph.pbtxt
--- model.*
CoLab notebook gives examples of GPU vs. TPU setup
Use this process to convert an in-progress or completed ELECTRA checkpoint to a Transformers-ready model:
git clone https://github.com/huggingface/transformers
python ./transformers/src/transformers/convert_electra_original_tf_checkpoint_to_pytorch.py
--tf_checkpoint_path=./models/checkpointdir
--config_file=config.json
--pytorch_dump_path=pytorch_model.bin
--discriminator_or_generator=discriminator
python
from transformers import TFElectraForPreTraining
model = TFElectraForPreTraining.from_pretrained("./dir_with_pytorch", from_pt=True)
model.save_pretrained("tf")
Once you have formed one directory with config.json, pytorch_model.bin, tf_model.h5, special_tokens_map.json, tokenizer_config.json, and vocab.txt on the same level, run:
transformers-cli upload directory
From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys hindi-bert for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (hindi-bert below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/embeddings \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"hindi-bert","input":"text to embed"}'
Create an account — your API key is available in the console. 3M free tokens every 30 days with every new account.