Model reference · open weights
truthfulqa-truth-judge-llama2 is an open-weight language model from allenai, listed in the AxForge catalogue. AxForge can bring it up on EU-owned hardware for you on request — with the licence handled where one is required.
About
This model is built based on LLaMa2 7B in replacement of the truthfulness/informativeness judge models that were originally introduced in the TruthfulQA paper. That model is based on OpenAI's Curie engine using their finetuning API. However, as of February 08, 2024, OpenAI has taken down its Curie engine, and thus, we cannot use it for TruthfulQA evaluation anymore. So, we decided to train the judge models using an open model (i.e., LLaMa), which can make the evaluation more accessible and reproducible. Released Models We released two models for the truthfulness and informativeness evaluation, respectively. Truthfulness Judge Informativenss Judge Training Details The training code and validation results of these models can be found here Usage These models are only intended for the TruthfulQA evaluation. They are intended to generalize to the evaluation of new models on the fixed set of prompts, but they may fail to generalize to new prompts. You can try the model using the following scripts:
Summarised from the published model card. Read the full card on the HuggingFace links below.
Specifications
| Maker | allenai |
|---|---|
| Type | Language models |
| Context | 4k tokens |
| Variants | 1 |
| Runs with | transformers |
| Released | 2024-02-07 |
| Popularity | 27k downloads / month |
| Likes | 6 |
| Licence | Open weights |
How it works
Variants
Open weights ship in several sizes and precisions. One page, all the variants — pick the one that fits your GPU. VRAM figures are estimates from model size.
| Variant | Params | Precision | VRAM | Fits 16 GB | Weights |
|---|---|---|---|---|---|
| truthfulqa-truth-judge-llama2-7B | — | BF16 | — | — | Weights ↗ |
Using it via the API
Once AxForge deploys truthfulqa-truth-judge-llama2 for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (truthfulqa-truth-judge-llama2 below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"truthfulqa-truth-judge-llama2","messages":[{"role":"user","content":"Hello"}]}'
Details
Languages
Trained / evaluated on
Tags
Licence
Open weights under apache-2.0 — commercial use is permitted. Deploy it on AxForge EU hardware on request. Read the licence ↗