Model reference · open weights
Nous-Hermes-2-SOLAR is an open-weight language model from NousResearch. AxForge deploys and operates it for you on dedicated EU-owned hardware — with the licence handled where one is required.
Available as managed deployment — configured and operated for you on dedicated EU hardware, quoted per deployment.
What it is
| Maker | NousResearch |
|---|---|
| Type | Language models |
| Task | Text gen |
| Parameters (lead) | 10.7B |
| Context | 4k tokens |
| Runs with | transformers |
| Based on | upstage/SOLAR-10.7B-v1.0 |
| Released | 2024-01-01 |
| Popularity | 9k downloads / month |
| Licence | Open weights |
About
Nous Hermes 2 - SOLAR 10.7B is the flagship Nous Research model on the SOLAR 10.7B base model..
Nous Hermes 2 SOLAR 10.7B was trained on 1,000,000 entries of primarily GPT-4 generated data, as well as other high quality data from open datasets across the AI landscape.
Nous-Hermes 2 on SOLAR 10.7B is a major improvement across the board on the benchmarks below compared to the base SOLAR 10.7B model, and comes close to approaching our Yi-34B model!
GPT4All:
AGIEval:
BigBench:
TruthfulQA:
GPT-4All Benchmark Set
| Task |Version| Metric |Value | |Stderr|
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge| 0|acc |0.5768|_ |0.0144|
| | |acc_norm|0.6067|_ |0.0143|
|arc_easy | 0|acc |0.8375|_ |0.0076|
| | |acc_norm|0.8316|_ |0.0077|
|boolq | 1|acc |0.8875|_ |0.0055|
|hellaswag | 0|acc |0.6467|_ |0.0048|
| | |acc_norm|0.8321|_ |0.0037|
|openbookqa | 0|acc |0.3420|_ |0.0212|
| | |acc_norm|0.4580|_ |0.0223|
|piqa | 0|acc |0.8161|_ |0.0090|
| | |acc_norm|0.8313|_ |0.0087|
|winogrande | 0|acc |0.7814|_ |0.0116|
Average: 74.69%
AGI-Eval
| Task |Version| Metric |Value | |Stderr|
|------------------------------|------:|--------|-----:|---|-----:|
|agieval_aqua_rat | 0|acc |0.3189|_ |0.0293|
| | |acc_norm|0.2953|_ |0.0287|
|agieval_logiqa_en | 0|acc |0.5438|_ |0.0195|
| | |acc_norm|0.4977|_ |0.0196|
|agieval_lsat_ar | 0|acc |0.2696|_ |0.0293|
| | |acc_norm|0.2087|_ |0.0269|
|agieval_lsat_lr | 0|acc |0.7078|_ |0.0202|
| | |acc_norm|0.6255|_ |0.0215|
|agieval_lsat_rc | 0|acc |0.7807|_ |0.0253|
| | |acc_norm|0.7063|_ |0.0278|
|agieval_sat_en | 0|acc |0.8689|_ |0.0236|
| | |acc_norm|0.8447|_ |0.0253|
|agieval_sat_en_without_passage| 0|acc |0.5194|_ |0.0349|
| | |acc_norm|0.4612|_ |0.0348|
|agieval_sat_math | 0|acc |0.4409|_ |0.0336|
| | |acc_norm|0.3818|_ |0.0328|
Average: 47.79%
BigBench Reasoning Test
| Task |Version| Metric |Value | |Stderr|
|------------------------------------------------|------:|---------------------|-----:|---|-----:|
|bigbench_causal_judgement | 0|multiple_choice_grade|0.5737|_ |0.0360|
|bigbench_date_understanding | 0|multiple_choice_grade|0.7263|_ |0.0232|
|bigbench_disambiguation_qa | 0|multiple_choice_grade|0.3953|_ |0.0305|
|bigbench_geometric_shapes | 0|multiple_choice_grade|0.4457|_ |0.0263|
| | |exact_str_match |0.0000|_ |0.0000|
|bigbench_logical_deduction_five_objects | 0|multiple_choice_grade|0.2820|_ |0.0201|
|bigbench_logical_deduction_seven_objects | 0|multiple_choice_grade|0.2186|_ |0.0156|
|bigbench_logical_deduction_three_objects | 0|multiple_choice_grade|0.4733|_ |0.0289|
|bigbench_movie_recommendation | 0|multiple_choice_grade|0.5200|_ |0.0224|
|bigbench_navigate | 0|multiple_choice_grade|0.4910|_ |0.0158|
|bigbench_reasoning_about_colored_objects | 0|multiple_choice_grade|0.7495|_ |0.0097|
|bigbench_ruin_names | 0|multiple_choice_grade|0.5938|_ |0.0232|
|bigbench_salient_translation_error_detection | 0|multiple_choice_grade|0.3808|_ |0.0154|
|bigbench_snarks | 0|multiple_choice_grade|0.8066|_ |0.0294|
|bigbench_sports_understanding | 0|multiple_choice_grade|0.5101|_ |0.0159|
|bigbench_temporal_sequences | 0|multiple_choice_grade|0.3850|_ |0.0154|
|bigbench_tracking_shuffled_objects_five_objects | 0|multiple_choice_grade|0.2160|_ |0.0116|
|bigbench_tracking_shuffled_objects_seven_objects| 0|multiple_choice_grade|0.1634|_ |0.0088|
|bigbench_tracking_shuffled_objects_three_objects| 0|multiple_choice_grade|0.4733|_ |0.0289|
Average: 44.84%
TruthfulQA:
| Task |Version|Metric|Value | |Stderr|
|-------------|------:|------|-----:|---|-----:|
|truthfulqa_mc| 1|mc1 |0.3917|_ |0.0171|
| | |mc2 |0.5592|_ |0.0154|
Average Score Comparison between OpenHermes-1 Llama-2 13B and OpenHermes-2 Mistral 7B against OpenHermes-2.5 on Mistral-7B:
| Bench | OpenHermes-2.5 Mistral 7B | Nous-Hermes-2-SOLAR-10B | Change/OpenHermes2.5 |
|---------------|---------------------------|------------------------|-----------------------|
|GPT4All | 73.12| 74.69| +1.57|
|--------------------------------------------------------------------------------------------|
|BigBench | 40.96| 44.84| +3.88|
|--------------------------------------------------------------------------------------------|
|AGI Eval | 43.07|From the published model card. Full card on the HuggingFace links in the sidebar.
Using it via the API
Once AxForge deploys nous-hermes-2-solar for you, it answers on the OpenAI-compatible API — the same base URL and keys as every other model. (nous-hermes-2-solar below is illustrative; you get the exact model name on deployment.)
$ curl -sS https://api.axforge.ai/v1/chat/completions \
-H "Authorization: Bearer $AXFORGE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"nous-hermes-2-solar","messages":[{"role":"user","content":"Hello"}]}'
Create an account — your API key is available in the console. 5M tokens/month currently included with every new account at launch.