Hermes-3-Llama-3.2

Model Description

Hermes 3 3B is a small but mighty new addition to the Hermes series of LLMs by Nous Research, and is Nous's first fine-tune in this parameter class.

For details on Hermes 3, please see the Hermes 3 Technical Report.

Hermes 3 is a generalist language model with many improvements over Hermes 2, including advanced agentic capabilities, much better roleplaying, reasoning, multi-turn conversation, long context coherence, and improvements across the board.

Hermes 3 3B is a full parameter fine-tune of the Llama-3.2 3B foundation model, focused on aligning LLMs to the user, with powerful steering capabilities and control given to the end user.

The Hermes 3 series builds and expands on the Hermes 2 set of capabilities, including more powerful and reliable function calling and structured output capabilities, generalist assistant capabilities, and improved code generation skills.

Hermes 3 3B was trained on H100s on LambdaLabs GPU Cloud. Check out LambdaLabs' cloud offerings here.

Benchmarks

Hermes 3 is competitive, if not superior, to Llama-3.1 Instruct models at general capabilities, with varying strengths and weaknesses attributable between the two.

GPT4All:

|    Tasks    |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------|------:|------|-----:|--------|---|-----:|---|-----:|
|arc_challenge|      1|none  |     0|acc     |↑  |0.4411|±  |0.0145|
|             |       |none  |     0|acc_norm|↑  |0.4377|±  |0.0145|
|arc_easy     |      1|none  |     0|acc     |↑  |0.7399|±  |0.0090|
|             |       |none  |     0|acc_norm|↑  |0.6566|±  |0.0097|
|boolq        |      2|none  |     0|acc     |↑  |0.8327|±  |0.0065|
|hellaswag    |      1|none  |     0|acc     |↑  |0.5453|±  |0.0050|
|             |       |none  |     0|acc_norm|↑  |0.7047|±  |0.0046|
|openbookqa   |      1|none  |     0|acc     |↑  |0.3480|±  |0.0213|
|             |       |none  |     0|acc_norm|↑  |0.4280|±  |0.0221|
|piqa         |      1|none  |     0|acc     |↑  |0.7639|±  |0.0099|
|             |       |none  |     0|acc_norm|↑  |0.7584|±  |0.0100|
|winogrande   |      1|none  |     0|acc     |↑  |0.6590|±  |0.0133|

Average: 64.00

AGIEval:

|            Tasks             |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|------------------------------|------:|------|-----:|--------|---|-----:|---|-----:|
|agieval_aqua_rat              |      1|none  |     0|acc     |↑  |0.2283|±  |0.0264|
|                              |       |none  |     0|acc_norm|↑  |0.2441|±  |0.0270|
|agieval_logiqa_en             |      1|none  |     0|acc     |↑  |0.3057|±  |0.0181|
|                              |       |none  |     0|acc_norm|↑  |0.3272|±  |0.0184|
|agieval_lsat_ar               |      1|none  |     0|acc     |↑  |0.2304|±  |0.0278|
|                              |       |none  |     0|acc_norm|↑  |0.1957|±  |0.0262|
|agieval_lsat_lr               |      1|none  |     0|acc     |↑  |0.3784|±  |0.0215|
|                              |       |none  |     0|acc_norm|↑  |0.3588|±  |0.0213|
|agieval_lsat_rc               |      1|none  |     0|acc     |↑  |0.4610|±  |0.0304|
|                              |       |none  |     0|acc_norm|↑  |0.4275|±  |0.0302|
|agieval_sat_en                |      1|none  |     0|acc     |↑  |0.6019|±  |0.0342|
|                              |       |none  |     0|acc_norm|↑  |0.5340|±  |0.0348|
|agieval_sat_en_without_passage|      1|none  |     0|acc     |↑  |0.3981|±  |0.0342|
|                              |       |none  |     0|acc_norm|↑  |0.3981|±  |0.0342|
|agieval_sat_math              |      1|none  |     0|acc     |↑  |0.2500|±  |0.0293|
|                              |       |none  |     0|acc_norm|↑  |0.2636|±  |0.0298|

Average: 34.36

BigBench:

|                         Tasks                         |Version|Filter|n-shot| Metric |   |Value |   |Stderr|
|-------------------------------------------------------|------:|------|-----:|--------|---|-----:|---|-----:|
|leaderboard_bbh_boolean_expressions                    |      1|none  |     3|acc_norm|↑  |0.7560|±  |0.0272|
|leaderboard_bbh_causal_judgement                       |      1|none  |     3|acc_norm|↑  |0.6043|±  |0.0359|
|leaderboard_bbh_date_understanding                     |      1|none  |     3|acc_norm|↑  |0.3280|±  |0.0298|
|leaderboard_bbh_disambiguation_qa                      |      1|none  |     3|acc_norm|↑  |0.5880|±  |0.0312|
|leaderboard_bbh_formal_fallacies                       |      1|none  |     3|acc_norm|↑  |0.5280|±  |0.0316|
|leaderboard_bbh_geometric_shapes                       |      1|none  |     3|acc_norm|↑  |0.3560|±  |0.0303|
|leaderboard_bbh_hyperbaton                             |      1|none  |     3|acc_norm|↑  |0.6280|±  |0.0306|
|leaderboard_bbh_logical_deduction_five_objects         |      1|none  |     3|acc_norm|↑  |0.3400|±  |0.0300|
|leaderboard_bbh_logical_deduction_seven_objects        |      1|none  |     3|acc_norm|↑  |0.2880|±  |0.0287|
|leaderboard_bbh_logical_deduction_three_objects        |      1|none  |     3|acc_norm|↑  |0.4160|±  |0.0312|
|leaderboard_bbh_movie_recommendation                   |      1|none  |     3|acc_norm|↑  |0.6760|±  |0.0297|
|leaderboard_bbh_navigate                               |      1|none  |     3|acc_norm|↑  |0.5800|±  |0.0313|
|leaderboard_bbh_object_counting                        |      1|none  |     3|acc_norm|↑  |0.3640|±  |0.0305|
|leaderboard_bbh_penguins_in_a_table                    |      1|none  |     3|acc_norm|↑  |0.3836|±  |0.0404|
|leaderboard_bbh_reasoning_about_colored_objects        |      1|none  |     3|acc_norm|↑  |0.3560|±  |0.0303|
|leaderboard_bbh_ruin_names                             |      1|none  |     3|acc_norm|↑  |0.4160|±  |0.0312|
|leaderboard_bbh_salient_translation_error_detection    |      1|none  |     3|acc_norm|↑  |0.3080|±  |0.0293|
|lead