Nous-Hermes-2-SOLAR

Model description

Nous Hermes 2 - SOLAR 10.7B is the flagship Nous Research model on the SOLAR 10.7B base model..

Nous Hermes 2 SOLAR 10.7B was trained on 1,000,000 entries of primarily GPT-4 generated data, as well as other high quality data from open datasets across the AI landscape.

Table of Contents

1. Example Outputs
2. Benchmark Results
   • GPT4All
   • AGIEval
   • BigBench
   • TruthfulQA
3. Prompt Format
4. Quantized Models

Benchmark Results

Nous-Hermes 2 on SOLAR 10.7B is a major improvement across the board on the benchmarks below compared to the base SOLAR 10.7B model, and comes close to approaching our Yi-34B model!

Example Outputs

Ask for help creating a discord bot:

Benchmarks Compared

GPT4All:

AGIEval:

BigBench:

TruthfulQA:

GPT4All

GPT-4All Benchmark Set

|    Task     |Version| Metric |Value |   |Stderr|
|-------------|------:|--------|-----:|---|-----:|
|arc_challenge|      0|acc     |0.5768|_  |0.0144|
|             |       |acc_norm|0.6067|_  |0.0143|
|arc_easy     |      0|acc     |0.8375|_  |0.0076|
|             |       |acc_norm|0.8316|_  |0.0077|
|boolq        |      1|acc     |0.8875|_  |0.0055|
|hellaswag    |      0|acc     |0.6467|_  |0.0048|
|             |       |acc_norm|0.8321|_  |0.0037|
|openbookqa   |      0|acc     |0.3420|_  |0.0212|
|             |       |acc_norm|0.4580|_  |0.0223|
|piqa         |      0|acc     |0.8161|_  |0.0090|
|             |       |acc_norm|0.8313|_  |0.0087|
|winogrande   |      0|acc     |0.7814|_  |0.0116|

Average: 74.69%

AGI-Eval

|             Task             |Version| Metric |Value |   |Stderr|
|------------------------------|------:|--------|-----:|---|-----:|
|agieval_aqua_rat              |      0|acc     |0.3189|_  |0.0293|
|                              |       |acc_norm|0.2953|_  |0.0287|
|agieval_logiqa_en             |      0|acc     |0.5438|_  |0.0195|
|                              |       |acc_norm|0.4977|_  |0.0196|
|agieval_lsat_ar               |      0|acc     |0.2696|_  |0.0293|
|                              |       |acc_norm|0.2087|_  |0.0269|
|agieval_lsat_lr               |      0|acc     |0.7078|_  |0.0202|
|                              |       |acc_norm|0.6255|_  |0.0215|
|agieval_lsat_rc               |      0|acc     |0.7807|_  |0.0253|
|                              |       |acc_norm|0.7063|_  |0.0278|
|agieval_sat_en                |      0|acc     |0.8689|_  |0.0236|
|                              |       |acc_norm|0.8447|_  |0.0253|
|agieval_sat_en_without_passage|      0|acc     |0.5194|_  |0.0349|
|                              |       |acc_norm|0.4612|_  |0.0348|
|agieval_sat_math              |      0|acc     |0.4409|_  |0.0336|
|                              |       |acc_norm|0.3818|_  |0.0328|

Average: 47.79%

BigBench Reasoning Test

|                      Task                      |Version|       Metric        |Value |   |Stderr|
|------------------------------------------------|------:|---------------------|-----:|---|-----:|
|bigbench_causal_judgement                       |      0|multiple_choice_grade|0.5737|_  |0.0360|
|bigbench_date_understanding                     |      0|multiple_choice_grade|0.7263|_  |0.0232|
|bigbench_disambiguation_qa                      |      0|multiple_choice_grade|0.3953|_  |0.0305|
|bigbench_geometric_shapes                       |      0|multiple_choice_grade|0.4457|_  |0.0263|
|                                                |       |exact_str_match      |0.0000|_  |0.0000|
|bigbench_logical_deduction_five_objects         |      0|multiple_choice_grade|0.2820|_  |0.0201|
|bigbench_logical_deduction_seven_objects        |      0|multiple_choice_grade|0.2186|_  |0.0156|
|bigbench_logical_deduction_three_objects        |      0|multiple_choice_grade|0.4733|_  |0.0289|
|bigbench_movie_recommendation                   |      0|multiple_choice_grade|0.5200|_  |0.0224|
|bigbench_navigate                               |      0|multiple_choice_grade|0.4910|_  |0.0158|
|bigbench_reasoning_about_colored_objects        |      0|multiple_choice_grade|0.7495|_  |0.0097|
|bigbench_ruin_names                             |      0|multiple_choice_grade|0.5938|_  |0.0232|
|bigbench_salient_translation_error_detection    |      0|multiple_choice_grade|0.3808|_  |0.0154|
|bigbench_snarks                                 |      0|multiple_choice_grade|0.8066|_  |0.0294|
|bigbench_sports_understanding                   |      0|multiple_choice_grade|0.5101|_  |0.0159|
|bigbench_temporal_sequences                     |      0|multiple_choice_grade|0.3850|_  |0.0154|
|bigbench_tracking_shuffled_objects_five_objects |      0|multiple_choice_grade|0.2160|_  |0.0116|
|bigbench_tracking_shuffled_objects_seven_objects|      0|multiple_choice_grade|0.1634|_  |0.0088|
|bigbench_tracking_shuffled_objects_three_objects|      0|multiple_choice_grade|0.4733|_  |0.0289|
Average: 44.84%

TruthfulQA:

|    Task     |Version|Metric|Value |   |Stderr|
|-------------|------:|------|-----:|---|-----:|
|truthfulqa_mc|      1|mc1   |0.3917|_  |0.0171|
|             |       |mc2   |0.5592|_  |0.0154|

Average Score Comparison between OpenHermes-1 Llama-2 13B and OpenHermes-2 Mistral 7B against OpenHermes-2.5 on Mistral-7B:

|     Bench     | OpenHermes-2.5 Mistral 7B | Nous-Hermes-2-SOLAR-10B | Change/OpenHermes2.5 |
|---------------|---------------------------|------------------------|-----------------------|
|GPT4All        |                      73.12|                   74.69|                  +1.57|
|--------------------------------------------------------------------------------------------|
|BigBench       |                      40.96|                   44.84|                  +3.88|
|--------------------------------------------------------------------------------------------|
|AGI Eval       |                      43.07|