openchat.2_super

Usage

To use these models, we highly recommend installing the OpenChat package by following the installation guide and using the OpenChat OpenAI-compatible API server by running the serving command from the table below. The server is optimized for high-throughput deployment using vLLM and can run on a GPU with at least 48GB RAM or two consumer GPUs with tensor parallelism. To enable tensor parallelism, append --tensor-parallel-size 2 to the serving command.

When started, the server listens at localhost:18888 for requests and is compatible with the OpenAI ChatCompletion API specifications. See the example request below for reference. Additionally, you can access the OpenChat Web UI for a user-friendly experience.

To deploy the server as an online service, use --api-keys sk-KEY1 sk-KEY2 ... to specify allowed API keys and --disable-log-requests --disable-log-stats --log-file openchat.log for logging only to a file. We recommend using a HTTPS gateway in front of the server for security purposes.

curl http://localhost:18888/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openchat_v3.2",
    "messages": [{"role": "user", "content": "You are a large language model named OpenChat. Write a poem to describe yourself"}]
  }'

For inference with Huggingface Transformers (slow and not recommended), follow the conversation template provided below:

# Single-turn V3.2 (SUPER)
tokenize("GPT4 User: HelloGPT4 Assistant:")
# Result: [1, 402, 7982, 29946, 4911, 29901, 15043, 32000, 402, 7982, 29946, 4007, 22137, 29901]

# Multi-turn V3.2 (SUPER)
tokenize("GPT4 User: HelloGPT4 Assistant: HiGPT4 User: How are you today?GPT4 Assistant:")
# Result: [1, 402, 7982, 29946, 4911, 29901, 15043, 32000, 402, 7982, 29946, 4007, 22137, 29901, 6324, 32000, 402, 7982, 29946, 4911, 29901, 1128, 526, 366, 9826, 29973, 32000, 402, 7982, 29946, 4007, 22137, 29901]

Benchmarks

We have evaluated our models using the two most popular evaluation benchmarks **, including AlpacaEval and MT-bench. Here we list the top models with our released versions, sorted by model size in descending order. The full version can be found on the MT-bench and AlpacaEval leaderboards.

To ensure consistency, we used the same routine as ChatGPT / GPT-4 to run these benchmarks. We started the OpenAI API-compatible server and set the openai.api_base to http://localhost:18888/v1 in the benchmark program.