ERNIE-4.5-2Bits-TP4-Paddle

ERNIE 4.5 Highlights

The advanced capabilities of the ERNIE 4.5 models, particularly the MoE-based A47B and A3B series, are underpinned by several key technical innovations:

1. Multimodal Heterogeneous MoE Pre-Training: Our models are jointly trained on both textual and visual modalities to better capture the nuances of multimodal information and improve performance on tasks involving text understanding and generation, image understanding, and cross-modal reasoning. To achieve this without one modality hindering the learning of another, we designed a heterogeneous MoE structure, incorporated modality-isolated routing, and employed router orthogonal loss and multimodal token-balanced loss. These architectural choices ensure that both modalities are effectively represented, allowing for mutual reinforcement during training.

2. Scaling-Efficient Infrastructure: We propose a novel heterogeneous hybrid parallelism and hierarchical load balancing strategy for efficient training of ERNIE 4.5 models. By using intra-node expert parallelism, memory-efficient pipeline scheduling, FP8 mixed-precision training and finegrained recomputation methods, we achieve remarkable pre-training throughput. For inference, we propose multi-expert parallel collaboration method and convolutional code quantization algorithm to achieve 4-bit/2-bit lossless quantization. Furthermore, we introduce PD disaggregation with dynamic role switching for effective resource utilization to enhance inference performance for ERNIE 4.5 MoE models. Built on PaddlePaddle, ERNIE 4.5 delivers high-performance inference across a wide range of hardware platforms.

3. Modality-Specific Post-Training: To meet the diverse requirements of real-world applications, we fine-tuned variants of the pre-trained model for specific modalities. Our LLMs are optimized for general-purpose language understanding and generation. The VLMs focuses on visuallanguage understanding and supports both thinking and non-thinking modes. Each model employed a combination of Supervised Fine-tuning (SFT), Direct Preference Optimization (DPO) or a modified reinforcement learning method named Unified Preference Optimization (UPO) for post-training.

Model Overview

ERNIE-4.5-300B-A47B is a text MoE Post-trained model, with 300B total parameters and 47B activated parameters for each token. The following are the model configuration details:

Quickstart

Using FastDeploy

Service deployment can be quickly completed using FastDeploy in the following command. For more detailed usage instructions, please refer to the FastDeploy Repository.

Note: To deploy on a configuration with 4 GPUs each having at least 80G of memory, specify --quantization wint4. If you specify --quantization wint8, then resources for 8 GPUs are required.

python -m fastdeploy.entrypoints.openai.api_server \
       --model baidu/ERNIE-4.5-300B-A47B-Paddle \
       --port 8180 \
       --metrics-port 8181 \
       --quantization wint4 \
       --tensor-parallel-size 8 \
       --engine-worker-queue-port 8182 \
       --max-model-len 32768 \
       --max-num-seqs 32

To deploy the W4A8C8 quantized version using FastDeploy, you can run the following command.

python -m fastdeploy.entrypoints.openai.api_server \
       --model baidu/ERNIE-4.5-300B-A47B-W4A8C8-TP4-Paddle \
       --port 8180 \
       --metrics-port 8181 \
       --engine-worker-queue-port 8182 \
       --tensor-parallel-size 4 \
       --max-model-len 32768 \
       --max-num-seqs 32

To deploy the WINT2 quantized version using FastDeploy on four GPUs, run the following command.

python -m fastdeploy.entrypoints.openai.api_server \
       --model "baidu/ERNIE-4.5-300B-A47B-2Bits-TP4-Paddle" \
       --port 8180 \
       --metrics-port 8181 \
       --engine-worker-queue-port 8182 \
       --tensor-parallel-size 4 \
       --max-model-len  32768 \
       --max-num-seqs 128

The following contains a code snippet illustrating how to use ERNIE-4.5-300B-A47B-FP8 generate content based on given inputs.

from fastdeploy import LLM, SamplingParams

prompts = [
    "Hello, my name is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=128)

model = "baidu/ERNIE-4.5-300B-A47B-FP8-Paddle"
llm = LLM(model=model, tensor_parallel_size=8, max_model_len=8192, num_gpu_blocks_override=1024, engine_worker_queue_port=9981)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs.text
    print("generated_text", generated_text)

Best Practices

Sampling Parameters

To achieve optimal performance, we suggest using Temperature=0.8, TopP=0.8.

Prompts for Web Search

For Web Search, {references}, {date}, and {question} are arguments.

For Chinese question, we use the prompt:

ernie_search_zh_prompt = \
'''下面你会收到当前时间、多个不同来源的参考文章和一段对话。你的任务是阅读多个参考文章，并根据参考文章中的信息回答对话中的问题。
以下是当前时间和参考文章：
---------
#当前时间
{date}

#参考文章
{references}

---------
请注意：
1. 回答必须结合问题需求和当前时间，对参考文章的可用性进行判断，避免在回答中使用错误或过时的信息。
2. 当参考文章中的信息无法准确地回答问题时，你需要在回答中提供获取相应信息的建议，或承认无法提供相应信息。
3. 你需要优先根据百科、官网、权威机构、专业网站等高权威性来源的信息来回答问题。
4. 回复需要综合参考文章中的相关数字、案例、法律条文、公式等信息，使你的答案更专业。
5. 当问题属于创作类任务时，需注意以下维度：
   - 态度鲜明：观点、立场清晰明确，避免模棱两可，语言果断直接
   - 文采飞扬：用词精准生动，善用修辞手法，增强感染力
   - 有理有据：逻辑严密递进，结合权威数据/事实支撑论点
---------
下面请结合以上信息，回答问题，补全对话
{question}'''

For English question, we use the prompt:

ernie_search_en_prompt = \
'''
Below you will be given the current time, multiple references from different sources, and a conversation. Your task is to read t