Tele-FLM

Model Details

• Developed by: BAAI & TeleAI
• Language(s): English; Chinese; Other languages
• License: Apache 2.0

Technical Report

Tele-FLM Technical Report

Bias, Risks, and Limitations

Although we've made extensive efforts to thoroughly clean and filter the training corpus for the model, due to the open nature of the dataset, the model may still have picked up on some unsafe examples. Consequently, the model may still generate unexpected content, including but not limited to discrimination, bias, or offensive language. We would like to strongly advise users not to spread any unsafe content generated by the model. The project developers cannot be held responsible for any repercussions stemming from the dissemination of harmful information.

Quick Start

Use the code below to get started with Tele-FLM.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained('CofeAI/Tele-FLM', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained('CofeAI/Tele-FLM', torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, device_map="auto", trust_remote_code=True)
inputs = tokenizer('Beijing is the capital of China.', return_tensors='pt').to(model.device)
generated = model.generate(**inputs, max_new_tokens=128, repetition_penalty=1.03)
print(tokenizer.decode(generated.cpu()[0], skip_special_tokens=True))

Training Details

Training Data

Our training dataset comprises a variety of domains, as detailed in the table below. The total amount of data is roughly 2 trillion, with English and Chinese data in a ratio of about 2:1. In line with the methodology of GPT-4, we collected some instruct data and incorporated it into our pre-training data after removing the test sets of common datasets using the strict n-gram-based method. We deliberately avoid “training on the test set” or any other benchmark-oriented trick.

Model Architecture

We adopt the architecture of FLM-101B as the backbone for Tele-FLM, with several modifications:

• Rotary Positional Embedding (RoPE)
• RMSNorm for normalization
• SwiGLU for activation function
• Linear bias disabled
• Embedding and language model head untied
• Input and output multiplier

Consequently, Tele-FLM is largely compatible with Llama architecturally. To maximize convenience for the community, we made minimal adjustments to Llama's code to adapt it to Tele-FLM and released it as open source.

In the pre-training stage, we employ μP for optimal hyperparameter search. The μP model (Tele-FLM_μP) is architecturally identical to Tele-FLM except for the model width. The architecture of Tele-FLM and Tele-FLM_μP is listed below. For more details of μP, please refer to our technical report and the original Tensor Program papers.

Training Hyperparameters

Due to the smaller size, Tele-FLM_μP allows for significantly more experimental runs within fixed time and resource constraints. We searched seven hyperparameters for pretraining. All the hyperparameters are shown below.

Training Loss

Hardware

Tele-FLM is trained on a cluster of 112 A800 SXM4 GPU servers, each with 8 NVLink A800 GPUs and 2TB of RAM. The nodes have varied CPU configurations: 96 nodes with Intel 8358 (128x 2.60GHz) CPUs and 16 nodes with AMD 7643 (96x 2.30GHz) CPUs. A