finance-Llama3

• 2026/1/23: Released LLM-in-Sandbox Elicits General Agentic Intelligence, where the data of Instruction Pre-Training achieves robust generalization in agentic RL!
• 2024/11/30: Released the multimodal version of the instruction synthesizer: Visual Instruction Synthesizer
• 2024/9/20: Our paper has been accepted by EMNLP 2024 main conference🎉
• 2024/9/11: Updated FAQ on continual pre-training from Llama3
• 2024/8/29: Updated guidelines on evaluating any 🤗Huggingface models on the domain-specific tasks
• 2024/7/31: Updated pre-training suggestions in the Advanced Usage section of instruction-synthesizer
• 2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M. The performance trend on downstream tasks throughout the pre-training process:
• 2024/6/21: Released the paper, code, and resources

Resources

🤗 We share our data and models with example usages, feel free to open any discussions at this page! 🤗

• Thanks to the demo davanstrien/instruction-synthesizer for implementing our approach
• Context-Based Instruction Synthesizer: instruction-synthesizer
• Fine-Tuning Data for the Synthesizer: ft-instruction-synthesizer-collection
• General Models Pre-Trained from Scratch (on 100B tokes):
  • InstructLM-500M
  • InstructLM-1.3B
• Domain-Specific Models Pre-Trained from Llama3-8B:
  • Finance-Llama3-8B
  • Biomedicine-Llama3-8B
• General Instruction-Augmented Corpora: general-instruction-augmented-corpora
• Domain-Specific Instruction-Augmented Corpora (no finance data to avoid ethical issues): medicine-instruction-augmented-corpora

Domain-Adaptive Continued Pre-Training

Following AdaptLLM, we augment the domain-specific raw corpora with instruction-response pairs generated by our context-based instruction synthesizer.

1. To chat with the finance-Llama3-8B model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("instruction-pretrain/finance-Llama3-8B")
tokenizer = AutoTokenizer.from_pretrained("instruction-pretrain/finance-Llama3-8B")

# Put your input here, NO prompt template is required
user_input = '''Use this fact to answer the question: Title of each class Trading Symbol(s) Name of each exchange on which registered
Common Stock, Par Value $.01 Per Share MMM New York Stock Exchange
MMM Chicago Stock Exchange, Inc.
1.500% Notes due 2026 MMM26 New York Stock Exchange
1.750% Notes due 2030 MMM30 New York Stock Exchange
1.500% Notes due 2031 MMM31 New York Stock Exchange

Which debt securities are registered to trade on a national securities exchange under 3M's name as of Q2 of 2023?'''

inputs = tokenizer(user_input, return_tensors="pt", add_special_tokens=True).input_ids.to(model.device)
outputs = model.generate(input_ids=inputs, max_new_tokens=400)[0]

answer_start = int(inputs.shape[-1])
pred = tokenizer.decode(outputs[answer_start:], skip_special_tokens=True)

print(pred)

2. To evaluate any Huggingface LMs on domain-specific tasks (💡New!)

You can use the following script to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that the script is NOT applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct).

1). Set Up Dependencies

git clone https://github.com/microsoft/LMOps
cd LMOps/adaptllm
pip install -r requirements.txt

2). Evaluate the Model

# Select the domain from ['biomedicine', 'finance']
DOMAIN='finance'

# Specify any Huggingface LM name (Not applicable to models requiring specific prompt templates)
MODEL