Tucano2-qwen

All datasets, source code, and training recipes used to develop the Tucano2 series are fully open and reproducible.

Details

• Architecture: a Transformer-based model (qwen3)
• Size: 3,759,341,056 parameters
• Context length: 4,096 tokens
• Dataset(s):
  • Polygl0t/gigaverbo-v2-sft
  • Polygl0t/gigaverbo-v2-preferences
• Training time: ~ 140 hours
• Emissions: 110 KgCO2 (Germany)
• Total energy consumption: 290 kWh

This repository has the source code used to train this model. The full configuration used for training is available in the following config files:

• Single stage Supervised Fine-Tuning (linear warmup with cosine decay): training_config_sft.yaml
• Single stage Anchored Preference Optimization (linear warmup with cosine decay): training_config_apo.yaml
• Training Logs (loss, lr, rewards, etc.): train_logs_apo.parquet, train_logs_sft.parquet

Intended Uses

The primary intended use Tucano2-qwen-3.7B-Instruct is to serve as foundations for research and development involving Portuguese language modeling. You may also fine-tune and adapt Tucano2-qwen-3.7B-Instruct for deployment if your use follows the Apache 2.0 license. If you decide to use Tucano2-qwen-3.7B-Instruct as a basis for your fine-tuned model, please conduct your own risk and bias assessment.

Basic usage

from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
import torch

# Load model and tokenizer
model_id = "Polygl0t/Tucano2-qwen-3.7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto"
)

# Configure generation parameters
generation_config = GenerationConfig(
    do_sample=True,
    temperature=0.1,
    top_k=50,
    top_p=1.0,
    repetition_penalty=1.2,
    max_new_tokens=150,
    pad_token_id=tokenizer.eos_token_id,
)

# Prepare chat messages
messages = [
    {"role": "user", "content": "Qual é a capital de Cabo Verde?"}
]

# Apply chat template and generate
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, generation_config=generation_config)

# Decode and print response
response = tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=True)
print(f"🤖 {response}")

Limitations

Like almost all other language models trained on large text datasets scraped from the web, the Tucano2-qwen-3.7B-Instruct shows behavior that does not make it an out-of-the-box solution to many real-world applications, especially those requiring factual, reliable, and nontoxic text generation. Tucano2-qwen-3.7B-Instruct is subject to the following:

• Hallucinations: Tucano2-qwen-3.7B-Instruct can produce content that can be mistaken as true facts, but are misleading or entirely false, i.e., hallucination.

• Biases and Toxicity: Tucano2-qwen-3.7B-Instruct inherits the social and historical stereotypes from the data used to train it. Given these biases, the model can produce toxic content, i.e., harmful, offensive, or detrimental to individuals, groups, or communities.

• Language Limitations: Tucano2-qwen-3.7B-Instruct is primarily designed to interact with Portuguese. Other languages might challenge its comprehension, leading to potential misinterpretations or errors in response.

• Repetition and Verbosity: Tucano2-qwen-3.7B-Instruct may get stuck on repetition loops (especially if the repetition penalty during generations is set to a meager value) or produce verbose responses unrelated to the prompt it was given.

Hence, even though Tucano2-qwen-3.7B-Instruct is released with a permissive license, we urge users to perform their risk analysis on them if they intend to use them for real-world applications.

Evaluations

The table below compares the Tucano2 (Instruct variant) series against other chat models of similar size. We divide our evaluations into three sets:

• Knowledge & Reasoning: ARC-Challenge, ENEM, BLUEX, OAB Exams, BELEBELE, MMLU, GSM8K-PT
• Instruction Following: IFEval-PT
• Coding: HumanEval

The NPM (Normalized Performance Metric) provides a balanced view of model performance across tasks, accounting for each task's inherent difficulty by normalizing its evaluation score relative to its random baseline.