SecureBERT2.0-cross_encoder

Model Details

Model Description

• Developed by: Cisco AI
• Model type: Cross Encoder (Sentence Similarity)
• Architecture: ModernBERT (fine-tuned via Sentence Transformers)
• Max Sequence Length: 1024 tokens
• Output Labels: 1 (similarity score)
• Language: English
• License: Apache-2.0
• Finetuned from model: cisco-ai/SecureBERT2.0-base

Uses

Direct Use

• Semantic text similarity in cybersecurity contexts
• Text and code reranking for information retrieval (IR)
• Threat intelligence question–answer relevance scoring
• Cybersecurity report and log correlation

Downstream Use

Can be integrated into:

• Cyber threat intelligence search engines
• SOC automation pipelines
• Cybersecurity knowledge graph enrichment
• Threat hunting and incident response systems

Out-of-Scope Use

• Generic text similarity outside the cybersecurity domain
• Tasks requiring generative reasoning or open-domain question answering

Bias, Risks, and Limitations

The model reflects the distribution of cybersecurity-related data used during fine-tuning. Potential risks include:

• Overrepresentation of specific malware, technologies, or threat actors
• Bias toward technical English sources
• Reduced performance on non-English or mixed technical/natural text

Recommendations

Users should evaluate results for domain alignment and combine with other retrieval models or heuristic filters when applied to non-cybersecurity contexts.

How to Get Started with the Model

Using the Sentence Transformers API

Install dependencies

pip install -U sentence-transformers

Run Inference

from sentence_transformers import CrossEncoder

# Load the model
model = CrossEncoder("cisco-ai/SecureBERT2.0-cross-encoder")

# Example pairs
pairs = [
    ["How does Stealc malware extract browser data?",
     "Stealc uses Sqlite3 DLL to query browser databases and retrieve cookies, passwords, and history."],
    ["Best practices for post-acquisition cybersecurity integration?",
     "Conduct security assessment, align policies, integrate security technologies, and train employees."],
]

# Compute similarity scores
scores = model.predict(pairs)
print(scores)

Rank Candidate Responses

query = "How to prevent Kerberoasting attacks?"
candidates = [
    "Implement MFA and privileged access management",
    "Monitor Kerberos tickets for anomalous activity",
    "Apply zero-trust network segmentation",
]
ranking = model.rank(query, candidates)
print(ranking)

Framework Versions

• python: 3.10.10
• sentence_transformers: 5.0.0
• transformers: 4.52.4
• PyTorch: 2.7.0+cu128
• accelerate: 1.9.0
• datasets: 3.6.0

Training Details

Training Dataset

The model was fine-tuned on a cybersecurity sentence-pair similarity dataset for cross-encoder training.

• Dataset Size: 35,705 samples
• Columns: sentence1, sentence2, label

Average Lengths (first 1000 samples)

Example Schema

Training Objective and Loss

The model was trained using a contrastive ranking objective to learn high-quality similarity scores between cybersecurity-related text pairs.

• Loss Function: CachedMultipleNegativesRankingLoss

Loss Parameters

{
    "scale": 10.0,
    "num_negatives": 10,
    "activation_fn": "torch.nn.modules.activation.Sigmoid",
    "mini_batch_size": 24
}

Evaluation

Testing Data, Factors & Metrics

Testing Data

The evaluation was performed on a held-out test set of cybersecurity-related question–answer pairs and document retrieval tasks. Data includes:

• Threat intelligence descriptions and related advisories
• Exploit procedure and mitigation text pairs
• Cybersecurity Q&A and incident analysis examples

Factors

Evaluation considered multiple aspects of similarity and relevance:

• Domain diversity: different cybersecurity subfields (malware, vulnerabilities, network defense)
• Task diversity: retrieval, reranking, and relevance scoring
• Pair length: from short queries to long technical documents

Metrics

The model was evaluated using standard information retrieval metrics:

• Mean Average Precision (mAP): measures ranking precision across all retrieved results
• Recall@1 (R@1): measures the proportion of correct top-1 matches
• Normalized Discounted Cumulative Gain (NDCG@10): evaluates ranking quality up to the 10th result
• Mean Reciprocal Rank (MRR@10): assesses the average rank position of the first correct answer

Results

Summary

The SecureBERT 2.0 Cross-Encoder achieves state-of-the-art retrieval and ranking performance on cybersecurity text similarity tasks.

Compared to the general-purpose ms-marco-TinyBERT-L2 baseline:

• It improves mAP by +0.035
• Achieves nearly perfect R@1 and MRR@10, indicating highly accura