nusaiba19/roberta-immigrant-hate-speech

RoBERTa — US Immigrant-Targeted Hate Speech Detection

Fine-tuned roberta-base for binary classification of hate speech targeting immigrants in US social media contexts.

Model

• Base: roberta-base (125M params)
• Task: Binary classification (hate vs. not-hate)
• Language: English
• Training device: Apple M4 (MPS backend)

Training Data

Filtered subset of UC Berkeley Measuring Hate Speech dataset targeting immigrants:

• Filter: target_origin_immigrant > 0
• Total: 11,193 comments
• Train: 8,954 / Val: 1,119 / Test: 1,120

Training Setup

• LR: 2e-5
• Batch: 16
• Max length: 256
• Epochs: 3
• Class weights: balanced

Results (Test)

Metric	Value
Accuracy	0.9527
Macro-F1	0.9520
Weighted-F1	0.9527
Gap	0.0007

Per-class

Class	Precision	Recall	F1
not-hate	0.951	0.942	0.946
hate	0.954	0.962	0.958

Key Finding

The near-zero accuracy-macro-F1 gap (0.0007) on this balanced dataset (ratio 1.25) validates the diagnostic framework: when class distribution is balanced, accuracy and macro-F1 agree. This contrasts with imbalanced datasets like X-Fact Hindi, where the gap reaches 0.412 and reveals complete minority-class failure.

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("nusaiba19/roberta-immigrant-hate-speech")
model = AutoModelForSequenceClassification.from_pretrained("nusaiba19/roberta-immigrant-hate-speech")

text = "Example comment here"
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256)
with torch.no_grad():
    pred = model(**inputs).logits.argmax(-1).item()
print(["not-hate", "hate"][pred])