IC-569Mistral-7B-instruct-v0.1 achieves only F1 of 0.419 on zero-shot stance detection for the X-Stance German dataset, substantially below the fine-tuned BERT baseline (F1 0.693)

Stefan Sylvius Wagner, Maike Behrendt, Marc Ziegele, Stefan Harmeling

SourceThe Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions

The paper evaluates Mistral-7B-instruct-v0.1 directly on the X-Stance German dataset (10 selected questions) for zero-shot stance detection, using a prompt that asks the model to classify a comment as favor or against a given political question. The model achieves an average F1 of 0.419 across the 10 questions, well below the fine-tuned BERT baseline at 0.693. The authors attribute the poor performance to the niche political topics being absent from Mistral's training data, the model's tendency to produce inconsistent classification outputs or refuse to predict stance, and its verbose response style that does not align with single-label classification. They also note that other similar-sized open-source LLMs (Llama, OpenAssistant, Falcon) struggled similarly.

Evidence

correlational

Key metric

F1 score (avg. 10 questions): zero-shot LLM 0.419, baseline (fine-tuned BERT) 0.693

Caveat

The evaluation is on 10 selected questions from the X-Stance German test set, a narrow slice of the full dataset. The authors note that better prompting strategies could improve performance, and that the model's verbose outputs required accepting any response containing 'favor' or 'against'.

Model

Mistral 7B / Mistral / Mistral 3 7B / Mistral-0.2-7B / Mistral-v0.1

Concepts

Failure mode

Datasets

X-Stance [eval]

Methods

LoRA [supporting]

Extraction

automatic-extraction