FallacyRAG | Deborah Dore

Paper Code Under review at EACL 2027 · with Greta Damo, Elena Cabrio and Serena Villata

The question

Spotting a fallacy takes more than the sentence itself: it needs knowledge of the topic under debate and of how the argument relates to the ones around it. Earlier work showed that argument structure helps, but only as fixed classifier features. Can the support and attack relations between arguments instead steer what a model retrieves, so that external knowledge actually helps fallacy detection?

How it works

Each sentence becomes a retrieval query: either the bare sentence, or the sentence enriched with the argument components that support or attack it. The query goes to a 15 GB knowledge base of political documents through one of four retrievers (three dense models or BM25), optionally restricted to documents that predate the debate and reranked by a cross-encoder. The retrieved passages, the sentence and its speech turn form the model input, and an encoder or an LLM makes the prediction.

What we found

Across 42 retrieval configurations and 14 models on ElecDeb60to20, argumentatively guided retrieval gives the best results for both tasks:

Task	Best without retrieval	Best with guided retrieval
Fallacy detection (macro-F1)	0.772	0.864
Fallacy classification (macro-F1)	0.653	0.725

Both best scores come from ModernBERT. Retrieval improves every encoder tested except NeoBERT, while decoder-only LLMs get worse once retrieved context is added. External knowledge helps when retrieval follows argument structure, but so far only encoder models benefit from it.

Paper

Retrieving Relations, Detecting Fallacies: A RAG Approach To Political Debate Analysis

Deborah Dore, Greta Damo, Elena Cabrio, and Serena Villata

In , 2026

Under review at EACL 2027

Abs arXiv Code

Fallacies are arguments that employ invalid reasoning, making their automatic detection critical in sensitive contexts such as high-stakes political debates, where public opinion is shaped. Spotting a fallacious argument requires contextual knowledge beyond its pure surface text. This entails world knowledge pertaining to the subject matter under discussion, as well as knowledge of the relationships that exist between arguments within the argumentative discourse. Prior work on fallacy analysis has shown that argumentative discourse structure can beneficially improve classification performance. However, such structure is typically encoded only as static classifier features, limiting its flexibility. Building on this intuition while addressing this limitation, we introduce a guided retrieval-augmented methodology for fallacy detection and classification that leverages argumentative relations of support and attack to dynamically steer the extraction of relevant documents. We evaluate our approach on the ElecDeb60to20 benchmark across 42 retrieval configurations and 14 models, performing retrieval over a 15GB knowledge base of collected political-related documents. Our approach improves macro-F1 up to 0.864 for fallacy detection and up to 0.725 for classification over non-retrieval baselines. These results show that incorporating external knowledge significantly enhances fallacy detection and classification when retrieval is argumentatively guided.

← All projects