RooseBERT | Deborah Dore

Paper Models Code WI-IAT 2026 · with Elena Cabrio and Serena Villata

The question

Political debates rely on hidden communication strategies and implicit arguments, and their vocabulary is specialised. General-purpose language models struggle with that. Does pre-training on political discourse produce a model that analyses debates better than general-purpose ones?

What we built

RooseBERT is pre-trained on 11 GB of English political debates and speeches: parliamentary proceedings from the UK, Ireland, Scotland, Canada, Australia, New Zealand, Ghana and South Africa, EU and UN speeches, and US presidential debates from 1960 to 2024. We trained it in two ways, each cased and uncased:

Model	Strategy
`RooseBERT-cont-cased`	Continued pre-training from BERT, original vocabulary
`RooseBERT-cont-uncased`	Continued pre-training from BERT, original vocabulary
`RooseBERT-scr-cased`	From scratch, with a political-domain WordPiece vocabulary
`RooseBERT-scr-uncased`	From scratch, with a political-domain WordPiece vocabulary

Training from scratch gives a vocabulary where political terms are single tokens instead of being split into word pieces.

What we found

We fine-tuned RooseBERT on stance detection, sentiment analysis, argument component detection and classification, argument relation prediction and classification, policy classification and named entity recognition. We compared it with BERT, ModernBERT, ConfliBERT and PoliBERTweet. RooseBERT improves significantly over the general-purpose models on most of these tasks, which shows that domain-specific pre-training pays off for political debate analysis.

Use it

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("MARIANNE-INRIA/RooseBERT-scr-cased")
model = AutoModel.from_pretrained("MARIANNE-INRIA/RooseBERT-scr-cased")

Paper

RooseBERT: A New Deal For Political Language Modelling

Deborah Dore, Elena Cabrio, and Serena Villata

In The 25th IEEE/WIC International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT 2026), Brisbane, Australia, 2026

Abs arXiv Code Website

The increasing amount of political debates and politics-related discussions calls for the definition of novel computational methods to automatically analyse such content with the final goal of lightening up political deliberation to citizens. However, the specificity of the political language and the argumentative form of these debates (employing hidden communication strategies and leveraging implicit arguments) make this task very challenging, even for current general-purpose pre-trained Language Models (LMs). To address this, we introduce a novel pre-trained LM for political discourse language called RooseBERT. Pre-training a LM on a specialised domain presents different technical and linguistic challenges, requiring extensive computational resources and large-scale data. RooseBERT has been trained on large political debate and speech corpora (11GB) in English. To evaluate its performances, we fine-tuned it on multiple downstream tasks related to political debate analysis, i.e., stance detection, sentiment analysis, argument component detection and classification, argument relation prediction and classification, policy classification, named entity recognition (NER). Our results show significant improvements over general-purpose LMs on the majority of these tasks, highlighting how domain-specific pre-training enhances performance in political debate analysis.

← All projects