EuroBERT – Advancing Multilingual NLP Through Open Collaboration

Multilingual NLP remains a persistent challenge. While large generative models like GPT-4 have transformed text generation, encoder-based models are still essential for search engines, document retrieval, and classification systems. However, existing multilingual encoder models often struggle with performance inconsistencies, short context windows, and limited domain adaptability, making them inefficient for real-world applications.

EuroBERT changes that. It is a great example of collaboration between government, the private sector, and academia, demonstrating how open research and cross-border cooperation can drive real innovation. Developed through a cross-European effort, EuroBERT brings together MICS Laboratory at CentraleSupélec, Diabolocom, Artefact, Unbabel, Instituto Superior Técnico & Universidade de Lisboa, with technological support from AMD and CINES and funding from the France 2030 program. By combining expertise from machine learning researchers, industry engineers, and public institutions, the project delivers a practical, high-performance multilingual encoder model that overcomes the limitations of its predecessors, including BERT, ModernBERT, and XLM-RoBERTa.

EuroBERT is designed to push the boundaries of efficiency, scalability, and multilingual precision. It is the most advanced multilingual encoder model for retrieval (RAG), classification, and quality estimation (e.g., summarization, translation) across 15 languages. With support for 8 major European languages and 7 of the world’s most spoken non-European languages, it ensures broad linguistic coverage. Unlike earlier models, EuroBERT introduces a long-context window of 8,192 tokens, allowing it to process entire documents more effectively than the standard 512-token limit seen in previous encoder models.

With 5 trillion tokens of training data—twice as much as typical encoder models and even surpassing generative models like Llama 2 (2 trillion tokens)—EuroBERT delivers unparalleled linguistic understanding without additional usage costs. Unlike previous models with limited transparency, it is fully open-source, with all training checkpoints, datasets, and code available for public use.

This initiative sets a powerful precedent for collaborative AI development, proving that diverse institutions working together can achieve results beyond what any single entity could accomplish alone. No single institution or country can tackle these challenges alone. Other countries should take note—this model of cooperation can be replicated to build high-quality language models that better serve local linguistic and cultural needs. By organizing similar efforts, nations can strengthen their AI ecosystems and ensure their languages remain well-represented in the rapidly evolving world of NLP.

Why EuroBERT Matters

Despite the progress in multilingual NLP, most encoder models have struggled with three major issues:

1. Limited Multilingual Coverage – Many models either focus on a single language (like the original BERT) or struggle with performance disparities across languages.
2. Short Context Windows – Models like BERT are constrained to sequences of 512 tokens, making them ineffective for processing long documents.
3. Lack of Domain Adaptability – General-purpose models perform well in casual text but often fail in structured fields like programming and mathematics.

EuroBERT addresses these challenges by introducing a multilingual training approach, improved architectural efficiency, and a significantly longer context window.

How EuroBERT Improves on Previous Models

EuroBERT builds upon ModernBERT and previous multilingual encoder models like XLM-RoBERTa. The key advancements include:

1. Expanded Multilingual Training

EuroBERT was trained on a 5-trillion-token dataset spanning 15 languages, ensuring better linguistic representation. This diverse dataset includes high-quality sources, reducing biases and improving cross-lingual transfer learning.

2. Enhanced Architecture

Several design improvements contribute to EuroBERT’s efficiency:

• Grouped Query Attention (GQA) – Increases inference speed and reduces memory usage, an upgrade borrowed from transformer architectures used in LLaMA-2.
• Rotary Position Embeddings (RoPE) – Enables improved handling of long-context dependencies.
• Root Mean Square (RMS) Normalization – Stabilizes training across different languages, improving generalization.

3. Longer Context Support

Unlike BERT (512-token limit) and ModernBERT, EuroBERT supports sequences up to 8,192 tokens, making it suitable for document retrieval, long-form summarization, and legal text analysis.

4. Domain-Specific Knowledge

EuroBERT incorporates additional training on mathematical and programming data, making it more effective for tasks like code search and structured reasoning. This feature makes it particularly useful for developers, data scientists, and industries that require structured data interpretation.

Benchmark Performance

EuroBERT has been evaluated across several NLP benchmarks, consistently outperforming previous multilingual encoder models:

• Multilingual Retrieval (MIRACL, Wikipedia, CC-News) – Improved document ranking and search relevance.
• Text Classification (XNLI, PAWS-X, Amazon Reviews) – Higher accuracy in natural language inference and sentiment analysis.
• Regression Tasks (SeaHorse, WMT, SummEval) – Better performance in similarity evaluation.
• Mathematical and Code Understanding (MathShepherd, CodeSearchNet) – Enhanced reasoning and search capabilities in structured fields.

These results shown on the EuroBERT Github indicate that EuroBERT is not just a theoretical improvement—it provides tangible performance benefits across a range of real-world NLP applications.

Practical Applications for Businesses and Developers

EuroBERT’s improvements have direct implications for both enterprises and technical teams:

For Businesses

• Multilingual AI Models Without High Costs – Companies can deploy a single model instead of maintaining multiple monolingual ones, reducing infrastructure complexity.
• Better Customer Support Automation – EuroBERT’s multilingual capabilities improve chatbot performance and cross-lingual document retrieval.
• Regulatory Compliance & Risk Analysis – Legal teams can analyze large volumes of text (e.g., GDPR compliance) using the long-context capabilities.

For Developers

• Easy Model Adoption – Pretrained versions of EuroBERT (ranging from 210M to 2.1B parameters) are available on Hugging Face, allowing for rapid integration.
• Customization & Fine-Tuning – Developers can fine-tune EuroBERT for specialized industry applications without needing to train models from scratch.
• Efficient Hardware Support – Optimized for both NVIDIA CUDA and AMD ROCm, making it flexible for different GPU architectures.

The Open-Source and Cross-Border Collaboration Behind EuroBERT

EuroBERT’s development is a testament to the power of open collaboration. Unlike proprietary AI models developed in isolation, EuroBERT is the result of a coordinated effort across academia, industry, and government, demonstrating how pooling resources and expertise can accelerate NLP advancements.

I want to personally extend my deepest appreciation and congratulations to the entire EuroBERT team for their outstanding work in developing this groundbreaking multilingual encoder model. Your commitment to innovation, collaboration, and open research has resulted in a tool that will have a lasting impact on the NLP community.

By making EuroBERT fully open-source, you have not only pushed the boundaries of multilingual AI but have also given a valuable gift to researchers, developers, and businesses worldwide. The transparency, accessibility, and performance of this model set a new standard for what multilingual NLP can achieve.

This project brought together leading researchers, engineers, and AI practitioners from institutions across France, Portugal, and other European countries, including:

• MICS Laboratory (CentraleSupélec)?
• Diabolocom
• Artefact
• Unbabel
• Instituto Superior Técnico & Universidade de Lisboa

By openly sharing datasets, training methodologies, and model architectures, EuroBERT ensures that the entire AI community—not just a select few organizations—can benefit from and build upon its innovations. The decision to make EuroBERT fully open-source, with all training checkpoints, datasets, and implementation details available, fosters transparency, reproducibility, and continuous improvement.

Why Open-Source Matters for NLP Development

Unlike closed models, which limit access and restrict adaptation, EuroBERT’s open-source nature allows:

• Community-Driven Enhancements – Researchers and developers worldwide can contribute improvements, fine-tune it for specialized domains, and optimize its efficiency.
• Adaptation to Local Languages & Industries – Countries and businesses can modify EuroBERT for underrepresented languages or tailor it to specific industry needs, such as healthcare, finance, and law.
• Ethical AI Development – Open access ensures greater transparency, allowing the community to audit biases, improve fairness, and address ethical concerns in multilingual AI.

A Model for Future Collaboration

The success of EuroBERT is a clear example of how cross-border and cross-sector partnerships can drive meaningful AI progress. Governments, research institutions, and enterprises worldwide should take note—by working together and prioritizing open research, they can build stronger, more inclusive AI systems that serve a broader range of users and linguistic communities.

By fostering a global, open-source NLP ecosystem, EuroBERT lays the groundwork for future innovations that will further improve multilingual AI and democratize access to powerful language models.

Why Other Countries and Languages Should Follow EuroBERT’s Example

While EuroBERT is a major breakthrough in multilingual NLP, it is only the beginning. Many languages around the world remain severely underrepresented in AI models, making it difficult for speakers of these languages to access high-quality NLP tools. If AI is to be truly inclusive, it must go beyond a handful of dominant languages and ensure global linguistic diversity is preserved and empowered.

Underrepresented Languages in AI

Most large-scale language models are disproportionately trained on English, Chinese, and a few widely spoken European languages, while many African, Indigenous, and regional languages have little to no representation in AI systems. This has serious implications:

• Marginalization of smaller languages – Without AI support, low-resource languages risk becoming obsolete in the digital world.
• Poor machine translation and voice recognition – Limited training data means these languages often have subpar NLP performance, reducing accessibility for native speakers.
• Exclusion from AI-driven economies – Nations without strong AI capabilities in their local languages may fall behind in automation, customer support, education, and other AI-powered fields.

Community Collaboration: The Key to Expanding Language Coverage

The success of EuroBERT proves that open-source, cross-sector collaboration is the most effective way to build powerful, multilingual AI models. No single company, institution, or country can tackle this alone. By working together—pooling research efforts, sharing datasets, and refining models—the global NLP community can ensure that every language has a place in AI.

Governments, researchers, and developers should take inspiration from EuroBERT and:

1. Invest in Open-Source NLP Projects – Open research enables transparency, accessibility, and continuous improvement.
2. Curate and Share Diverse Datasets – High-quality, linguistically diverse training data is critical for fair and effective AI models.
3. Develop Models for Low-Resource Languages – More funding and research should be directed toward languages that lack digital representation.
4. Encourage Multinational AI Research – Cross-border cooperation leads to better, more representative models, as demonstrated by EuroBERT.

Time for Language Groups to Team Up and Collaborate

The success of EuroBERT proves that collaborative, open-source NLP models are the best way to advance multilingual AI. However, many languages are still severely underrepresented, leaving billions of people without access to high-quality AI-driven language tools. The next step is for linguistic communities, researchers, and institutions to team up and develop specialized encoder models that cover more regions, more languages, and more domains.

There are already some regional and domain-specific BERT models in existence, demonstrating the growing demand for specialized NLP solutions, such as:

• XLM-R (Cross-lingual RoBERTa) – A multilingual encoder model trained on 100+ languages.
• CamemBERT – A French-focused BERT model trained on extensive French-language corpora.
• AraBERT – A BERT model tailored for Arabic, supporting both Modern Standard Arabic and dialects.
• AfriBERTa – One of the few African-focused models, trained on 11 African languages.
• SciBERT – An encoder model optimized for scientific literature and research papers.

While these models serve specific linguistic and industry needs, there are still huge gaps in many underrepresented languages—especially African, Indigenous, and low-resource Asian languages.

?? Not enough regional and domain-specific BERT models exist today.

?? Many languages remain completely unsupported.

?? It’s time to team up, collaborate, and build new models to fill these gaps.

Regional Multilingual Encoder Models to Build

1. AsiaBERT+ ?? Covers a wide range of Asian languages across East Asia, South Asia, Southeast Asia, and Central Asia, including:

• East Asian: Mandarin, Cantonese, Japanese, Korean
• South Asian: Hindi, Bengali, Tamil, Telugu, Punjabi, Marathi, Gujarati, Urdu, Sinhala
• Southeast Asian: Thai, Vietnamese, Tagalog, Khmer, Burmese, Lao, Javanese, Malay, Sundanese
• Central Asian: Kazakh, Uzbek, Turkmen, Kyrgyz, Pashto, Dari, Tajik
• Addresses script diversity challenges (logographic, abugida, syllabic, and alphabetic scripts).
• Useful for cross-lingual search, legal and financial AI, and government applications in Asia.

2. AfricaBERT ?? Supports both widely spoken and low-resource African languages, including:

• Major languages: Swahili, Yoruba, Hausa, Amharic, Somali, Igbo, Wolof, Zulu, Xhosa
• Low-resource languages: Shona, Ewe, Tswana, Berber, Tigrinya, Malagasy
• Critical for bridging AI gaps in African governance, translation, and commerce.

3. LatAmBERT ?? Tailored for Latin American Spanish, Brazilian Portuguese, and Indigenous languages such as Quechua, Guarani, and Nahuatl.

• Helps improve government NLP systems, legal document retrieval, and regional business applications.

4. SlavBERT ?? Covers Slavic languages including Russian, Polish, Czech, Slovak, Ukrainian, Bulgarian, Serbian, and Croatian.

• Useful for legal research, multilingual search, and translation across Slavic nations.

5. IndoBERT ???????????? Focuses on South Asian languages, such as Hindi, Bengali, Tamil, Telugu, Punjabi, Marathi, Gujarati, Urdu, and Sinhala.

• Crucial for e-governance, multilingual AI in commerce, and cross-border communication in the Indian subcontinent.

6. ArabBERT++ ?? Expands on existing ArabBERT to include Modern Standard Arabic, regional Arabic dialects, and related languages like Persian and Kurdish.

• Tackles dialectal variance, multilingual translation, and search engine optimization in Arabic-speaking regions.

Domain-Specific Encoder Models

While general multilingual encoders like EuroBERT and XLM-R are designed to handle broad language tasks, domain-specific encoder models are optimized for specialized fields where precision, context awareness, and technical language are critical. These models, such as MedBERT for healthcare, LegalBERT for law, SciBERT-X for scientific research, and FinBERT-World for finance, are trained on highly structured, domain-specific datasets, ensuring they understand industry terminology, regulatory nuances, and contextual dependencies far better than general-purpose models. By fine-tuning on professionally curated data, these encoders enable more accurate legal text retrieval, medical diagnosis automation, patent analysis, and financial risk assessment, making AI applications safer, more reliable, and more effective in professional environments.

1. MedBERT ?? Optimized for medical texts, electronic health records (EHRs), and clinical documentation. Supports multilingual medical research and AI-driven healthcare tools.
2. LegalBERT-Global ?? Trained on legal texts, case law, and international regulations. Helps with automated policy analysis, legal research, and compliance automation.
3. SciBERT-X ?? Enhances scientific literature search, patent analysis, and technical document retrieval. Supports multilingual academic publishing, research synthesis, and open-access knowledge sharing.
4. FinBERT-World ?? A financial AI model optimized for banking, risk analysis, and multilingual market insights. Helps financial institutions and regulators process cross-border finance documentation.

Why These Models Matter

• Many languages remain critically underrepresented—billions of people still lack access to accurate NLP tools.
• Cross-lingual search, document retrieval, and translation quality need urgent improvement for low-resource languages.
• Industry-specific models (medical, legal, financial) must support more languages to ensure global AI accessibility.

Start Building Today

The best way to kickstart multilingual NLP development is to leverage the tools already available. The EuroBERT GitHub repository offers a strong foundation for multilingual encoder models, providing:

? Pretrained models

? Training pipelines

? Multilingual benchmarks

?? Check out the EuroBERT GitHub for tools and resources: EuroBERT GitHub

By building on top of existing open-source work, the community can accelerate progress, expand NLP inclusivity, and develop AI that serves speakers of all languages—not just those that dominate today’s models. The next multilingual encoder model could be yours—let’s build it together! ??

Once We Get More Models, What is Next? Can We Merge Them All?

Merging Existing Multilingual Encoder Models into One Large Model: What Would It Take?

Combining regional and domain-specific BERT models such as those listed above (e.g., EuroBERT, XLM-R, AraBERT, CamemBERT, AfriBERTa, SciBERT) into one massive, unified multilingual encoder would be a complex but potentially valuable endeavor. Here’s what it would take and the potential pros and cons of such an approach.

Steps to Merge Existing Models & Datasets

1. Standardizing & Aligning Datasets

Challenge:

• Each model was trained on different corpora, with different preprocessing techniques. To merge, we need standardized tokenization, filtering, and balancing across languages.

Solution:

• Develop a universal dataset pipeline to normalize text sources.
• Use multilingual high-quality datasets (e.g., Common Crawl, Wikipedia, OSCAR, CC-News, academic papers).
• Ensure equal representation across low-resource and high-resource languages to avoid bias.

2. Unifying Model Architectures & Training Objectives

Challenge:

• Different BERT variants use different architectures (e.g., RoBERTa, ALBERT, ELECTRA adaptations). Some models (like SciBERT) are domain-specific.

Solution:

• Choose a consistent architecture—likely an efficient Transformer with optimized Grouped Query Attention (GQA) and Rotary Position Embeddings (RoPE) for scalability.
• Fine-tune submodels for specific linguistic groups while maintaining a shared backbone for multilingual learning.

3. Computing Resources for Training a Massive Model

Challenge:

• Training a single merged model would require exponentially more computational resources than training individual models.

Solution:

• Utilize distributed training across multiple data centers (e.g., compute clusters from research institutions and cloud providers).
• Leverage sparse architectures (e.g., Mixture of Experts, efficient attention mechanisms) to scale without massive compute costs.

4. Handling Tokenization & Vocabulary Expansion

Challenge:

• Each model has different vocabulary sizes and tokenization strategies.

Solution:

• Adopt a single, unified tokenizer with subword segmentation (SentencePiece or WordPiece) that dynamically supports multiple scripts and character sets.
• Ensure it can handle agglutinative languages (e.g., Finnish, Turkish) and logographic languages (e.g., Chinese, Japanese).

5. Preventing Overfitting to High-Resource Languages

Challenge:

• If merged without careful balance, high-resource languages (e.g., English, Chinese, French) could dominate, degrading performance for low-resource languages.

Solution:

• Apply data sampling techniques to boost underrepresented languages during training.
• Use adaptive training schedules where low-resource languages receive higher weighting in fine-tuning phases.

Pros & Cons of a Merged Global Multilingual Encoder Model

? Pros (Why It Might Be Worth It)

1. A Single, Unified Model for Global NLP
2. Improved Cross-Language Understanding
3. More Efficient Than Training Separate Models
4. Better Domain Adaptation

? Cons (Challenges & Drawbacks)

1. Massive Computational Cost
2. Balancing Low-Resource & High-Resource Languages Is Hard
3. Storage & Deployment Complexity
4. Potential for Lower Task-Specific Performance

Alternative: A Modular Approach Instead of One Monolithic Model

Instead of one massive model, a better approach could be a "Modular BERT Ecosystem", where:

• A shared base model handles general multilingual encoding.
• Specialized language-family submodels (e.g., AsiaBERT, AfricaBERT, IndoBERT) fine-tune on their respective language clusters.
• Domain experts train task-specific heads (e.g., LegalBERT, MedBERT, SciBERT) while still leveraging the shared backbone.

This would allow scalability, efficiency, and adaptability while avoiding the pitfalls of a single, monolithic multilingual model.

Final Thoughts: Should We Merge Models?

A global multilingual encoder is possible, but it would require massive computing resources, thoughtful dataset balancing, and modular design to be practical. Instead of a single model, a hierarchical system of regional and domain-specific models could offer the best of both worlds—combining scalability with adaptability.

?? The key is collaboration—merging models will require open-source contributions from multiple countries, institutions, and research groups.

The Bottom Line: Open Collaboration is the Future of Multilingual NLP

EuroBERT is more than just another NLP model—it’s a proof of concept for what is possible when governments, academia, and industry collaborate across borders. It provides a high-performance, scalable solution for multilingual retrieval, classification, and structured information processing, addressing the limitations of previous encoder-based models such as BERT, ModernBERT, and XLM-RoBERTa.

With support for 15 languages, a 5-trillion-token training dataset, and a long-context processing capability of 8,192 tokens, EuroBERT establishes a new standard for multilingual NLP. By making it fully open-source, with all training checkpoints, datasets, and code publicly available, this project ensures that cutting-edge NLP technology is accessible and adaptable for global use.

But EuroBERT is also a call to action. No single institution or country can tackle these challenges alone—other nations must take note and act now. The dominance of a few languages in AI models risks leaving others behind. If countries want their languages to be properly represented in future AI systems, they must invest in large-scale, open collaborations to develop sovereign, high-quality multilingual models that serve their own linguistic and cultural needs.

The success of EuroBERT proves that real breakthroughs happen when governments, researchers, and industry align their efforts toward a common goal. The future of multilingual AI will not be built by observers, but by those who organize, collaborate, and contribute. The time to act is now.

Resources

• ?? Research Paper: arXiv:2503.05500
• ??? Pretrained Models: Hugging Face Model Hub
• ??? Training Code: GitHub Repository