Scholar·WanJuan 1.0
Multimodal Corpus
Scholar·WanJuan 1.0 Multimodal Corpus open-source version. Released by Shanghai Artificial Intelligence Laboratory, it includes three major types of data: text, images with text, and video, covering various types such as scientific literature, encyclopedias, community Q&A, textbooks, code, etc., with a total size exceeding 2TB, specifically designed for pre-training large language models and multimodal models.
Dataset Highlights
A large-scale multimodal pre-training corpus carefully constructed by the Shanghai Artificial Intelligence Laboratory
Multimodal Coverage
Covering three major modal types: text, images with text, and video, fully supporting the pre-training needs of large language models and multimodal models, with rich and diverse data forms.
High-Quality Text Corpus
The text data includes various types such as scientific literature, encyclopedic knowledge, community Q&A, textbooks, and code, rigorously cleaned and deduplicated, ensuring quality.
Ultra-Large Scale
The total data volume exceeds 2TB, with over 600 million text samples, more than 22 million image-text pairs, and over 1 million videos, meeting large-scale training needs.
Completely Open Source
Released under the Apache-2.0 license, supporting academic research and commercial use, with no concerns about license restrictions, freely accessible and usable.
Pre-training Ready
Designed specifically for LLM and multimodal model pre-training, with a unified standardized data format, directly usable in the training process of large models like InternLM.
Professionally Built Team
Professionally produced by the Shanghai Artificial Intelligence Laboratory (Shusheng·Puyu InternLM team), with rigorous and reliable data collection, cleaning, and quality control processes.
Applicable Scenarios
Widely applicable to various tasks in the AI field, from model pre-training to academic research
Pre-training of Large Language Models
Used for training large language models such as InternLM, providing rich and diverse text corpora to support model knowledge acquisition
Multimodal Model Training
Utilizing image-text pairs and video data to train visual-language understanding models, enhancing cross-modal capabilities
Academic Research
Supports research and experiments in fields such as natural language processing, computer vision, and multimodal learning
Domain Fine-tuning
Extracts domain data from subsets such as scientific literature, textbooks, and code for fine-tuning models in specific domains
Data Preview
The following is an overview of the data structure of the Wanjun 1.0 corpus
{
"text": [
{"type": "science", "samples": "100M+", "format": "jsonl"},
{"type": "encyclopedia", "samples": "50M+", "format": "jsonl"},
{"type": "qa_forum", "samples": "80M+", "format": "jsonl"},
{"type": "textbook", "samples": "30M+", "format": "jsonl"},
{"type": "code", "samples": "20M+", "format": "jsonl"}
],
"image_text": {
"samples": "22M+",
"format": "tar+jsonl",
"description": "# Image-text pair data, containing images and corresponding text descriptions"
},
"video": {
"samples": "1M+",
"format": "tar+jsonl",
"description": "# Video data, containing video clips and annotation information"
},
"license": "Apache-2.0",
"total_size": "~2TB",
"publisher": "Shanghai AI Lab (Shanghai Artificial Intelligence Laboratory)"
}
3 Steps to Get Started Quickly
Quickly integrate the Wanjun 1.0 dataset into your workflow from browsing to training
Browse the Dataset
View the details of the Wanjun 1.0 dataset on the Ace Data Cloud platform, and understand the metadata such as the scale, format, and licensing agreements of each modality subset.
Download or Stream
Download or stream the text, image-text, and video subset data files as needed, with support for selective retrieval by category.
Load into Training Pipeline
Load the data into your training pipeline, supporting mainstream frameworks like PyTorch DataLoader, and start model pre-training directly.
Start Exploring the Wanjun 1.0 Data
A large-scale multimodal open-source corpus, licensed under Apache-2.0, available now. Whether you are training large language models or multimodal models, Wanjun 1.0 is the ideal data foundation.