arxiv-latex

arXiv LaTeX Source Dataset This dataset provides the entire corpus of arXiv's LaTeX source files, pre-parsed, formatted, and aligned with official metadata in ready-to-query Parquet files. Why I Built This If you have ever tried to work with the complete history of arXiv papers at scale, you have likely run into two massive hurdles: 1. **Network Egress Costs**: While arXiv does offer public bulk access to its source files via S3 (`s3://arxiv`), the bucket is configured as "requester-pays." If you attempt to download the full 5 TB corpus of over 3 million papers to any machine outside of the AWS `us-east-1` (N. Virginia) region, you are hit with standard AWS egress fees. At **$0.09 per GB**, a single full download costs **more than $450**. 2. **Computational Friction**: The raw S3 data is packaged as hundreds of nested `.tar` archives, each containing gzip payloads (`.gz`) of individual papers. Extracting these, parsing the inner LaTeX code, and matching the files with their JSON metadata snapshots is extremely CPU-heavy, requiring complex local pipeline architecture. This dataset acts as an open mirror that solves both issues. The project ingests the S3 data inside `us-east-1` (where data transfer is free), indexes and processes the LaTeX source documents, aligns them directly to their metadata snapshot, and uploads the finalized Parquet files here. Researchers and developers can download clean, structured data without worrying about network egress bills or spending days writing ingestion code. Sync Automation & Tooling The pipeline that builds and updates this dataset is completely open-source. If you wish to mirror this data, customize the processing logic, or run local extractions directly from AWS S3, you can use the sync infrastructure codebase: **GitHub Repository**: arthiondaena/arxivETL_sync This accompanying repository features a high-performance **parallel Rust engine** paired with a Python orchestrator designed to handle zero-copy memory-mapped metadata p