AI data preparation: what it is and how to do it in six steps

• AI data preparation is the work of turning raw records, documents and web pages into a dataset a model can learn from or retrieve from: collect, clean, label, split, transform, optimize.
• Data is the usual point of failure. A 2024 RAND report puts the AI project failure rate above 80%, and 30 of its 50 interviewees named data quality as a persistent problem.
• It eats the schedule. Anaconda's 2022 survey found data scientists spend about 38% of their time on preparation and cleansing.
• RAG, fine-tuning and inference need different prep. RAG wants chunks with metadata, with 256 to 512 tokens as a starting range to tune per task; fine-tuning wants JSONL instruction pairs; inference wants cached, PII-safe inputs.
• Format matters. Converting raw HTML to Markdown cut the token count by 70% to 98% on three pages we measured in October 2026.
• Firecrawl collects web pages and documents and returns Markdown or JSON, so the collection and transformation steps collapse into one call. No API key is needed to start.

Ask an engineer why their AI project stalled and the answer is rarely the model. A 2024 RAND report interviewed 50 practitioners about why AI projects fail. By some estimates more than 80% do, twice the rate of IT projects without AI, and 30 of the 50 interviewees described persistent data quality problems.

Andrej Karpathy, former Director of AI at Tesla, gave the same advice in A Recipe for Training Neural Networks:

The first step to training a neural net is to not touch any neural net code at all and instead begin by thoroughly inspecting your data.

Teams skip this, jump to modeling, find the data is broken, and spend weeks backtracking. The preparation work below is what avoids the backtrack, and it changes shape depending on whether the data is for RAG, fine-tuning or inference.

AI data preparation is the process of turning raw data from databases, APIs, documents and web pages into a dataset that an AI model can train on or retrieve from. It covers collecting the data, cleaning it, transforming it into the format the model expects, labeling examples, splitting the data into training and test sets, and optimizing it for the target workload.

The pipeline usually runs in this order:

• Collect data from internal systems, third-party APIs, documents and the web
• Clean by removing duplicates, standardizing formats and dropping unrepairable records
• Label examples for supervised learning
• Split into training, validation and test sets, then validate quality
• Transform into the structure the model expects (JSON, Markdown, imputed and encoded features), fitting on the training split, and augment where data is thin
• Optimize for the workload: chunks for RAG, JSONL pairs for fine-tuning, cached inputs for inference

Preparing data for AI differs from the preparation that goes into reports and dashboards. The table shows where.

The unstructured half is where most of the new work sits. Business intelligence pipelines rarely have to turn a 40-page PDF or a JavaScript-rendered product page into something a model can read.

Data preparation decides whether a model learns the pattern you want or the noise you left in. A model trained on duplicated, mislabeled or stale records reproduces those faults at scale, and a RAG system that retrieves from badly chunked documents answers confidently from the wrong context. The cost shows up as failed projects, wasted compute and wrong decisions.

Three numbers put a size on it:

• RAND's 2024 interviews found data quality to be the most common complaint, raised by 30 of 50 practitioners. One interviewee summarized the job as "80 percent of AI is the dirty work of data engineering."
• Gartner research from 2020, still cited on its data quality page, puts the cost of poor data quality at $12.9 million a year for the average organization.
• Anaconda's 2022 State of Data Science report found respondents spend about 38% of their time on data preparation and cleansing, split 22% and 16%.

The gap between coursework and production surprises many developers. One Hacker News commenter put it this way:

Universities and online challenges provide clean labeled data, and score on model performance. The real world will provide you... 'real data' and score you by impact.

Data is ready for an AI model when it is relevant to the task, free of duplicates and errors, consistent across sources, structured in the format the model consumes, and documented well enough that you can trace any record back to where it came from. Volume matters less than these five properties.

• Relevant: every record relates to the problem the model solves. Dumping an entire data warehouse into a vector database adds retrieval noise and cost.
• Clean: duplicates, missing values and outliers have been handled deliberately, with the decisions recorded.
• Consistent: dates, currencies, units, encodings and category labels match across every source.
• Structured for the consumer: JSONL for fine-tuning, chunked Markdown with metadata for RAG, encoded features for tabular models.
• Traceable: each record carries its source, ingestion time and processing version, so a bad output can be traced to a bad input.

The recurring obstacles are data quality, unstructured sources, privacy, bias, scale and the lack of a shared schema. Most teams hit all six, and the unstructured-source problem is the one that grows fastest as more training and retrieval data comes from documents and the web.

• Data quality: source data arrives incomplete, inconsistent or outdated. Missing values, duplicates and conflicting formats have to be found and handled before training starts.
• Unstructured sources: web pages, PDFs and office documents carry navigation menus, ads, scripts and layout artifacts that a model should never see. Extracting the content takes rendering and parsing work that a SQL export never needed.
• Privacy and security: training data often contains personal information. GDPR and CCPA restrict what you can collect and how long you can keep it, so PII detection and retention rules belong in the pipeline from day one.
• Bias: skewed class distributions push models toward majority outcomes, and historical bias in labels carries forward. Catching it takes audits that repeat every time the dataset changes.
• Scale: datasets outgrow memory. Batching, streaming and distributed processing become necessary long before the modeling work gets harder.
• No standard schema: data arrives as CSV, JSON, XML, HTML and proprietary exports, with no universal format for training data. Teams end up writing a parser per source.

Preparing data for AI follows six steps: collect from your sources, explore and clean, label, split and validate, transform and augment, then optimize for the workload. The split comes before any fitted transform so nothing from validation or test data leaks into training. Each step depends on the previous one, and problems skipped early reappear later as bad predictions or bad retrievals.

Each source needs its own extraction method.

Internal data lives in databases, warehouses and existing datasets. SQL exports, internal APIs and direct connections bring it in. The usual problems are access permissions and working from stale snapshots instead of production-quality data.

External APIs return weather, financial, social or industry data as JSON. Plan for authentication, rate limits, pagination and error recovery, and expect to flatten nested structures.

Indexed and licensed data covers the sources that already come back structured. Through Firecrawl Alexandria, an API key on a team with Firecrawl Alexandria enabled reaches third-party data providers and Firecrawl's own indexes: the Research Index of about 43 million paper abstracts and the Developer Index of 70 million or more code and documentation pages. Records arrive as typed JSON with their source, so the cleaning step below gets shorter. Discovery is free and each capability shows its price in credits before you call it. Keyless sessions do not include the Firecrawl Alexandria providers. The Government & Legal Index of 30 million US public pages works differently: its searches are free, use no credits and need no API key. Every paper cited in this guide was found through the Research Index.

Documents such as PDFs, Word files and spreadsheets hold unstructured content that is awkward to extract. PDF layouts prioritize appearance over structure, and scanned pages need OCR before any text comes out. Libraries like PyMuPDF, pdfplumber and python-docx cover common cases but leave edge cases for manual cleanup. For documents on disk, Firecrawl's document parsing endpoint accepts PDF, DOCX, XLSX, PPTX and other uploads and returns Markdown, trying text extraction first and falling back to OCR on pages that need it. The best PDF parsers guide and the best document parsing APIs guide compare the open-source and managed options.

Web data needs scraping or crawling. Scraping pulls one page; crawling follows links across a site, which is what most training and retrieval sets need. For a worked example from search to validated JSONL, see how to build an AI training dataset from the web. Beautiful Soup handles simple HTML and Playwright handles JavaScript-heavy sites, but you then own rendering, retries and parsing for every site. A web data API handles rendering and format conversion in one step. When you already have a list of URLs, batch URL extraction converts them in parallel. If you are new to this, start with the web scraping beginner's guide.

Whatever the source, record provenance and respect access restrictions. For web data, check robots.txt and the site's terms before collecting at scale.

Look before you clean. Profile each field for missing rates, value ranges, distributions and outliers so the cleaning decisions are based on what the data contains rather than on assumptions. This exploratory pass also catches structural surprises, like a date column stored as three different formats.

Cleaning then covers three tasks: removing duplicates, standardizing formats and dropping what cannot be repaired.

Duplicates overweight repeated examples during training, and the damage is larger than it looks. Lee et al. found a single 61-word sentence repeated more than 60,000 times in the C4 corpus, and showed in Deduplicating Training Data Makes Language Models Better that deduplication cut verbatim memorized output ten-fold and reduced train-test overlap that affected over 4% of standard validation sets. Hernandez et al. measured the other direction in Scaling Laws and Interpretability of Learning from Repeated Data: repeating 0.1% of the data 100 times degraded an 800M-parameter model to the performance of a model half its size. Hash-based deduplication catches exact matches; fuzzy matching catches near-duplicates with minor text differences.

Missing values get a first decision here. Drop rows or columns where more than about half the values are missing, and flag the rest. Filling the remaining gaps by imputation is a fitted transform, so it waits until after the split in step 5.

Format standardization covers dates, currencies, units, text encodings and category spellings that drift when data comes from several sources.

Supervised learning needs labels, the correct answers the model learns to predict. The label type follows the task:

Label quality matters as much as label count. Use several annotators on subjective tasks and measure agreement with Fleiss' kappa or Krippendorff's alpha (Cohen's kappa only compares two annotators at a time); low agreement means the guidelines are ambiguous, not that the annotators are careless. Active learning cuts cost by asking people to label only the examples the model is least sure about.

For fine-tuning, a few hundred well-written instruction-response pairs usually beat thousands of mediocre ones. The reference result is LIMA, a 65B LLaMA model fine-tuned on 1,000 curated prompts and responses with no reinforcement learning; in a human study its answers were equivalent to or preferred over GPT-4's in 43% of cases. A 2024 survey of data selection for instruction tuning collects the follow-up work and reaches the same conclusion: dataset quality matters more than quantity at this stage. The guide to custom instruction datasets walks through building them from documentation.

Split the data now, before any fitted transform or augmentation, so nothing learned from validation or test records can leak into training and you can measure how the model behaves on examples it has never seen. The common split is 80% training, 10% validation and 10% test. Use stratified sampling to keep class balance in each set, and a time-based split when the data has a time order, so the model is never tested on records older than the ones it trained on. The test set should mirror the production distribution rather than being a random slice of whatever was collected.

Then validate the prepared data against three checks:

• Completeness: are required fields populated?
• Consistency: do values fall in expected ranges?
• Accuracy: do spot checks against source data hold up?

Audit for bias here as well. Check class distributions and demographic representation, because imbalance at this stage becomes skewed behaviour in production. Finish with schema validation. Define the expected fields, types and constraints, and reject or flag records that break them.

Three formats cover most cases: JSON for structured records, plain text for simple sequences, and Markdown for content with hierarchy.

Markdown is the right target for most LLM work. It keeps headings, lists and emphasis while dropping tags, scripts and styling, and it appears throughout model training data, so models read it without special handling. Format is not cosmetic: He et al. formatted the same inputs as plain text, Markdown, JSON and YAML in Does Prompt Formatting Have Any Impact on LLM Performance? and saw GPT-3.5-turbo's results on a code translation task swing by up to 40% between templates, with GPT-4 less sensitive. For web content, converting pages to Markdown is the standard step before chunking or fine-tuning.

The size difference is large. On October 28 2026 we scraped three pages as raw HTML and as Markdown and counted tokens with OpenAI's o200k_base tokenizer:

The pages with heavy navigation and tracking shrink the most. Other tokenizers will give different absolute counts, but the ratio holds: most of a raw page is markup the model pays for and gets nothing from.

For tabular data, transformation means encoding categorical variables. One-hot encoding suits low-cardinality nominal features. Integer label encoding only fits categories with a real order (small, medium, large), because a numeric split treats the assigned values as ordered; for unordered categories use one-hot encoding or a tree library with native categorical support such as LightGBM or CatBoost. High-cardinality features do better with target encoding or learned embeddings. Scale numeric features so large-magnitude columns do not dominate.

With the split in place, fill the missing values you flagged in step 2. The options per field:

• Imputation: fill gaps with the mean, median or a predicted value
• Indicators: add a feature that flags the gap

Fit any imputation logic on the training set only, then apply it to validation and test sets. Fitting on the full dataset leaks information and inflates your metrics. Kapoor and Narayanan's survey Leakage and the Reproducibility Crisis in ML-based Science traced this class of error through 329 papers across 17 fields, several of which reported results that failed to reproduce once the leak was closed.

Augmentation fills gaps when examples are scarce or classes are imbalanced. For text: synonym replacement, back-translation (translate out and back to get a paraphrase) and sentence shuffling. For tabular data, SMOTE generates synthetic minority-class rows by interpolating between real ones. Augment the training set only. Augmented validation or test data inflates evaluation scores.

The last step depends on what the data is for.

For RAG, chunking strategy matters most. A common starting point is 256 to 512 token segments with 50 to 100 tokens of overlap, but treat that as a default to test rather than a rule. A multi-dataset analysis of fixed-size chunking found 64 to 128 token chunks best for short factual answers and 512 to 1,024 tokens best when answers need broader context. A 2026 benchmark of 36 segmentation methods across six domains found content-aware chunking (grouping by paragraph, splitting at structure) well ahead of fixed-length character splitting on retrieval accuracy. And a systematic study on Natural Questions measured no benefit from overlap at all while it raised indexing cost. Attach metadata (source URL, title, section heading) for citations and filtering, then store embeddings in a vector database such as Pinecone, Chroma or Weaviate. Test chunk sizes against real queries, because retrieval quality tracks chunk quality. The chunking strategies for RAG and vector database guides compare the options.

For fine-tuning, write JSONL with prompt and completion fields, or a messages array for chat models, and keep the 80/10/10 split from step 4.

For inference pipelines, cache cleaned versions of frequently used data, run PII detection and redaction on user inputs in real time, and design every transform around the latency budget of the application.

The practices that keep a pipeline reliable are the same ones that make it debuggable: start from the model's job, fit every transform on training data only, automate the repeatable steps, record lineage, enforce privacy rules in code, and watch for drift once the model is live.

• Start from the task. The business question decides which fields matter, which records are in scope and which labels you need.
• Fit on training data only. Imputation, scaling and encoding are fitted on the training split, then applied unchanged to the validation and test splits so every set goes through the same transform. Augmentation is applied to the training split alone.
• Automate collection and conversion. Manual exports go stale. Scheduled scrapes, API pulls and parsing jobs keep the dataset current and reproducible.
• Record lineage. Store source URL, ingestion timestamp, processing version and schema version with every record. Without it, tracing a bad prediction to its input is guesswork, and proving to a regulator what trained the model is impossible. Datasheets for Datasets is the standard template for documenting a dataset's motivation, composition, collection process and recommended uses.
• Build in governance. Automated PII detection and retention schedules are much cheaper to add at the start than to retrofit. GDPR and CCPA limit what you can keep and for how long.
• Monitor drift. Track mean, variance and feature distributions of incoming data against the training baseline, and set thresholds that trigger an alert or a retrain. A model trained on last year's behaviour degrades quietly. Meta's engineers describe the production version of this in Moving Fast With Broken Data, where every new data partition is summarized with quality metrics, compared against history, and blocked from retraining when it looks corrupted.

No single tool covers the pipeline. Teams combine an ingestion tool for internal systems, a web and document collection API, a validation library, a labeling tool and a vector database.

Pick the smallest set that covers your sources. Most of the time saved in preparation comes from the collection and conversion steps, because that is where the custom code accumulates.

Firecrawl handles the collection and transformation steps for web pages and documents. Search finds the sources, Scrape turns any URL into Markdown or JSON, Parse does the same for uploaded documents, and Crawl collects a site's pages within the page limit, depth and path rules you set. That removes the HTML cleanup step; chunking the returned Markdown for RAG is still your pipeline's job.

That replaces custom scrapers per site, headless browsers for JavaScript rendering, parsing logic for stripping HTML artifacts, and a separate PDF pipeline. Sites built with React, Vue or other JavaScript frameworks render the same as static pages.

Two of the slowest jobs in Step 1 get shorter. The first is pulling a bulk dataset: Crawl follows a site's links up to the page limit and depth you set and returns each page as Markdown, batch scrape takes a list of URLs and processes them in parallel, and Search with scraping enabled returns the full content of each result rather than a link list. A documentation site, a product catalog or a few thousand article URLs becomes a folder of Markdown files in one job instead of a scraper you maintain. The second is building a curated dataset: the json format applies the same schema to every page, so records from different sites land with the same fields, and Firecrawl Alexandria adds indexed and licensed sources that are already structured and carry their provenance. The curation work left over is choosing what to include, not reshaping what came back.

The formats map to the preparation steps above. Markdown keeps headings, lists and paragraphs for chunking and embedding. The json format returns the fields you specify for structured datasets. The summary and highlights formats return a condensed version or the passages that answer a query, which cuts tokens further when you only need part of a page. Scrape detects PDFs and DOCX files at a URL and parses them the same way, and Parse accepts files from disk in twenty formats, including spreadsheets and presentations.

from firecrawl import Firecrawl
 
firecrawl = Firecrawl()  # no API key needed to get started
 
doc = firecrawl.scrape("https://example.com/report.pdf", formats=["markdown"])
print(doc.markdown)

The same works from the command line:

npx firecrawl-cli@latest scrape https://example.com/docs/getting-started

For data that is already structured, Firecrawl Alexandria runs on the same API key once it is enabled for your team. An agent can search the Research Index, the Developer Index or a third-party provider and get typed records back, with discovery free and each capability priced in credits up front. Government & Legal Index searches cost nothing and work without a key.

A common pattern is ingesting a documentation site for RAG, which means crawling the docs paths you want, chunking the Markdown and loading it into a vector database. For a worked example with an agent on top, see how to turn a documentation site into an AI agent with LangGraph.

Firecrawl's keyless free tier needs no API key and is capped per IP address per day; a free account adds an API key with 1,000 credits a month. Get started at docs.firecrawl.dev.

AI projects fail on data more often than on models. The fix is procedural. Collect from the right sources, explore and clean, label, split and validate, transform and augment, then optimize for RAG, fine-tuning or inference.

Each step depends on the one before it, and the steps skipped early come back as bad predictions and bad retrievals later. The most time goes into collection and conversion, so that is where automation pays off first.