Structured vs Semi-Structured Documents: Why Your Labeling Strategy Needs to Treat Them Differently

Most ML teams building document AI hit the same wall at some point. They spend weeks labeling a clean batch of invoices, train a model that performs brilliantly in testing, then watch it fall apart the moment a vendor sends an invoice in a slightly different layout. The annotation strategy that worked perfectly for one document type produces a model that cannot generalize to another.

The root cause is almost always the same: the team treated structured and semi-structured documents as the same problem. They are not.

Getting this distinction right before you build your labeling pipeline is not a minor optimization. It determines which tools you use, how you design your taxonomy, which model architecture you train, and how much data you actually need. This article breaks down the real differences and gives you a concrete strategy for each

What Structured Documents Actually Are

Structured documents follow a rigid, predefined schema where every piece of data lives in a fixed location. Think of a SQL database table exported to a spreadsheet. Column A is always "Customer ID," column B is always "Date," column C is always "Amount." The position of every field is guaranteed.

In practice, structured documents look like:

• CSV and TSV exports from ERP systems

• Standardized government forms where every field has a fixed position (IRS Form 1040, for example)

• Database records and data warehouse exports

• Fixed-format EDI files used in supply chain

The defining characteristic is not the file format but the schema contract. Every instance of the document has the same fields in the same positions. If you train a model on 100 of them, it has seen every meaningful layout variation that will ever exist.

What Semi-Structured Documents Actually Are

Semi-structured documents contain fields and values, like structured documents, but they have no guaranteed layout. The data is there, the labels are there, but where they appear on the page varies from document to document.

An invoice from a law firm looks nothing like an invoice from a logistics company. Both have a "Total Due" field. Both have a "Bill To" section. Both have line items. But the position of each element, the label text used to identify it, the formatting, the font size, the column structure, and the number of pages are all different.

Common semi-structured document types include:

• Invoices and purchase orders

• Bank statements and credit card statements

• Contracts, NDAs, and legal agreements

• Medical records, discharge summaries, and clinical notes

• Lease agreements and real estate documents

• Insurance policies and claims forms

• Annual reports and SEC filings

According to research cited in a USPTO patent on document extraction systems, up to 80% of an organization's data exists in unstructured or semi-structured formats. McKinsey estimates that 60% of enterprise documents remain unstructured, slowing down automation efforts. The documents piling up in your organization's inboxes and shared drives are almost certainly semi-structured, not structured.

Why the Difference Is Not Always Obvious

The confusion arises because semi-structured documents look organized. An invoice has a clear grid. A contract has numbered sections. A bank statement has tidy rows. They look structured to the human eye because humans are good at reading layout and inferring meaning from context.

A machine learning model does not have that luxury by default. Without the right labeling approach, the model sees pixel patterns or token sequences without a reliable map of what means what. The apparent visual organization of a semi-structured document is exactly what makes it deceptive as a training data problem.

The three-category model that helps most teams is:

• Fully structured: Fixed schema, guaranteed positions. Examples: spreadsheet exports, database dumps, standardized government forms.

• Semi-structured: Variable layout, consistent field types, no guaranteed positions. Examples: invoices, contracts, statements, medical records.

• Fully unstructured: No schema, no fields, just free-form content. Examples: news articles, emails, social media posts, free-text notes.

Most document AI projects live in the semi-structured middle category. The labeling strategies that work for the extremes both fail here.

How Labeling Works for Structured Documents

Structured document labeling is fundamentally a schema mapping problem. You are telling the model which field in the schema corresponds to which column, row, or cell in the document.

Because the layout is fixed, you can often label a small number of documents and achieve high generalization. The annotation taxonomy is straightforward: this column is "Date," this column is "Amount," this column is "Description." Once the model learns the mapping from one instance, it applies to all instances.

Key characteristics of a structured labeling strategy:

• Small dataset requirement. Fifty to two hundred labeled examples is often sufficient because there is no layout variation to account for.

• Field-level annotation. You annotate at the level of individual values, not document regions.

• Validation rules work well. You can apply hard rules to check that dates look like dates and amounts look like numbers, reducing label error rates.

• Template-based approaches are viable. Because the layout does not change, a template that defines field positions will work across all instances.

• Auto-label accuracy is very high. On well-structured documents with consistent layouts, AI auto-labeling achieves 95%+ field-level accuracy after minimal calibration.

The tooling for structured annotation is also simpler. You are essentially teaching the model a lookup table. The challenge is not annotation complexity but data quality: ensuring the source data is clean, consistently formatted, and free of schema drift across exports.

How Labeling Works for Semi-Structured Documents

Semi-structured labeling is a fundamentally different problem. You are not mapping fields to positions because the positions change. You are teaching the model to recognize and extract fields based on their semantic context, visual layout, spatial relationships, and surrounding content.

This is where most labeling strategies break down. Teams that import their structured document workflow into a semi-structured project end up with models that memorize layout rather than understanding content, which fail on any document that deviates from the training distribution.

The correct approach requires annotating at the level of document regions with bounding boxes, semantic labels, and contextual relationships. What makes this hard is that the model needs to learn not just "this text says 'Total Due'" but "this is a Total Due field regardless of where on the page it appears, what font it uses, how many decimals the amount has, and whether it appears above or below the line items."

Key characteristics of a semi-structured labeling strategy:

• Larger dataset requirement. You need enough examples to cover layout variation. For a single document class like invoices, this typically means 500 to 2,000+ labeled pages depending on how diverse your vendor base is.

• Bounding box annotation. Each labeled element gets a bounding box on the page, capturing its spatial position along with the text and label classification.

• Hierarchy and relationship labeling. You need to capture not just individual fields but relationships between them. A line item is not just a text string — it has a relationship to a quantity, a unit price, a description, and a total that must all be linked.

• Domain-trained models outperform general ones. A model trained only on financial documents will significantly outperform a general document model on invoice extraction, because the taxonomy of financial document elements is specialized.

• Confidence scoring is essential. Unlike structured documents, semi-structured auto-labels carry genuine uncertainty. Every labeled region needs a confidence score so low-confidence predictions can be flagged for human review before entering your training set.

• Template approaches fail. Because the layout varies, a template that defines field positions will not generalize. You need a model that reasons from context.

The export format matters too. For semi-structured labeling, your output JSON needs to capture bounding box coordinates, the extracted text content, the label classification, the confidence score, and ideally the page hierarchy. This structured output is what makes the labeled data compatible with document understanding models like LayoutLM and feeds cleanly into PyTorch DataLoaders or HuggingFace Transformers.

If you are building or evaluating annotation workflows, it is worth reading through what the Indie Hackers community has documented about 5 most effective ways to label data for ML projects — particularly the discussion of when to use hybrid approaches versus fully automated annotation.

The 5 Labeling Strategy Differences That Actually Matter

1. Taxonomy Design

For structured documents, your taxonomy maps directly to the schema: field names become label classes. The taxonomy is flat and deterministic.

For semi-structured documents, you need a hierarchical taxonomy that captures document sections, element types, and field-level entities. An invoice taxonomy might have: Document Header, Vendor Information, Buyer Information, Line Items (with sub-labels for Description, Quantity, Unit Price, Total), Tax Section, Payment Terms, and Totals. Getting this taxonomy right before you label the first page is critical because changing it mid-project forces relabeling everything already annotated.

2. Model Architecture Choice

Structured documents feed naturally into classical ML models. Gradient boosting methods like LightGBM and XGBoost handle structured inputs well, train quickly, and are highly explainable, which matters in regulated industries.

Semi-structured documents require models that understand both text and visual layout simultaneously. LayoutLM and its successors (LayoutLMv2, LayoutLMv3) are the standard architecture for this because they were specifically designed to process text tokens and their spatial positions together. Feeding a semi-structured document into a purely text-based transformer like BERT will underperform because the model has no awareness of layout.

3. Annotation Tooling Requirements

Structured annotation can be done with simple spreadsheet editors or basic form-fill interfaces. The annotations are values in fields, not visual markups.

Semi-structured annotation requires a tool that renders the actual document page, allows annotators to draw bounding boxes over regions, assign labels to those boxes, and link related boxes to capture relationships. PDF rendering quality matters here — annotation tools that convert PDFs to low-resolution images before display introduce positional drift that degrades the bounding box accuracy of your labels.

4. Quality Control Strategy

For structured documents, quality control is largely automated. You validate that each labeled value matches expected data types, ranges, and formats. Outliers are easy to detect because they violate schema expectations.

For semi-structured documents, quality control requires human review processes. Inter-annotator agreement metrics (measuring whether two independent annotators label the same region the same way) are essential because the correct label for an ambiguous region requires judgment. Target an IAA score above 0.85 before treating your labels as ground truth. Regions where annotators disagree frequently are signals that your taxonomy needs clarification or that the document type has more layout variation than your current strategy accounts for.

5. Generalization Testing

For structured documents, generalization testing is simple: if the model works on a held-out set from the same schema, it will work in production.

For semi-structured documents, your test set must deliberately include layout variations not seen in training. This means actively sourcing documents from different vendors, different software that generates the document type, different geographic regions, and different time periods. A model that scores 95% accuracy on a test set drawn from the same sources as training may drop to 70% on real production documents if layout variation was not captured in the original dataset.

The Common Mistake: Applying Structured Thinking to Semi-Structured Problems

The most expensive mistake in document AI projects is using a template-based or position-based labeling approach on semi-structured documents. It looks like it works during development because your development set probably came from a handful of consistent sources. It breaks in production when the real-world diversity of document layouts surfaces.

Manual invoice processing currently carries error rates of 2-5% in enterprise operations. The reason those errors persist even in organizations that have invested in automation is that most automation tools were built around the structured assumption. They define templates for known vendors, match incoming documents to templates, and fall apart when a new vendor format arrives.

The right approach is to invest upfront in semantic labeling that teaches the model what a "Total Due" field means in context, not where it lives on the page. That is a harder labeling problem, but it produces a model that generalizes to new vendors and document formats without requiring manual template creation for each one.

The ML infrastructure piece on Indie Hackers by Pratyusha Singaraju from Microsoft makes a related point about building feedback loops into ML systems. The same principle applies to document labeling: if your annotation strategy does not capture and learn from real-world variation, the system cannot self-correct as new document types arrive.

Practical Recommendations Before You Start Labeling

Classify your document types first. Before touching an annotation tool, audit the documents in your pipeline and categorize each type as structured, semi-structured, or unstructured. This determines everything else.

Inventory your layout variation. For each semi-structured document type, collect at least 50-100 samples from real sources before designing your taxonomy. Look at how much the layout actually varies. A set of invoices from three large ERP systems will have very different variation than invoices from 200 individual vendors.

Design your taxonomy top-down, not bottom-up. Start with the highest-level document sections, then define the field entities within each section, then the relationship types between fields. Annotators working bottom-up on individual fields tend to miss structural relationships that are critical for model performance.

Use confidence scoring from the start. Even if you are doing manual annotation, build confidence tracking into your process. Flag regions where annotators disagreed or where the document layout was unusual. Those regions are where your model will be weakest, and having that signal explicitly in your training data lets you prioritize review effort on the right samples.

Plan for retraining. Semi-structured document collections evolve. Vendors change their invoice templates. Regulatory changes alter the fields in compliance documents. Build your pipeline with the expectation that you will need to add new labeled examples and retrain regularly, not just once.

Tools That Handle the Distinction Well

Tools built for general image annotation or text annotation typically struggle with semi-structured PDFs because they are not designed around the bounding-box-plus-text-plus-label structure that document AI models need.

The most effective workflows use platforms that were purpose-built for document intelligence. AI Asset Management's document intelligence platform is one example of a system built from the ground up with this distinction in mind, with separate modeling layers for different document structures. Their PDF data labeling platform uses semantic segmentation to handle the layout variability of semi-structured documents, which is the core technical challenge general-purpose annotation tools sidestep. The output JSON it produces — including bounding coordinates, segment text, classifications, and confidence scores — matches exactly what LayoutLM, PyTorch, and HuggingFace expect, which removes a significant data engineering step from the pipeline.

The broader point is that whatever tooling you choose, verify that it was designed for document-level annotation rather than adapted from an image or text annotation background. The difference shows up in how bounding boxes are handled, how multi-page documents are processed, and whether confidence scoring is a native output or an afterthought.

What This Means for Your Model Performance

The payoff for getting labeling strategy right is significant. LLM-assisted annotation workflows that are correctly designed for semi-structured documents show 40-60% reduction in time per labeled page compared to fully manual approaches. But that efficiency gain only materializes if the underlying annotation schema was correctly designed for the document type.

More importantly, models trained on well-designed semi-structured labeling datasets generalize to new document layouts that were not in the training set. That generalization is the actual business value. An invoice processing model that can only handle invoices from your known vendors is not a scalable solution. A model trained on a diverse, well-labeled dataset of semi-structured invoices will handle new vendors without requiring template engineering for each one.

Building AI-ready data infrastructure — as covered in the Building AI-Ready Data Platforms piece on Indie Hackers — requires treating your labeling strategy as a first-class infrastructure decision, not just a preprocessing step. The distinction between structured and semi-structured documents is one of the clearest places where that decision has a direct, measurable impact on what your model can actually do.

Summary

The structured versus semi-structured distinction is not a theoretical concern. It determines your taxonomy design, your tooling choice, your model architecture, your quality control process, your dataset size requirements, and your approach to generalization testing.

Treating semi-structured documents with a structured labeling strategy is the single most common cause of document AI models that work in development and fail in production. The fix is not more data or a better model. It is a labeling strategy built around the actual nature of the documents.

Identify your document types, design a schema-appropriate taxonomy, use tools built for document-level annotation with bounding boxes and confidence scores, test explicitly for layout variation, and plan for continuous retraining as your document collection evolves.

The upfront investment in getting the labeling strategy right is small compared to the cost of rebuilding a pipeline after discovering your model cannot handle real-world document diversity.