Can Smoldocling make documents analysis more efficient?

This clear, structured format minimizes ambiguity, a common issue with direct conversion methods to formats like HTML or Markdown.

Deep Dive: Dataset Training and Model Architecture

Dataset Training

A key pillar of SmolDocling’s success is its rich, diverse training data:

• Pre-training Data:
  
  • DocLayNet-PT: A 1.4M page dataset extracted from unique PDF documents sourced from CommonCrawl, Wikipedia, and business documents. This dataset is enriched with weak annotations covering layout elements, table structures, language, topics, and figure classifications.
  • DocMatix: Adapted using a similar weak annotation strategy as DocLayNet-PT, this dataset includes multi-task document conversion tasks.
• Task-Specific Data:
  
  • Layout & Structure: High-quality annotated pages from DocLayNet v2, WordScape, and synthetically generated pages from SynthDocNet ensure robust layout and table structure learning.
  • Charts, Code, and Equations: Custom-generated datasets provide extensive visual diversity. For instance, over 2.5 million charts are generated using three different visualization libraries, while 9.3M rendered code snippets and 5.5M formulas provide detailed coverage of technical document elements.
• Instruction Tuning: To reinforce the recognition of different page elements and introduce document-related features and no-code pipelines, rule-based techniques and the Granite-3.1-2b-instruct LLM were leveraged. Using samples from DocLayNet-PT pages, one instruction was generated by randomly sampling layout elements from a page. These instructions included tasks such as:
  • “Perform OCR at bbox”
  • “Identify page element type at bbox”
  • “Extract all section headers from the page”
• Additionally, training with the Cauldron dataset helps avoid catastrophic forgetting due to the introduction of numerous conversation datasets.

Model Architecture of SmolDocling

SmolDocling builds upon the SmolVLM framework and incorporates several innovative techniques to ensure efficiency and effectiveness:

• Vision Encoder with SigLIP Backbone: The model uses a SigLIP base 16/512 encoder (93M parameters) which applies an aggressive pixel shuffle strategy. This compresses each 512×512 image patch into 64 visual tokens, significantly reducing the number of image hidden states.
• Enhanced Tokenization: By increasing the pixel-to-token ratio (up to 4096 pixels per token) and introducing special tokens for sub-image separation, tokenization efficiency is markedly improved. This design ensures that both full-page documents and cropped elements are processed uniformly.
• Curriculum Learning Approach: Training begins with freezing the vision encoder, focusing on aligning the language model with the new DocTags format. Once the model is familiar with the output structure, the vision encoder is unfrozen and fine-tuned along with task-specific datasets, ensuring comprehensive learning.
• Efficient Inference: With a maximum sequence length of 8,192 tokens and the ability to process up to three pages at a time, SmolDocling achieves page conversion times of just 0.35 seconds using VLLM on an A100 GPU, while occupying only 0.489 GB of VRAM.

Comparative Analysis: SmolDocling Versus Other Models

A thorough evaluation of SmolDocling against leading vision-language models highlights its competitive edge:

Text Recognition (OCR) and Document Formatting

Insights: SmolDocling outperforms larger models across all key metrics in full-page transcription. The significant improvements in F1-score, precision, and recall reflect its superior capability in accurately reproducing textual elements and preserving reading order.

Specialized Tasks: Code Listings and Equations

• Code Listings: For tasks like code listing transcription, SmolDocling exhibits an impressive F1-score of 0.92 and precision of 0.94, highlighting its expertise at handling indentation and syntax that carry semantic significance.
• Equations: In the domain of equation recognition, SmolDocling closely matches or exceeds the performance of models like Qwen2.5 VL and GOT, achieving an F1-score of 0.95 and precision of 0.96.

These results underscore SmolDocling’s ability to not only match but often surpass the performance of models that are significantly larger in size, affirming that a compact model can be both efficient and effective when built with a focused architecture and optimized training strategies.

Code Demonstration and Output Visualization

To provide a practical glimpse into how SmolDocling operates, the following section includes a sample code snippet along with an illustration of the expected output. This example demonstrates how to convert a document image into the DocTags markup format.

Example 1: Sample Code Snippet

!pip install docling_core

!pip install flash-attn

import torch

from docling_core.types.doc import DoclingDocument

from docling_core.types.doc.document import DocTagsDocument

from transformers import AutoProcessor, AutoModelForVision2Seq

from transformers.image_utils import load_image

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# Load images

# Initialize processor and model

processor = AutoProcessor.from_pretrained("ds4sd/SmolDocling-256M-preview")

model = AutoModelForVision2Seq.from_pretrained(

    "ds4sd/SmolDocling-256M-preview",

    torch_dtype=torch.bfloat16,

    _attn_implementation="flash_attention_2"# if DEVICE == "cuda" else "eager",

).to(DEVICE)

model.device

# Load images

image = load_image("https://user-images.githubusercontent.com/12294956/47312583-697cfe00-d65a-11e8-930a-e15fd67a5bb1.png")

# Create input messages

messages = (

    {

        "role": "user",

        "content": (

            {"type": "image"},

            {"type": "text", "text": "Convert this page to docling."}

        )

    },

)

# Prepare inputs

prompt = processor.apply_chat_template(messages, add_generation_prompt=True)

inputs = processor(text=prompt, images=(image), return_tensors="pt")

inputs = inputs.to(DEVICE)

# Generate outputs

generated_ids = model.generate(**inputs, max_new_tokens=8192)

prompt_length = inputs.input_ids.shape(1)

trimmed_generated_ids = generated_ids(:, prompt_length:)

doctags = processor.batch_decode(

    trimmed_generated_ids,

    skip_special_tokens=False,

)(0).lstrip()

# Populate document

doctags_doc = DocTagsDocument.from_doctags_and_image_pairs((doctags), (image))

print(doctags)

# create a docling document

doc = DoclingDocument(name="Document")

doc.load_from_doctags(doctags_doc)

from IPython.display import display, Markdown

display(Markdown(doc.export_to_markdown()))

Input image

Production

This output illustrates how several documents (text blocks, tables and code listings are marked precisely with their content and spatial information, which prepares them for additional processing or analysis. But the model cannot convert the entire marking format of Text Doctags. As you can see, the model did not read the human written text.

Example 2: Sample code fragment

!curl -L -o image2.png https://i.imgur.com/BFN038S.png

The entrance image is the receipt and now we are extracting the text.

image = load_image("./image2.png")

# Create input messages
messages = (
    {
        "role": "user",
        "content": (
            {"type": "image"},
            {"type": "text", "text": "Convert this page to docling."}
        )
    },
)

# Prepare inputs
prompt1 = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs1 = processor(text=prompt1, images=(image), return_tensors="pt")
inputs1 = inputs1.to(DEVICE)

# Generate outputs
generated_ids = model.generate(**inputs1, max_new_tokens=8192)
prompt_length = inputs1.input_ids.shape(1)
trimmed_generated_ids = generated_ids(:, prompt_length:)
doctags = processor.batch_decode(
    trimmed_generated_ids,
    skip_special_tokens=False,
)(0).lstrip()

# Populate document
doctags_doc = DocTagsDocument.from_doctags_and_image_pairs((doctags), (image))
print(doctags)
# create a docling document
doc = DoclingDocument(name="Document")
doc.load_from_doctags(doctags_doc)

# export as any format
# HTML
# doc.save_as_html(output_file)
# MD
print(doc.export_to_markdown())

from IPython.display import display, Markdown

display(Markdown(doc.export_to_markdown()))

Production

It is quite impressive since the model extracted the entire content of the receipt and it is better than the example of Obove given.

Notebook with complete code: Click here

Conclusion and future addresses

Smoldocling establishes a new reference point in the conversion of documents by demonstrating that the smallest and most efficient models can rival and even overcome the capabilities of their larger counterparts. Its innovative use of doctags and an end -to -end conversion strategy provide a convincing plan for the next generation of vision language models. It works well with receipts in general and works acceptably with other documents, although it is not always perfectly that this serves as a result of the design of its memory savings model.

Key control

• Efficiency: With a 256 m compact parameter architecture, Smoldocling achieves a quick page conversion with a minimal computational overload.
• Sturdiness: Extensive data sets prior to training and specific tasks, together with a curricular learning approach, ensure that the model is generally well in various types of documents.
• Comparative superiority: Through rigorous evaluations, Smoldocling has demonstrated higher performance in OCR, code transcription and recognition of equations compared to larger models.

As the research community continues to refine the techniques for the location of elements and multimodal understanding, Smoldocling provides a clear route towards more efficient and versatile document processing solutions of resources. With plans to launch the data sets that accompany it publicly, this work paves the way for new advances and collaborations in the field.