Text extraction

A PDF page stores positioned glyphs, not semantic lines or paragraphs, so both plain-text surfaces have structure to infer. PdfElixide.Document.text/2 preserves the visual rows it infers; PdfElixide.Document.to_plain_text/2 infers prose blocks and reading order. Their bang variants return a string directly; the non-bang variants return {:ok, string} or {:error, %PdfElixide.Error{}}. On an untagged prose page the two strings often differ; on a simple or trustworthy-tagged page they may not.

alias PdfElixide.Document

doc = Document.open!("path/to/file.pdf")

Document.text!(doc, 0)
#=> "Automating the construction of index pairs,\nessential ingredients of the\ntheory."

Document.to_plain_text!(doc, 0)
#=> "Automating the construction of in- dex pairs, essential ingredients of the theory."

The examples below reuse this doc; close it when you are done.

This guide focuses on those two functions. The geometric extractors — PdfElixide.Document.chars/2, PdfElixide.Document.words/2, PdfElixide.Document.text_lines/2, PdfElixide.Document.spans/2 — return glyphs and geometry, PdfElixide.Document.structured/2 returns typed regions, and PdfElixide.Document.to_markdown/2 and PdfElixide.Document.to_html/2 return markup. The last section summarizes them.

text/2 — the page as laid out

PdfElixide.Document.text/2 walks the page's glyph runs and assembles them row by row. It keeps its inferred visual line breaks, can rejoin lowercase words split across lines by hyphenation, and can restrict what it reads. The rejoining is conservative, not a general-purpose dehyphenator.

Reach for it when:

• the line structure carries meaning — an address block, a code listing, a form, a table of contents;
• you want to extract part of a page, or drop a watermark layer or a spot ink (only this surface takes :region, :exclude_regions, :exclude_layers and :exclude_inks);
• you are extracting a whole document and want a page that fails to be skipped rather than to fail the call (:on_page_error);
• you want ligatures expanded (:expand_ligatures).

Text lying entirely outside the page's crop box is not returned — see What the crop box hides from extraction, which also names the one extractor that reports it anyway.

to_plain_text/2 — the page as prose

PdfElixide.Document.to_plain_text/2 groups spans into blocks, orders the blocks — detecting columns, not just sorting by height — and generally reflows prose paragraphs onto one line, separating them with a blank line. Tables and some columnar layouts retain internal line breaks.

Reach for it when you want paragraph text rather than page text: indexing, chunking for retrieval, feeding a model, or diffing two versions of a document whose line wrapping has changed.

On typeset or OCR-backed scanned prose, a hyphen at a line break stays where it fell. Where PdfElixide.Document.text/2 rejoins in- and dex into index, this returns in- dex. Prefer text/2 when rejoining such words matters more than paragraph shape.

Neither function performs OCR. An image-only scanned page normally returns an empty string; supply an OCR text layer before using either extractor.

A tagged PDF collapses the difference

When a document carries a readable structure tree and its producer has not marked the tags suspect, PdfElixide.Document.to_plain_text/2 reads the page in the order those tags declare and returns exactly what PdfElixide.Document.text/2 returns for it.

tagged = Document.open!("path/to/tagged.pdf")

Document.to_plain_text!(tagged, 0) == Document.text!(tagged, 0)
#=> true

:ok = Document.close(tagged)

On such a document, :reading_order and :include_form_fields have no effect: the tags determine the order and field values are always inlined.

PdfElixide.Document.has_structure_tree?/1 is a coarse signal here: it reports a readable tree even when the producer marked its tags suspect, while extraction falls back to geometric order in that case. There is no public predicate that distinguishes the two. If the distinction matters for a corpus, compare the two outputs on representative pages rather than deciding from that predicate alone.

Which options each takes

The two surfaces overlap on :extract_tables and :table_detection and nowhere else. Passing a key the other one owns raises ArgumentError naming it, rather than being accepted and ignored.

:exclude_layers and :exclude_inks combine only with :region and :region_mode; any other key given a non-default value alongside them raises ArgumentError. See PdfElixide.Document.text_opts/0.

On an untagged document, two of :reading_order's values behave alike: :structure_tree and :column_aware both run the column-detecting pass, and :top_to_bottom sorts blocks by vertical position instead. On a single-column page they all agree.

PdfElixide.Document.text_opts/0 and PdfElixide.Document.plain_text_opts/0 document each key in full.

Whole documents, and getting back to pages

Both have a whole-document arity, and they join pages differently:

Document.text!(doc)            # pages separated by a form feed, "\f"
Document.to_plain_text!(doc)   # pages separated by "\n\n---\n\n"

For a document with at least one page, a successful PdfElixide.Document.text/1 splits on the form feed into exactly PdfElixide.Document.page_count/1 parts; under on_page_error: :skip a page that failed to extract stays as an empty part. A zero-page document returns "", which String.split/2 represents as one empty part rather than none. The --- boundary is not as safe: PdfElixide.Document.to_plain_text/2 separates paragraphs with a blank line, so a page whose own text has a --- line between two paragraphs emits the same bytes and splits into an extra part, shifting every page after it. Enumerate pages when page boundaries must be unambiguous.

Neither bounds memory: each builds the whole result before returning. A document is enumerable over its pages, so process each page's result without accumulating it to bound memory used by results:

doc
|> Stream.map(&PdfElixide.Document.Page.to_plain_text!/1)
|> Stream.each(&IO.puts/1)
|> Stream.run()

Collecting this stream into a list retains every page's text. See "Whole-document extraction and memory" in PdfElixide.Document for the memory and handle-release guidance shared by the extractors.

Use the same page-wise shape for concurrent extraction; see the Concurrency guide.

When a page fails

The "When a page cannot be read" section of PdfElixide.Document lists which whole-document calls fail or skip. To tolerate a failure, work a page at a time and decide per page. This keeps which page failed and why, so a handle failure such as :closed or :panic is not mistaken for an empty page:

alias PdfElixide.Document.Page

results = Enum.map(doc, &{&1.index, Page.chars(&1)})

chars =
  Enum.flat_map(results, fn
    {_index, {:ok, chars}} -> chars
    {_index, {:error, _}} -> []
  end)

failed =
  for {index, {:error, %PdfElixide.Error{} = error}} <- results,
      do: {index, error}

This handles only returned errors. See the ":on_page_error and partly extractable documents" section of PdfElixide.Document.text_opts/0 for cases that produce empty text instead.

Tables inside a text result

With :extract_tables on — the default for both — a recognized table is rendered from its detected cells. Both assemblers try to suppress flowing spans that those cells already represent, so ordinary cell text appears once.

Some PDFs can still repeat cell text when their page text cannot be associated reliably with the detected cells. Trustworthy tagged extraction avoids adding a second table rendering. For matching rather than reading, prefer PdfElixide.Document.words/2, or PdfElixide.Document.search/2 when the query is known in advance. The "Choosing an extractor for search and matching" section of PdfElixide.Document compares these surfaces.

When the separate cell rendering is emitted, PdfElixide.Document.to_plain_text/2 keeps its column padding while PdfElixide.Document.text/2 collapses the padding to single spaces.

Running headers and footers

PdfElixide.Document.running_regions/2 finds the spans of each page's running header and footer, keyed by page index. It changes nothing on the document, so later extraction still includes the text.

regions = Document.running_regions!(doc)
#=> %{0 => [%PdfElixide.Document.Span{text: "Quarterly Report"}, ...], 1 => [...]}

A document that tags its headers and footers as artifacts is answered from those tags. Otherwise a line counts when the same text sits at about the same place in the top or bottom margin on at least :threshold of the pages, 80% by default. Lower it for a document whose running header skips some pages, such as chapter openings:

Document.running_regions!(doc, area: :header, threshold: 0.5)

Bare page numbers are excluded. A margin line whose number changes is usually missed on the first page; see PdfElixide.Document.running_regions/2.

To read text without them, pass each page's boxes as :exclude_regions. The spans are in the frame that option filters in, rotated pages included:

text =
  Enum.map_join(doc, "\f", fn page ->
    boxes = regions |> Map.get(page.index, []) |> Enum.map(& &1.bbox)
    Page.text!(page, exclude_regions: boxes)
  end)

The whole-document text/2 applies one :exclude_regions list to every page, so it cannot take a per-page map. PdfElixide.Document.words/2 and PdfElixide.Document.text_lines/2 take no exclusion list; passing them include_artifacts: false drops text the document tags as artifacts instead.

To hide them in a saved copy, paint over the same boxes with PdfElixide.Editor.erase_regions/3, which takes raw, unrotated user space. Spans on a rotated page may be in the displayed frame; map those back with PdfElixide.Geometry.Rect.to_user_space/3 as Bringing boxes into one frame describes:

alias PdfElixide.Geometry.Rect

editor = PdfElixide.Editor.open!("path/to/file.pdf")

editor =
  Enum.reduce(regions, editor, fn {index, spans}, editor ->
    page = Document.page!(doc, index)
    rotation = Page.rotation!(page)
    media_box = Page.media_box!(page)

    boxes =
      for span <- spans do
        if rotation == 180 or span.rotation != 0.0,
          do: Rect.to_user_space(span.bbox, rotation, media_box),
          else: span.bbox
      end

    PdfElixide.Editor.erase_regions!(editor, index, boxes)
  end)

editor
|> PdfElixide.Editor.save!("path/to/without-running-text.pdf")
|> PdfElixide.Editor.close()

Do this before moving or deleting pages, while the editor's page indices still match the document the spans came from. The result is a white rectangle over text that is still there; see the "Erasing regions" section of the Editing guide. Text that comes from an annotation, such as a comment or stamp repeated in the margin, is drawn above the page content, so the rectangle does not cover it.

Beyond plain text

• Glyphs and geometry. PdfElixide.Document.chars/2, PdfElixide.Document.spans/2, PdfElixide.Document.words/2 and PdfElixide.Document.text_lines/2 return structs with bounding boxes, fonts and colors. Use them to locate text on the page rather than to read it.
• Typed regions. PdfElixide.Document.structured/2 groups a page's spans into PdfElixide.Document.StructuredPage.Regions — body text with one region per column, running header, footer and page number where the PDF marks them as artifacts, marginal numerals — each carrying its spans and their union box. The reading to reach for when the role of a block matters more than its glyphs.
• Markup. PdfElixide.Document.to_markdown/2 represents detected headings, lists and tables; PdfElixide.Document.to_html/2 returns an escaped fragment, optionally absolutely positioned. Both take markup options the plain-text surfaces have no use for, and to_markdown/2 takes the largest set.
• Finding a known string. PdfElixide.Document.search/2 indexes each page once and returns match geometry — see the Text search guide.

:ok = Document.close(doc)