pdftxtract
Ingesting pdfs into an LLM or grepping their contents is difficult. Tools like pandoc often do not preserve the formatting of tables well, or the relative positions of elements on the page. When trying to read a pdf from the terminal, or ingest a pdf into an LLM, this broken formatting is a cause for confusion.
One of my perennial inspirations is the browsh project, which effects fully-functional interactive web browsing via a headless Firefox instance with custom CSS to render text as a monospaced grid. I have always found this an elegant solution to address the problem of viewing variable-text media in a monospaced grid environment like the terminal, so I applied this concept to PDF rendering.
Features
- Grid size is autodetected from the page by default, but manually overridable. Characters are snapped to their position in the grid.
- Horizontal and vertical lines are specially handled so table structures are preserved.
- Collision handling is configurable to account for two characters which are almost overlapping and would snap to the same grid cell.
Examples
This tool was inspired by page 39 of the Allwinner A133 User Manual, with a complex nested table structure that all the existing tools I tried failed to convert in a way that was correct and readable by a human or LLM.
Here is how the table looks before
And here is how it was converted by popular tools
pdftotext (poppler)
System
AXI2MBUS Logic Circuit Reset
0: assert
1: de-assert
27:26
25
24
23:20
pdftotext -layout (poppler) — poppler in layout-preserving mode
AXI2MBUS Logic Circuit Reset
0: assert
1: de-assert
27:26 / / /
MBIST_RST
CPUBIST Reset
25 R/W 0x1 The reset signal is for test
0: assert
1: de-assert
SOC_DBG_RST
Cluster SOC Debug Reset
24 R/W 0x1
This is closest to what I would consider acceptable, but still vague and undelimited by table rows/columns, so it's not clear where the rows start and stop in that fourth column.
mutool (MuPDF):
System
A133 User Manual(Revision 1.1)
Copyright©2020 Allwinner Technology Co.,Ltd. All Rights Reserved.
Page 39
AXI2MBUS Logic Circuit Reset
0: assert
1: de-assert
27:26
/
/
/
pdfminer.six:
AXI2MBUS Logic Circuit Reset
0: assert
1: de-assert
27:26
/
/
/
System
25
pdftxtract:
System
+-----------------------------------------------------------------------------------------------------------+
+--------+--------------+--------------+-----------------------------------------------------------------------+
| | | |AXI2M BU SLogicCircuitReset |
| | | | |
| | | |0:assert |
| | | |1:de-assert |
+--------+--------------+--------------+-----------------------------------------------------------------------+
|27:26 |/ |/ |/ |
+--------+--------------+--------------+-----------------------------------------------------------------------+
| | | |M BIST_RST |
| | | |CPU BISTReset |
| | | | |
|25 |R/W |0x1 |The resetsignalisfortest |
| | | | |
| | | |0:assert |
| | | |1:de-assert |
+--------+--------------+--------------+-----------------------------------------------------------------------+
The only drawback here is that we display each character's position on the page faithfully, and since text is often laid out one character at a time, this sometimes results in space between characters (MBIST_RST vs M BIST_RST). A small price to pay.