The backend side of this project is comprised of two components (for now):

1. PDF Ingestion
2. AI Interactions

This issue is meant to capture work and discussion related to the backend side of PDF Ingestion.

Initial Exploration of Python PDF parsing libraries

Candidates:

• PyPDF - a pure Python PDF parser (which I think used to be called PyPDF2)
• pdfplumber - which builds on pdfminer.six. This library is used in AI2's mmda and vila.
• PyMuPDF, which is what Haystack's PDF ingestion builds on
• mmda - a work-in-progress from AI2, and uses pdfplumber and pdf2image under the hood
• grobid - used by s2orc-doc2json. Written in Java, which would mean running another tauri-sidecar.
• s2orc-doc2json - Python tool that builds on grobid and is used by AI2's Semantic Scholar. Notably, this would require also having a grobid process running on a refstudio user's machine (and making sure Java is installed).

pypdf

I used this for prototyping the pdf chatbot in scratch-pdf-bot. It did a fine job at extracting text, but I didn't delve too deep beyond that. Seems fairly similar to pdfplumber, with the one notable difference that pdfplumber has additional functionality for table extraction.

mmda

As noted in its readme, this library seems still in its early development. Installing from pypi via poetry add, I ran into ModuleNotFoundError exceptions trying to run the code snippet provided in the readme (missing dependencies for tokenizers, layoutparser, vila) and also noted an empty setup.py.

I think this project is worth continuing to watch, but don't think it's a good fit for refstudio right now.

Given its functionality and adoption in other projects, I'm going to explore pdfplumber further.