To recap, when a Tesseract PDF (3.0x or 4.x) is run through Ghostscript the OCR layer will be mangled. Ghostscript's pdfwrite (gs -sDEVICE=pdfwrite -o out.pdf in.pdf) will display spaces between every character and get confused about word boundaries. Other PDF viewers tend to work but usually have problems with searching for text, because they read as the text as having spaces in between.
Before (pdftext)
Portez ce vieux whisky au juge
blond qui fume sur son île
After
P o r t e z c e v i e u x w h i s k y a u j u g e
o n d q u i f u m e s u r s o n île
(A related issue I reported was fixed in Ghostscript 9.20, but unfortunately that is not complete solution. Ghostscript <9.20 also corrupts any characters above U+00FF that happen to be present.)
There are lots of reasons someone might run a Tesseract PDF through Ghostscript pdfwrite: producing lower DPI renderings, PDF/A conversion, merging PDFs, changing paper sizes, sanitizing potential security holes like Javascript. There are also a lot of programs and services that use Ghostscript internally, sometimes without the user being aware of this. It's unfortunate that Tesseract PDFs don't play nicely with Ghostscript.
Ken Sharp (Ghostscript PDF dev) swears up and down that he can't do anything about it, essentially because Ghostscript interprets the input PDF into a page description language that is then rendered using pdfwrite. The output is visually identical, but otherwise the file is rewritten. Artifex also views preserving OCR text or other metadata as a bonus; if pdfwrite produces visually identical output they are satisfied.
See this comment from 2015:
https://bugs.ghostscript.com/show_bug.cgi?id=696116
Ken Sharp explains the essential difference is that the /DW (default glyph width) parameter on the GlyphLessFont is not understood by GhostPDL so it sets /DW 0 and manually positions each glyph (the -500).
[(T)-500(h)-500(e)-500]TJ
In English, Tesseract renders OCR with a font whose glyphs are 500 arbitrary units wide. Ghostscript reinterprets this as glyphs that are 0 units wide and moves the cursor 500 units between characters, and insists that it's the same thing.
I tried surgery on a pdfwrite-mangled file. I removed all of the -500 offsets, set to /DW 500 on the main font object, and removed the individual glyph width array /W [...] from the same. That works. (pdfwrite makes other minor changes to the PDF output too, but these don't matter as far as I know.) Writing a little script to fix mangled PDFs is possible, but it would be better to find a workaround.
So, is there any possibility of adjusting the glyphless font to work more like what Ghostscript expects so it survives the trip... without losing all of the other considerable and much appreciated effort that has gone into making glyphless work great with most other interpreters?
What are the commercial OCR tools doing to avoid similar issues?
To recap, when a Tesseract PDF (3.0x or 4.x) is run through Ghostscript the OCR layer will be mangled. Ghostscript's pdfwrite (
gs -sDEVICE=pdfwrite -o out.pdf in.pdf) will display spaces between every character and get confused about word boundaries. Other PDF viewers tend to work but usually have problems with searching for text, because they read as the text as having spaces in between.Before (
pdftext)After
(A related issue I reported was fixed in Ghostscript 9.20, but unfortunately that is not complete solution. Ghostscript <9.20 also corrupts any characters above U+00FF that happen to be present.)
There are lots of reasons someone might run a Tesseract PDF through Ghostscript pdfwrite: producing lower DPI renderings, PDF/A conversion, merging PDFs, changing paper sizes, sanitizing potential security holes like Javascript. There are also a lot of programs and services that use Ghostscript internally, sometimes without the user being aware of this. It's unfortunate that Tesseract PDFs don't play nicely with Ghostscript.
Ken Sharp (Ghostscript PDF dev) swears up and down that he can't do anything about it, essentially because Ghostscript interprets the input PDF into a page description language that is then rendered using pdfwrite. The output is visually identical, but otherwise the file is rewritten. Artifex also views preserving OCR text or other metadata as a bonus; if pdfwrite produces visually identical output they are satisfied.
See this comment from 2015:
https://bugs.ghostscript.com/show_bug.cgi?id=696116
Ken Sharp explains the essential difference is that the
/DW(default glyph width) parameter on the GlyphLessFont is not understood by GhostPDL so it sets/DW 0and manually positions each glyph (the-500).In English, Tesseract renders OCR with a font whose glyphs are 500 arbitrary units wide. Ghostscript reinterprets this as glyphs that are 0 units wide and moves the cursor 500 units between characters, and insists that it's the same thing.
I tried surgery on a pdfwrite-mangled file. I removed all of the
-500offsets, set to/DW 500on the main font object, and removed the individual glyph width array/W [...]from the same. That works. (pdfwrite makes other minor changes to the PDF output too, but these don't matter as far as I know.) Writing a little script to fix mangled PDFs is possible, but it would be better to find a workaround.So, is there any possibility of adjusting the glyphless font to work more like what Ghostscript expects so it survives the trip... without losing all of the other considerable and much appreciated effort that has gone into making glyphless work great with most other interpreters?
What are the commercial OCR tools doing to avoid similar issues?