Skip to content

Do not use "use built-in mapper WIN32 only"#4310

Closed
cgohlke wants to merge 3 commits intopython-pillow:masterfrom
cgohlke:patch-1
Closed

Do not use "use built-in mapper WIN32 only"#4310
cgohlke wants to merge 3 commits intopython-pillow:masterfrom
cgohlke:patch-1

Conversation

@cgohlke
Copy link
Copy Markdown
Contributor

@cgohlke cgohlke commented Dec 27, 2019 •
edited by hugovk
Loading

Fixes the issue reported at https://stackoverflow.com/questions/59485047/importing-large-multilayer-tiff-as-numpy-array

The Win32 only mapper is 32-bit only and cannot access more than 2 GB. This code is ancient and can probably be replaced by Python's memmap.

import numpy as np
from PIL import Image

# generate test image
fname = r"test.tif"
img = np.random.randint(2**16, size=(257, 2048, 2048), dtype=np.uint16)

# save test image
im_list = [Image.fromarray(img[ii].astype('uint16')) for ii in range(img.shape[0])]
im_list[0].save(fname, save_all=True, append_images=im_list[1:])

# load image
im_loaded = Image.open(fname)

# this succceeds
im_loaded.seek(255)
arr255 = np.asarray(im_loaded, dtype=np.uint16)

# this fails
im_loaded.seek(256)
im_loaded.show()
arr256 = np.asarray(im_loaded, dtype=np.uint16)
File "pil2gbtiff.py", line 21, in <module>
  im_loaded.show()
File "X:\Python36\Lib\site-packages\PIL\Image.py", line 2128, in show
  _show(self, title=title, command=command)
File "X:\Python36\Lib\site-packages\PIL\Image.py", line 3040, in _show
  _showxv(image, **options)
File "X:\Python36\Lib\site-packages\PIL\Image.py", line 3046, in _showxv
  ImageShow.show(image, title, **options)
File "X:\Python36\Lib\site-packages\PIL\ImageShow.py", line 54, in show
  if viewer.show(image, title=title, **options):
File "X:\Python36\Lib\site-packages\PIL\ImageShow.py", line 72, in show
  image = image.convert(base)
File "X:\Python36\Lib\site-packages\PIL\Image.py", line 930, in convert
  self.load()
File "X:\Python36\Lib\site-packages\PIL\TiffImagePlugin.py", line 1098, in load
  return super(TiffImageFile, self).load()
File "X:\Python36\Lib\site-packages\PIL\ImageFile.py", line 186, in load
  self.map.seek(offset)

builtins.OverflowError: Python int too large to convert to C long

Changes proposed in this pull request:

  • use Python's memory mapping instead.

@hugovk hugovk added the Windows label Dec 27, 2019
@homm
Copy link
Copy Markdown
Member

homm commented Dec 27, 2019

  1. Need a test that fails with the previous implementation.
  2. All related code in _imaging.c and map.c could be removed?

homm
homm reviewed Dec 27, 2019
View reviewed changes
This was referenced Jan 24, 2021
Closed
Merged
@radarhere radarhere changed the title Do not use use built-in mapper WIN32 only Do not use "use built-in mapper WIN32 only" Jan 25, 2021
@radarhere
Copy link
Copy Markdown
Member

radarhere commented Jan 25, 2021

I've recreated this as #5224, since this needs to be rebased and the forked repository for this PR has disappeared.

@radarhere radarhere closed this Jan 25, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@homm homm homm left review comments

@radarhere radarhere radarhere left review comments

@hugovk hugovk hugovk approved these changes

Assignees

No one assigned

Labels

Needs Rebase Needs Tests Windows

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@cgohlke @homm @radarhere @hugovk