Skip to content

Document and type annotate UploadFile as a bytes-only interface. Not bytes or text.#1312

Merged
adriangb merged 19 commits intoKludex:masterfrom
adriangb:fix-UploadFile-type-annotations
Jan 10, 2022
Merged

Document and type annotate UploadFile as a bytes-only interface. Not bytes or text.#1312
adriangb merged 19 commits intoKludex:masterfrom
adriangb:fix-UploadFile-type-annotations

Conversation

@adriangb
Copy link
Copy Markdown
Contributor

No description provided.

if file is None:
file = tempfile.SpooledTemporaryFile(max_size=self.spool_max_size)
self.file = file
self.file = tempfile.SpooledTemporaryFile(max_size=self.spool_max_size) # type: ignore # noqa: E501
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SpooledTemporaryFile isn't fully typed, see python/typeshed#1780

@adriangb
Copy link
Copy Markdown
Contributor Author

It looks like tests are failing due to missing coverage, which is because I split up the branch on line 428, which I only did because type checkers didn't like assigning SpooledTemporaryFile to BytesIO (as per #1312 (comment))

So I guess the thing to do here is add a test for that branch?

@Kludex
Copy link
Copy Markdown
Owner

Kludex commented Nov 15, 2021

Why is this a bug?

Documentation needs to be fixed in case this is a bug: https://www.starlette.io/requests/#request-files, jfyk

@adriangb
Copy link
Copy Markdown
Contributor Author

Hmm, good point.

I was looking at this from the perspective of actual usage within Starlette.
As far as I can tell, UploadFile is only ever used in formparsers.py, where only bytes are written to it:
https://github.com/encode/starlette/blob/6af5c515e0a896cbf3f86ee043b88f6c24200bcf/starlette/formparsers.py#L225
The default file is also opened in bytes mode:
https://github.com/encode/starlette/blob/6af5c515e0a896cbf3f86ee043b88f6c24200bcf/starlette/datastructures.py#L425
Finally, I don't see anywhere in the tests where UploadFile.write or UploadFile.read are called with str.

So I assumed these were just bogus type annotations.
But I guess if it is in the docs, then this is not a bug report and instead is a request to deprecate the functionality of using str with UploadFile since it's is not used by Starlette and is not tested anywhere.
This would make it easier to use UploadFile in a typed environment since currently every time you call UploadFile.read you have to do assert isinstance(data, bytes) or a cast to make type checkers happy.

@adriangb adriangb changed the title bug: fix type annotations in UploadFile BREAKING_CHANGE: deprecate untested usage of UploadFile with str Nov 15, 2021
@adriangb
Copy link
Copy Markdown
Contributor Author

adriangb commented Nov 15, 2021

I pushed commits to:

  • Add the missing test for coverage (again, this is just testing something that was previously untested but was not being counted towards coverage because branch coverage is not being used)
  • Updates the docs to reflect the change

I also renamed the issue from a bug to a request for deprecation (technically a breaking change, although I doubt anyone was using UploadFile directly w/ str). Also, this is just type hints and docs, so even if it is a breaking change it isn't going to break anyone's code aside from static type checking.

@adriangb
Copy link
Copy Markdown
Contributor Author

adriangb commented Jan 4, 2022

@tomchristie I'm curious if you remember why UploadFile accepts strings in the first place? Like I mentioned above I don't see anywhere in Starlette where it is used that way, and I can't think of a reason why it would.

Comment on lines +230 to +239
@pytest.mark.anyio
async def test_upload_file_file_input():
"""Test passing file/stream into the UploadFile constructor"""
stream = io.BytesIO(b"data")
file = UploadFile(filename="file", file=stream)
assert await file.read() == b"data"
await file.write(b" and more data!")
assert await file.read() == b""
await file.seek(0)
assert await file.read() == b"data and more data!"
Copy link
Copy Markdown
Contributor Author

@adriangb adriangb Jan 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also this, I don't see anywhere else where the file parameter to UploadFile is being used. It makes sense as an attribute, but we're not using it in the constructor for testing or anything. But that would really be a breaking change (instead of being just type annotations, like this PR) so it probably doesn't make sense to change.

@lovelydinosaur
Copy link
Copy Markdown

It's not really clear to me what's motiving this change.

We have str and bytes here because files might be in text or binary mode. It's possible that we only end up using binary mode within starlette, but it'd be good to have a better idea from an end-user / API perspective why the change is valuable. (Since all change also risks unintended breakages)

@adriangb
Copy link
Copy Markdown
Contributor Author

adriangb commented Jan 7, 2022

I view it as a breaking change because:

  1. It changes the docs
  2. It changes type annotations which might break user's tests if they are using static type checkers.

But like I said, in practice I don't think this would break anything for anyone.

We have str and bytes here because files might be in text or binary mode.

Are you referring to general file-like-object?
To me if UploadFile is only ever used in binary mode, it should be typed as such. I do not see the value in treating it as if it could be a file in text mode if in practice it is always going to be binary mode.

it'd be good to have a better idea from an end-user / API perspective why the change is valuable

I think it's valuable for 2 reasons:

  1. Saves boilerplate for users that run MyPy on their codebase. Currently, we have to do assert isinstance(await file.read(), UploadFile) or cast(...) or # type: ignore
  2. Simpler documentation and cognitive load. I for one was pretty confused from the documentation because I was trying to figure out in what situation I'd be getting back strings.

another option would be to make upload file a generic class and then annotate it as UploadFile[bytes] in FormData. But my understanding is thst you don't want to introduce generics to the codebase, so I did not consider that.

Copy link
Copy Markdown

@lovelydinosaur lovelydinosaur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm fine with this yup.

I think it could do with a slight change of title perhaps, and I don't consider it a breaking change. (the typing is tightened up, but the behaviour isn't actually changed.)

Eg. "Document and type annotate UploadFile as a bytes-only interface. Not bytes or text."

@adriangb adriangb changed the title BREAKING_CHANGE: deprecate untested usage of UploadFile with str Document and type annotate UploadFile as a bytes-only interface. Not bytes or text. Jan 10, 2022
@adriangb
Copy link
Copy Markdown
Contributor Author

Done @tomchristie , and synced with the main branch

@adriangb adriangb merged commit 4e86245 into Kludex:master Jan 10, 2022
@adriangb adriangb deleted the fix-UploadFile-type-annotations branch January 10, 2022 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants