Skip to content

fix: authorize upload error on create job endpoint#1642

Merged
davidgamez merged 13 commits intomasterfrom
fix/1632-authorize-upload
Jan 16, 2024
Merged

fix: authorize upload error on create job endpoint#1642
davidgamez merged 13 commits intomasterfrom
fix/1632-authorize-upload

Conversation

@davidgamez
Copy link
Copy Markdown
Member

@davidgamez davidgamez commented Jan 15, 2024 •
edited
Loading

Summary:

When utilizing the feed URL https://www.maricopa-az.gov/home/showpublisheddocument/6491/638307291320330000, an "Error authorizing upload" occurs. Upon attempting to save the file to GCP for validation, it was observed that the HTTP request lacked a user-agent, leading to rejection by the producer in this specific instance.

This is a missing fix from #1633.
Fixes #1632

Please make sure these boxes are checked before submitting your pull request - thanks!

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 15, 2024

✅ Rule acceptance tests passed.
New Errors: 0 out of 1479 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1479 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1479 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1479 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1479 sources (~0 %) are corrupted.
Commit: b146777
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

davidgamez
davidgamez commented Jan 15, 2024
View reviewed changes
web/service/src/main/java/org/mobilitydata/gtfsvalidator/web/service/util/StorageHelper.java
storage.signUrl(
blobInfo, 1, TimeUnit.HOURS, Storage.SignUrlOption.httpMethod(HttpMethod.POST));
try (WriteChannel writer = storage.writer(signedURL)) {
OutputStream outputStream = Channels.newOutputStream(writer);
Copy link
Copy Markdown
Member Author

@davidgamez davidgamez Jan 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using the Channels output stream reduces the memory footprint by avoiding loading the full file in memory.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jan 15, 2024

✅ Rule acceptance tests passed.
New Errors: 0 out of 1479 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Errors: 0 out of 1479 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
New Warnings: 0 out of 1479 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
Dropped Warnings: 0 out of 1479 datasets (~0%) are invalid due to code change, which is less than the provided threshold of 1%.
0 out of 1479 sources (~0 %) are corrupted.
Commit: 3517178
Download the full acceptance test report here (report will disappear after 90 days).
✅ Rule acceptance tests passed.

@davidgamez davidgamez marked this pull request as ready for review January 16, 2024 02:30
@davidgamez davidgamez changed the title Fix/1632 authorize upload fix: authorize upload error on create job endpoint Jan 16, 2024
@davidgamez davidgamez requested a review from jcpitre January 16, 2024 14:17
@davidgamez davidgamez merged commit 5384a3e into master Jan 16, 2024
@davidgamez davidgamez deleted the fix/1632-authorize-upload branch January 16, 2024 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jcpitre jcpitre jcpitre approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Feed generates error to authorize upload

2 participants

@davidgamez @jcpitre