Send white spaces to client till completeMultipart() process completes by krishnasrinivas · Pull Request #7198 · minio/minio

krishnasrinivas · 2019-02-05T23:41:36Z

Description

FS backend can take a long time in completeMultiPartUpload() if parts came in an order not favorable to background append OR if bandwidth between server and client is so high that background append is not fast enough causing completeMultiPartUpload() to be not quick. In such a situation we can send white spaces to the client just like aws-s3 server so that the client connection does not time out.

http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadComplete.html

Processing of a Complete Multipart Upload request could take several minutes to complete. After Amazon S3 begins processing the request, it sends an HTTP response header that specifies a 200 OK response. While processing is in progress, Amazon S3 periodically sends whitespace characters to keep the connection from timing out. Because a request could fail after the initial 200 OK response has been sent, it is important that you check the response body to determine whether the request succeeded.

#3223

Motivation and Context

Regression

No

How Has This Been Tested?

tested by putting sleep/print statements to see if whitespace was being sent.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have added unit tests to cover my changes.
I have added/updated functional tests in mint. (If yes, add mint PR # here: )
All new and existing tests passed.

cmd/object-handlers.go

codecov · 2019-02-06T00:19:30Z

Codecov Report

Merging #7198 into master will increase coverage by 0.04%.
The diff coverage is 63.63%.

@@            Coverage Diff             @@
##           master    #7198      +/-   ##
==========================================
+ Coverage   49.04%   49.09%   +0.04%     
==========================================
  Files         289      289              
  Lines       45207    45229      +22     
==========================================
+ Hits        22174    22203      +29     
+ Misses      21005    21001       -4     
+ Partials     2028     2025       -3

Impacted Files	Coverage Δ
cmd/object-handlers.go	`54.21% <63.63%> (+0.11%)`	⬆️
cmd/fs-v1-helpers.go	`68.8% <0%> (-0.31%)`	⬇️
cmd/posix.go	`65.38% <0%> (+0.32%)`	⬆️
cmd/xl-v1-metadata.go	`87.7% <0%> (+0.66%)`	⬆️
cmd/xl-sets.go	`36.9% <0%> (+0.74%)`	⬆️
cmd/bitrot-streaming.go	`82.52% <0%> (+3.88%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e4081ae...2e50a02. Read the comment docs.

minio-ops · 2019-02-06T01:22:41Z

Mint Automation

Test	Result
mint-compression-xl.sh	✔️
mint-xl.sh	✔️
mint-large-bucket.sh	✔️
mint-compression-dist-xl.sh	✔️
mint-compression-fs.sh	✔️
mint-worm.sh	✔️
mint-fs.sh	✔️
mint-dist-xl.sh	❌ more...
mint-gateway-nas.sh	❌ more...

7198-2e50a02/mint-gateway-nas.sh.log:

Running with
SERVER_ENDPOINT:      72.28.97.52:30569
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 1909cf89bbc4:/mint/log /tmp/mint-logs'

(1/13) Running aws-sdk-go tests ... done in 1 seconds
(2/13) Running aws-sdk-java tests ... done in 0 seconds
(3/13) Running aws-sdk-php tests ... done in 41 seconds
(4/13) Running aws-sdk-ruby tests ... FAILED in 2 seconds
{
  "name": "aws-sdk-ruby",
  "function": "presignedPut(bucket_name,file_name)",
  "args": {
    "bucket_name": "aws-sdk-ruby-bucket-a07f1825231d",
    "file_name": "datafile-1-MB"
  },
  "duration": 21.43,
  "error": "Connection reset by peer",
  "status": "FAIL"
}

Executed 3 out of 13 tests successfully.

7198-2e50a02/mint-dist-xl.sh.log:

Running with
SERVER_ENDPOINT:      72.28.97.52:31897
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 26f1faa05174:/mint/log /tmp/mint-logs'

(1/13) Running aws-sdk-go tests ... done in 1 seconds
(2/13) Running aws-sdk-java tests ... done in 0 seconds
(3/13) Running aws-sdk-php tests ... done in 45 seconds
(4/13) Running aws-sdk-ruby tests ... done in 4 seconds
(5/13) Running awscli tests ... done in 1 minutes and 54 seconds
(6/13) Running mc tests ... done in 32 seconds
(7/13) Running minio-dotnet tests ... FAILED in 2 minutes and 51 seconds

Unhandled Exception: System.AggregateException: One or more errors occurred. (One or more errors occurred. (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.)) ---> System.AggregateException: One or more errors occurred. (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) (Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.) ---> Minio.Exceptions.InternalClientException: Minio API responded with message=Unsuccessful response from server without XML error: An error occurred while sending the request. The server returned an invalid or unrecognized response.
   at Minio.MinioClient.ParseError(IRestResponse response) in /q/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 458
   at Minio.MinioClient.HandleIfErrorResponse(IRestResponse response, IEnumerable`1 handlers, DateTime startTime) in /q/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 513
   at Minio.MinioClient.ExecuteTaskAsync(IEnumerable`1 errorHandlers, IRestRequest request, CancellationToken cancellationToken) in /q/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 359
   at Minio.MinioClient.PutObjectAsync(String bucketName, String objectName, String uploadId, Int32 partNumber, Byte[] data, Dictionary`2 metaData, Dictionary`2 sseHeaders, CancellationToken cancellationToken) in /q/.q/sources/minio-dotnet/Minio/ApiEndpoints/ObjectOperations.cs:line 492
   at Minio.MinioClient.PutObjectAsync(String bucketName, String objectName, Stream data, Int64 size, String contentType, Dictionary`2 metaData, ServerSideEncryption sse, CancellationToken cancellationToken) in /q/.q/sources/minio-dotnet/Minio/ApiEndpoints/ObjectOperations.cs:line 254
   at Minio.Functional.Tests.FunctionalTest.PutObject_Task(MinioClient minio, String bucketName, String objectName, String fileName, String contentType, Int64 size, Dictionary`2 metaData, MemoryStream mstream) in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 923
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Minio.Functional.Tests.FunctionalTest.RemoveObjects_Test2(MinioClient minio) in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 2134
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Minio.Functional.Tests.FunctionalTest.Main(String[] args) in /mint/run/core/minio-dotnet/FunctionalTest.cs:line 221

Executed 6 out of 13 tests successfully.

challarao · 2019-02-06T07:47:30Z

Thank you so much guys. The white spaces are sent regardless of the backend right?

krishnasrinivas · 2019-02-06T07:49:48Z

@challarao yes

challarao · 2019-02-06T09:49:17Z

Thanks Krishna. This is needed for Abort Multipart as well?

krishnasrinivas · 2019-02-06T10:12:04Z

abort is always quick @challarao

arnaudbos · 2019-02-11T15:20:36Z

Hi 👋
I think this PR introduces a regression, let me elaborate:

We're running minio GCS gateway on Google Kubernetes Engine for our end-to-end tests and since last week (we're running minio's docker :latest tag...) we're having parsing errors of the Java AWS SDK's "completeMultipartUpload" method because of whitespaces preceding the XML content.

Googling around we've stumbled upon this issue which has been fixed in this PR in june 2016 with code that added the XML header first, then whitespaces and finally the content.

But then about two month later this has been changed to not send whitespaces anymore, which, I assume, still works as long as timeouts are long enough.

And finally this PR reintroduces the whitespaces, but before the XML header, which seems to break the Java AWS SDK parser.

Let me know if my analysis is correct or not, I'll try a new deployment with the tag RELEASE.2019-01-31T00-31-19Z.

harshavardhana · 2019-02-11T16:23:49Z

Whitespaces should be ignored, this is a bug in AWS SDK implementation.

We will validate it nonetheless.

krishnasrinivas · 2019-02-11T20:55:19Z

🤐

arnaudbos · 2019-02-12T11:09:49Z

FWIW our tests run fine with RELEASE.2019-01-31T00-31-19Z. Sticking to that for now 🙂

ghost · 2019-02-21T06:19:36Z

@arnaudbos We have the same problem as described by you. Have you reported the issue to the Java AWS S3 SDK team yet? Our workaround for now is to catch the exception and make sure the completeMultipart() was succeeded by checking if the object exists.

harshavardhana · 2019-02-21T07:52:52Z

The issue regarding Java SDK should be fixed #7253

arnaudbos · 2019-02-21T07:55:23Z

Oh wow, that was fast!

Thank you for all the hard work on Minio, it works well for our use case and saved us a lot of time to test our system's multipart upload on GCP without switching from AWS SDK.

harshavardhana reviewed Feb 5, 2019

View reviewed changes

cmd/object-handlers.go Outdated Show resolved Hide resolved

kannappanr requested review from donatello and harshavardhana February 5, 2019 23:49

krishnasrinivas force-pushed the send-white-spaces-complete-mpart branch from e5954f4 to d5f55a9 Compare February 6, 2019 00:00

harshavardhana reviewed Feb 6, 2019

View reviewed changes

cmd/object-handlers.go Outdated Show resolved Hide resolved

Send white spaces to client till completeMultipart() process completes

2e50a02

krishnasrinivas force-pushed the send-white-spaces-complete-mpart branch from d5f55a9 to 2e50a02 Compare February 6, 2019 00:36

harshavardhana approved these changes Feb 6, 2019

View reviewed changes

donatello approved these changes Feb 6, 2019

View reviewed changes

kannappanr merged commit 3dfbe0f into minio:master Feb 6, 2019

filimonov mentioned this pull request Oct 14, 2022

Increase request_timeout_ms for s3 disks. ClickHouse/ClickHouse#42321

Merged

Enmk mentioned this pull request Jan 23, 2023

22.8 Backport of #42321 s3 request_timeout_ms Altinity/ClickHouse#222

Merged

Conversation

krishnasrinivas commented Feb 5, 2019

Description

Motivation and Context

Regression

How Has This Been Tested?

Types of changes

Checklist:

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Feb 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

minio-ops commented Feb 6, 2019

Mint Automation

7198-2e50a02/mint-gateway-nas.sh.log:

7198-2e50a02/mint-dist-xl.sh.log:

Uh oh!

challarao commented Feb 6, 2019

Uh oh!

krishnasrinivas commented Feb 6, 2019

Uh oh!

challarao commented Feb 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krishnasrinivas commented Feb 6, 2019

Uh oh!

arnaudbos commented Feb 11, 2019

Uh oh!

harshavardhana commented Feb 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krishnasrinivas commented Feb 11, 2019

Uh oh!

arnaudbos commented Feb 12, 2019

Uh oh!

ghost commented Feb 21, 2019

Uh oh!

harshavardhana commented Feb 21, 2019

Uh oh!

arnaudbos commented Feb 21, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecov bot commented Feb 6, 2019 •

edited

Loading

challarao commented Feb 6, 2019 •

edited

Loading

harshavardhana commented Feb 11, 2019 •

edited

Loading