Skip to content

The /imp rest endpoint does not support quoted multipart boundary definition #5757

@rmja

Description

@rmja

To reproduce

Consider the following example using the .NET HttpClient:

            using var fileStream = file.OpenRead();
            using var request = new HttpRequestMessage(HttpMethod.Post, "http://localhost:9000/imp");
            var form = new MultipartFormDataContent()
            {
                {
                    new StreamContent(fileStream) { Headers = { ContentType = new("text/plain") } },
                    "data"
                },
            };

            // Needed to remove quotes around the boundary parameter
            // var boundary = form.Headers.ContentType.Parameters.First(o => o.Name == "boundary");
            // boundary.Value = boundary.Value.Replace("\"", "");

            request.Content = form;
            using var response = await httpClient.SendAsync(request, cancellationToken);
            response.EnsureSuccessStatusCode();

It produces a HTTP request that looks like this:

POST /imp HTTP/1.1
Host: localhost:9000
Content-Type: multipart/form-data; boundary="2cf10b2d-a962-4791-bfaf-9227872d500d"
Content-Length: 279347

--2cf10b2d-a962-4791-bfaf-9227872d500d
Content-Type: text/plain
Content-Disposition: form-data; name=data

Note the qoutes around the boundary parameter.
When I run this, I get "malformed start boundary" in the logs, which seems to occur here:

throw HttpException.instance("Malformed start boundary");
}
break;
case PRE_HEADERS:
switch (Unsafe.getUnsafe().getByte(ptr)) {
case '\n':
state = HEADERS;
// fall through
case '\r':
ptr++;
break;
case '-':
// make sure that we set the status to DONE only after we read the second '-'
if (!firstDashRead) {
firstDashRead = true;
// on the first '-' we just need to read the next byte
ptr++;
break;
}
processor.onPartEnd();
state = DONE;
return true;
default:
processor.onChunk(boundary.lo(), boundary.hi());
_lo = ptr;
state = BODY;
break;
}
break;
case START_PRE_HEADERS:
switch (Unsafe.getUnsafe().getByte(ptr)) {
case '\n':
state = START_HEADERS;
// fall through
case '\r':
ptr++;
break;
case '-':
return true;
default:
throw HttpException.instance("Malformed start boundary");

According to rfc2046 it is in fact allowed to include quotes around the boundary parameter in the Content-Type header.

A workaround is to remove the quotes manually from the header as shown in the example by including string Replace("\"", "")

QuestDB version:

8.3.3

OS, in case of Docker specify Docker and the Host OS:

Windows 11, Docker questdb/questdb:latest

File System, in case of Docker specify Host File System:

ntfs

Full Name:

Rasmus Melchior Jacobsen

Affiliation:

Utiliread

Have you followed Linux, MacOs kernel configuration steps to increase Maximum open files and Maximum virtual memory areas limit?

  • Yes, I have

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugIncorrect or unexpected behavior

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions