Skip to content
This repository was archived by the owner on Sep 27, 2023. It is now read-only.

Latest commit

 

History

History
83 lines (50 loc) · 8.96 KB

README.http_input.md

File metadata and controls

83 lines (50 loc) · 8.96 KB

Logplex HTTP Input

Logplex supports syslog-over-http (application/logplex-1) as an input source. Messages contained in an HTTP POST request will be individually routed in the same way as messages from the syslog-over-tcp transport that Logplex also supports. These messages will be delivered to tail sessions, logplex drains and buffered in redis as usual.

Authentication

Logplex HTTP Input uses the HTTP basic authentication mechanism. The username and password for basic authentication can either be based on an existing Logplex Token, or a new Logplex Credential.

Logplex token authentication

Logplex tokens allow you to write any number of messages to a single channel - the channel the token was created for. A logplex token takes the form t.fc6b856b-3332-4546-93de-7d0ee272c3bd. To use this with HTTP basic authentication, set the username to the string token and the password to the logplex token (e.g. "token:t.fc6b856b-3332-4546-93de-7d0ee272c3bd").

Logplex credential authentication

Logplex credentials are a new and more flexible authentication and authorization scheme for logplex. The credentials can be granted additional permissions such as the ability to write into any logplex channel (logplex tokens can only write into one channel)[1]. Logplex credentials can also be used to access the Logplex HTTP token/drain/session/channel API.

Logplex credentials are random usernames and passwords generated by a logplex cluster. At the moment, these credentials must be created manually by calling:

1> Cred = logplex_cred:new().
{cred,<<"807F8703C9739F61C78FC7B79972F27A">>,
  <<"1B2FF9E5757B284617E70CFCABCF4E1BB1A3AFE49C86BB7B58C0B96D26C33DCB">>,
  []}
2> logplex_cred:store(Cred).
ok

In this example, the username for basic authentication will be 807F8703C9739F61C78FC7B79972F27A (the #logplex_cred.id) and the password will be 1B2FF9E5757B284617E70CFCABCF4E1BB1A3AFE49C86BB7B58C0B96D26C33DCB (#logplex_cred.pass). Once this credential is granted the any_channel permission you'll be able to use it to submit logs.

Mime-type application/logplex-1

Messages sent to logplex via HTTP input must be encoded and sent using the mimetype: application/logplex-1. This mime-type is equivalent to a sequence of RFC5452 encoded messages with octet count framing.

To encode the message "Logsplat test message 1 from <0.40.0>", we first have to fill out all the fields of a syslog message as this is just the message body part. The most important part is to set the syslog message header app name to the logplex token (i.e. t.d6799f88-4a77-402f-b197-2b722a02cdbc). Other syslog fields we need to set are facility local0, the message priority info, the time of 2012-12-10T03:00:48Z+00:00, the host erlang, the process id to console.1 and the message id to the undefined value -. The facility and priority get encoded according to RFC5452 as <134>1 (the 1 after the brackets is the version of the syslog protocol we're using). You can use just about any value you like in the time/host/process id/message id fields as long as the values don't contain space characters.

Put together the syslog encoded message is:

<134>1 2012-12-10T03:00:48Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 1 from <0.40.0>.

Each message must then be framed - prepended with the length of the encoded message string in bytes. The length is the number of bytes of the encoded message, not the number of characters. Our example above is 130 bytes long, so when framed, it will look like:

130 <134>1 2012-12-10T03:51:41Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 1 from <0.40.0>.

That is the string 130, a space, then the 130 bytes of syslog encoded message.

If you want to include a return at the end of a message, you need to count that character as part of the length of the message:

131 <134>1 2012-12-10T03:51:41Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 1 from <0.40.0>.\n

You generally don't need to include a linefeed at the end of a message as Logplex output formatting will ensure that lines don't run together.

Because the individual messages are prepended with their length, logplex can accurately split a series of them back into individual messages. This example shows the encoding of 10 similar messages:

130 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 1 from <0.72.0>.130 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 2 from <0.72.0>.130 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 3 from <0.72.0>.130 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 4 from <0.72.0>.130 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 5 from <0.72.0>.130 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 6 from <0.72.0>.130 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 7 from <0.72.0>.130 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 8 from <0.72.0>.130 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 9 from <0.72.0>.131 <134>1 2012-12-10T03:56:29Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 10 from <0.72.0>.

There are no characters or spaces between the end of a message body and the start of a new frame (the number indicating the length of the new message).

This block of encoded messages is the mimetype application/logplex-1 and will be the body of the HTTP POST request you make to input the messages to Logplex.

POSTing log messages to Logplex over HTTP

Once you have encoded some messages as application/logplex-1 you can POST them as input to logplex.

In addition to the usual required HTTP headers, you MUST set the following headers:

  • Authorization to an appropriate HTTP Basic auth string (e.g. "token:t.d6799f88-4a77-402f-b197-2b722a02cdbc" --> "Basic dG9rZW46dC5kNjc5OWY4OC00YTc3LTQwMmYtYjE5Ny0yYjcyMmEwMmNkYmM=")
  • Content-Type to application/logplex-1
  • Content-Length to the length of the application/logplex-1 message body. (You can't use chunked transfer encoding at this time)

You MAY also set the following headers:

  • Logplex-Msg-Count to the number of messages present in the body. This allows Logplex to double check your request and make sure that you have encoded your messages in the same way the logplex understands to decode them.
  • Logplex-Frame-Id a new unique value if this is the first time you have transmitted this request, or the same value if you are retransmitting the request. (This as-yet-unimplemented feature allows Logplex to de-duplicate messages upon retransmission - it makes the request idempotent)

The following is the text of a POST to https://logplex.example.com using the token t.d6799f88-4a77-402f-b197-2b722a02cdbc for authentication. The request contains 10 messages:

POST /logs HTTP/1.1
Host: logplex.example.com
Authorization: Basic dG9rZW46dC5kNjc5OWY4OC00YTc3LTQwMmYtYjE5Ny0yYjcyMmEwMmNkYmM=
Content-Type: application/logplex-1
Logplex-Msg-Count: 10
Logplex-Frame-Id: 361838DD34D7C363F22B8E1C95948C8F
User-Agent: Logsplat/0.1.0
Content-Length: 1341

130 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 1 from <0.72.0>.130 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 2 from <0.72.0>.130 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 3 from <0.72.0>.130 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 4 from <0.72.0>.130 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 5 from <0.72.0>.130 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 6 from <0.72.0>.130 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 7 from <0.72.0>.130 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 8 from <0.72.0>.130 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 9 from <0.72.0>.131 <134>1 2012-12-10T04:05:25Z+00:00 erlang t.d6799f88-4a77-402f-b197-2b722a02cdbc console.1 - Logsplat test message 10 from <0.72.0>.