Description
When using awslogs driver with non-blocking logging mode to prevent containers hanging to mitigate issues with CloudWatch service, ECS tasks log events greater than 16 KB are split into multiple events in CloudWatch logs.
The same ECS service deployment without "mode": "non-blocking" log options treats log events longer than 16 KB as a single CloudWatch log event.
Steps to reproduce the issue:
One can reproduce this issue on Docker, ECS EC2 launch type, Fargate 1.3.0 and Fargate 1.4.0 launch types by running a container image that outputs 17 KB:
Here is my Dockerfile to build image
$ cat Dockerfile
FROM ubuntu:18.04
CMD tr -dc A-Za-z0-9 </dev/urandom | head -c 17000; echo ''
...
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/nonblocking",
"awslogs-region": "us-east-1",
"mode": "non-blocking",
"awslogs-stream-prefix": "ecs"
}
}
Also Running directly using docker command below has same result for non-blocking mode.
docker run -d --log-driver=awslogs --log-opt mode=non-blocking --log-opt max-buffer-size=4m --log-opt awslogs-group="/ecs/log-event-split-test" --log-opt mode=non-blocking --log-opt awslogs-region=ap-southeast-2 hello2parikshit/log-event-split-test:latest
Describe the results you received:
When the logging mode is set to default (blocking), the entire line is sent as a single event to CloudWatch.
When set to non-blocking, the event is split and sent to CloudWatch Logs with the exact same timestamp.
Describe the results you expected:
When set to non-blocking, the cloud watch event should be recorded entire log event without splitting to CloudWatch Logs with the exact same timestamp.
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
docker version
Client:
Version: 19.03.13-ce
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46
Built: Mon Oct 12 18:51:20 2020
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.13-ce
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46
Built: Mon Oct 12 18:51:50 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.1
GitCommit: c623d1b36f09f8ef6536a057bd658b3aa8632828
runc:
Version: 1.0.0-rc92
GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Output of docker info:
docker info
Client:
Debug Mode: false
Server:
Containers: 66
Running: 17
Paused: 0
Stopped: 49
Images: 8
Server Version: 19.03.13-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: amazon-ecs-volume-plugin local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c623d1b36f09f8ef6536a057bd658b3aa8632828
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0 (expected: fec3683)
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.203-156.332.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.618GiB
Name: ip-172-31-43-108.ap-southeast-2.compute.internal
ID: 7ENC:ICR2:4LS6:BQ2N:EQYC:FNOI:IRSW:VGCI:5LVB:5KYQ:CHXW:7XSR
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 147
Goroutines: 172
System Time: 2020-12-15T02:52:14.501116615Z
EventsListeners: 1
Username: hello2parikshit
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.): AWS EC2, Fargate PV 1.3.0, Fargate PV 1.4.0
Description
When using awslogs driver with non-blocking logging mode to prevent containers hanging to mitigate issues with CloudWatch service, ECS tasks log events greater than 16 KB are split into multiple events in CloudWatch logs.
The same ECS service deployment without "mode": "non-blocking" log options treats log events longer than 16 KB as a single CloudWatch log event.
Steps to reproduce the issue:
One can reproduce this issue on Docker, ECS EC2 launch type, Fargate 1.3.0 and Fargate 1.4.0 launch types by running a container image that outputs 17 KB:
Here is my Dockerfile to build image
Also Running directly using docker command below has same result for non-blocking mode.
Describe the results you received:
When the logging mode is set to default (blocking), the entire line is sent as a single event to CloudWatch.
When set to non-blocking, the event is split and sent to CloudWatch Logs with the exact same timestamp.
Describe the results you expected:
When set to non-blocking, the cloud watch event should be recorded entire log event without splitting to CloudWatch Logs with the exact same timestamp.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.): AWS EC2, Fargate PV 1.3.0, Fargate PV 1.4.0