-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Closed
Labels
feature request*Deprecated Label* Use enhancement label in general*Deprecated Label* Use enhancement label in generalv1
Description
Fluentd's output plugin somtimes hit un-recoverable error during chunk flush.
- chunk contains wrong record for output configuration
- output plugin has a bug for specific record
- broken chunk is generated by hardware problem
- wrong setup for destination
Currently, we use retry limit and secondary for handling these chunks but it has several problems.
- bad chunk occupy flush threads until reach retry limit
- non retry limit environment can't rescue bad chunk
So we should care bad chunk for stability and performance.
The idea is if output plugin raises un-recoverable error during chunk flush, such chunks are routed to backup directory.
- The recoverable errors are TypeError, NoMethodError, ArgumentError, etc
- Use UnrecoverableError for plugin specific error
In addition, <system> directive provide backup_dir parameter. The default is /tmp/fluentd.
mururu, handysama, mhietamaki, gmile and JiHyunSong
Metadata
Metadata
Assignees
Labels
feature request*Deprecated Label* Use enhancement label in general*Deprecated Label* Use enhancement label in generalv1