Skip to content

Vector keeps old Inodes, clogging up disk #11742

@MaxRink

Description

@MaxRink

A note for the community

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

We ran into the issue that Vector is keeping old Inodes open, thus filling up the disk

root@mdo-1-8vwrt:/home/a92615428# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4        49G   49G     0 100% /

INodes: https://gist.github.com/MaxRink/ee056e27a4b11a7b710e437e1f892984
After pkilling vector disk usage returned to normal

root@mdo-1-8vwrt:/home/a92615428# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4        49G  8.5G   38G  19% /
```̀

### Configuration

```text
api:
  address: 0.0.0.0:8686
  enabled: true
  playground: false
data_dir: /var/lib/vector/
sinks:
  prometheus_metrics:
    address: 0.0.0.0:9090
    inputs:
    - internal_metrics
    type: prometheus_exporter
  vector_sink:
    address: vector-aggregator:9000
    inputs:
    - kubernetes_logs
    - annotate_hostname
    type: vector
    version: "2"
sources:
  internal_metrics:
    type: internal_metrics
  json_logfiles_var_log:
    include:
    - /var/log/kube-apiserver/kube-apiserver.log
    max_line_bytes: 1536000
    type: file
  kubernetes_logs:
    type: kubernetes_logs
  logfiles_var_log:
    exclude:
    - /var/log/pods/**/*.log
    - /var/log/containers/**/*.log
    - /var/log/kube-apiserver/kube-apiserver.log
    - /var/log/kube-apiserver/kube-apiserver*.log
    - /var/log/**/*.gz
    - /var/log/journal/**/*
    include:
    - /var/log/**/*.log
    - /var/log/private
    - /var/log/lastlog
    - /var/log/syslog
    - /var/log/btmp
    - /var/log/faillog
    - /var/log/wtmp
    - /var/log/dmesg
    max_line_bytes: 409600
    type: file
transforms:
  annotate_hostname:
    inputs:
    - logfiles*
    source: .host = "${VECTOR_SELF_NODE_NAME}"
    type: remap
  logfiles_json_decode:
    inputs:
    - json_logfiles_var_log
    source: .msg_decoded = parse_json!(.message)
    type: remap

Version

0.20.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    source: fileAnything `file` source relatedtype: bugA code related bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions