Promtail의 중요한 디테일

compressed file을 ingest하는 것도 지원을 한다. 하지만 속도가 느린듯하다. 압축된 파일을 laze하게 압축 해제한다고 강조를 하는 걸 보니 말이다.

scrape_configs:
- job_name: system
  decompression:
    enabled:trueinitial_delay: 10s
    format: gz

 

 

• It relies on the \n character to separate the data into different log lines.
• The max expected log line is 2MB within the compressed file.
• The data is decompressed in blocks of 4096 bytes. i.e: it first fetches a block of 4096 bytes from the compressed file and processes it. After processing this block and pushing the data to Loki, it fetches the following 4096 bytes, and so on.
• It supports the following extensions:
  • .gz: Data will be decompressed with the native Gunzip Golang pkg (pkg/compress/gzip)
  • .z: Data will be decompressed with the native Zlib Golang pkg (pkg/compress/zlib)
  • .bz2: Data will be decompressed with the native Bzip2 Golang pkg (pkg/compress/bzip2)
  • .tar.gz: Data will be decompressed exactly as the .gz extension. However, because tar will add its metadata at the beginning of the compressed file, the first parsed line will contains metadata together with your log line. It is illustrated at ./clients/pkg/promtail/targets/file/decompresser_test.go.
• .zip extension isn’t supported as of now because it doesn’t support some of the interfaces Promtail requires.
• The decompression is quite CPU intensive and a lot of allocations are expected to occur, especially depending on the size of the file. You can expect the number of garbage collection runs and the CPU usage to skyrocket, but no memory leak is expected.
• Positions are supported. That means that, if you interrupt Promtail after parsing and pushing (for example) 45% of your compressed file data, you can expect Promtail to resume work from the last scraped line and process the rest of the remaining 55%.

• Since decompression and pushing can be very fast, depending on the size of your compressed file Loki will rate-limit your ingestion. In that case you might configure Promtail’s limits stage to slow the pace or increase ingestion limits on Loki.
• Log rotations on compressed files are not supported (log rotation is fully supported for normal files), mostly because it requires us modifying Promtail to rely on file inodes instead of file names.
• If you compress a file under a folder being scraped, Promtail might try to ingest your file before you finish compressing it. To avoid it, pick a initial_delay that is enough to avoid it.