File(s)

This module allows the Security Engine to acquire logs from text files (in one-shot and streaming mode), and GZ files in one-shot mode.

Configuration example

A basic configuration is as follows:

YAML

source: file
filenames:
 - /tmp/foo/*.log
 - /var/log/syslog
labels:
 type: syslog

Parameters

filename

A single path to a file to tail. Globbing is supported. Required if filenames is not provided.

filenames

A list of paths to files to tail. Globbing is supported. Required if filename is not provided.

force_inotify

default: false

If set to true, force an inotify watch on the log files folder, even if there is no log in it.

source

Must be file.

exclude_regexps

A list of regular expressions to exclude from the acquisition. Can be used to exclude files from a glob pattern (ie, * but not *.log.gz).

poll_without_inotify

info

This was not the default for version 1.4.6 and below. So users upgrading to 1.5 may encounter some issues with certain file systems. See this issue

default: false

If set to true, will poll the files using os.Stat instead of using inotify. This is useful if you want to watch files on a network share, for example. However, this will increase CPU usage significantly per file that is open.

discovery_poll_enable

default: false

If set to true, will periodically rescan the configured filename patterns and tail files that are not already being tailed. This is useful when the filesystem does not emit create events, for example on NFS.

discovery_poll_interval

default: 30s

How often to rescan when discovery_poll_enable is true. A duration such as 30s, 1m, or 1h. 0 uses the default of 30 seconds. A negative value is rejected.

DSN and command-line

This module supports acquisition directly from the command line, to read files in one shot.

A single file URI is accepted with the -dsn parameter, but globbing is supported for multiple files:

SH

crowdsec -type syslog -dsn file:///var/log/*.log

Supported parameters

log_level

Change the log level for the acquisition:

SH

crowdsec -type syslog -dsn file:///var/log/*.log?log_level=info

max_buffer_size

Maximum length of a single line.

Defaults to 65536.

SH

crowdsec -type syslog -dsn file:///var/log/*.log?max_buffer_size=42000

Notes

By default, if a glob pattern does not match any files in an existing directory, this directory will not be watched for new files (ie, /var/log/nginx/*.log does not match, but /var/log/nginx/ exists). You can override this behaviour with the force_inotify parameter, which will put a watch on the directory. If the filesystem does not notify the Security Engine of new files, set discovery_poll_enable to true instead. The patterns are rescanned every discovery_poll_interval (default 30s).