Skip to content

Config File#

log-store is configured primarily via a config file in TOML format. Command line options can be used to temporarily override some of the settings found in the config file.

The sections below describe the settings you specify in the config file to set up log-store. Most settings are optional with sane defaults.

Global Settings#

These settings are specified at the top of the config file as key/value pairs.

  • log_file - File location to store log-store’s logs. If a location is not provided, logs are written to standard out.
  • license_file - The location of the license file to use log-stor with. If you do not have a license file, please contact sales.
  • timestamp_field - The name of the JSON field that contains the timestamp. If this is left blank, it defaults to t.
  • timestamp_format - The format of the above-mentioned timestamp. There are 3 options: EPOCH, RFC2822, and RFC3339.
    • EPOCH (default) - The number of seconds or milliseconds (automatically determined) since Jan 1st 1970 at midnight in the UTC timezone.
    • RFC2822 - A long format timestamp specified in RFC-2822, Section 3.3; example Tue, 1 Jul 2003 10:52:37 +0200
    • RFC3339 - A compact format timestamp specified in RFC-3339; example 1996-12-19T16:39:57-08:00.
  • save_location - Optional parameter to specify where dashboards and saved-searched are saved: server (default) or browser. If they are saved on the server, they are shared by all users. In the browser, they are not shared.
  • web_address - The IP address and port for the web portal to be served from. Defaults to localhost and 8181 if not specified.
  • receive_handlers - The number of threads to use to receive logs; defaults to 2. If you are running on a low-powered device, try setting this value to 1. If you want to maximize the throughput of logs, you can set the value higher than 2; however, setting this value higher than the number of cores on the machine will result in reduced throughput.
  • page_pool_gbs - A floating value indicating the number of GiBs (gibibyte) of memory to use. log-store will use more memory than this setting specifies, but this setting specifies how much caching to use for data. This is particularly important if a lot of data is stored on S3 compatible storage, as it will save on network bandwidth. The default value is 75% of the memory on the machine. log-store will refuse to run on 2 MiBs of memory or less, and will produce a warning if it is set to 1,024 MiBs of memory or less.

WAL Settings#

log-store will optionally use a WAL (write-ahead log) to ensure logs are not lost if log-store does not shutdown cleanly. It is highly recommended to give log-store up to 120s to close to ensure everything is properly written to disk.

If you are performing a bulk insert of historical logs, then disable the WAL (set to false), load all the historical logs, shutdown log-store, and start it back up again with the WAL enabled.

  • use_wal - Boolean to enable or disable the use of WAL files. When enabled, ingestion speed is about 30% slower, but log-store can replay logs from the WAL file after a crash or unclean shutdown. Defaults to True.
  • wal_flush_count - How often the WAL should be flushed to the column and index files. The lower the number, the more often logs are persisted to column and index files, but ingest speed is decreased. The higher the number, the less often. Acceptable values are between 10 and 10,000,000; defaults to 100,000. This value is not used if use_wal is set to false.

Log Inputs#

These settings are also specified at the top of the config file, and configure how logs are received by log-store. You do not need to specify all of them, or even any of them. log-store can run in a read-only mode without specifying any inputs. See the section on Adding Logs for more information on the format logs need to be in, to be added to log-store.

  • unix_socket - The path for log-store to set up a Unix domain socket for local log ingesting. (See Adding Logs for more information.)
  • tcp_input_address - The IP address and port for log-store to ingest logs via TCP. You can use 0.0.0.0 for the first available IP address. The port will default to 1234 if one is not supplied. (See Adding Logs for more information.)
  • syslog_address - The IP address and port for log-store to ingest logs in Syslog format. (See Adding Logs for more information.)
  • syslog_protocol - Optional parameter to specify the transport for Syslog logs: TCP (default) or UDP.

Retention Settings#

Log retention is configured by a TOML table [retention], that contains two values: hot_days & warm_days. The total number of days a log will exist in log-store before being deleted is simply hot_days + warm_days. You do not need to specify this table; however, if you do, you must specify values for both hot_days and warm_days.

  • hot_days - The number of days logs should be kept in “hot” storage, or local disk. After this many days, logs will be rolled off to warm storage if configured, or deleted if not. The minimum value is 1, and the default value is 30 days.
  • warm_days - The number of days logs should be kept in “warm” storage, or S3 compatible storage. After this many days, logs will be deleted. The default value is 0 days.
  • data_dir_days - How many days worth of data should be stored in a single directory. This is the highest-level partitioning of data in log-store. Once set, it should NOT be changed or else you will not be able to find data correctly.

Storage Settings#

There are two TOML tables used to configure storage: [hot_storage] and [warm_storage]. The [hot_storage] table is required, because the minimum value for hot_days is 1.

[hot_storage]#

  • data_dir - The directory where log-store will store all of its data. This directory should be on a disk with a lot of free space. This setting is required.

[warm_storage]#

The [warm_storage] table configures storage of logs in an S3 compatible object store. This table is optional, but must be specified if the warm_days value is greater than zero. Moving logs from local disk to an S3 compatible object store is seamless to the end user. The only difference in searching logs on S3 is the performance.

Warning

It is HIGHLY recommended that you configure a bucket just for log-store, and that you create keys that only have permission to access this bucket. These keys (both access and secret) are stored in a config file that can be read by anyone who has access to the server.

  • access_key_id - The access key required to access the bucket.
  • secret_access_key - The secret key required to access the bucket.
  • region - The region the bucket resides in.
  • bucket - Name of the bucket.

Sample Config File#

The sample config file below shows a typical configuration of log-store, relying on defaults for those settings which would be considered only for performance tweaking:

log_file = "/var/log/log-store.log"  # you can use logrotate to rotate this log
license_file = "/etc/log-store/log-store.license"

# defaults to (t) for the timestamp field, and EPOCH for the format
# timestamp_field = 't'
# timestamp_format = 'EPOCH'

save_location = "server"  # searches and dashboards are saved on the server for all to use/see
web_address = "0.0.0.0:8181"  # bind to all interfaces on port 8181

# receive_handlers = 2  # defaults to 2 threads
# page_pool_gbs = 2  # defaults to 75% of the total memory on the system

# WAL settings
# use_wal = True  # defaults to True
# wal_flush_count = 100000  # defaults to 100,000

# Inputs
#unix_socket = ="/var/run/log-store.socket"
tcp_input_address = "0.0.0.0:1234"
#syslog_address = "127.0.0.1:1501"
#syslog_protocol="udp"

# retention settings
[retention]
hot_days = 7  # keep 7 days of logs on-disk
warm_days = 400  # then keep ~13 months of logs in S3 

# on-disk settings
[hot_storage]
data_dir = "/var/lib/log-store"  # location to store logs, saved searches, and dashboards

# settings for S3 storage
[warm_storage]
access_key_id = "MY_ACCESS_KEY"
secret_access_key = "MY_SECRET_ACCESS_KEY"
region = "us-east-1"
bucket = "log-store-archive"