Skip to content

Alerting#

Alerting in log-store works by constructing a search query, and providing a webhook endpoint to POST a JSON object to if the query returns any results. The logs returned by the query will be added to the JSON object via a top-level field logs, with an array as a value containing the logs.

Because log-store uses a webhook endpoint, you can build powerful constructs. For example, you can build an “alert” that runs once-an-hour, aggregates some data, and sends the aggregated values to another system for reporting. Given a webhook endpoint which can generate an email, you can receive daily reports about certain logs/events that occurred.

JSON Template#

Additional fields can be added to the JSON object that is sent to the webhook endpoint via the JSON template field. The template must always be a valid JSON object. If additional fields are not required, simply leave the template as an empty object: {}.

The logs returned from the search query specified by the alert are automatically added to the JSON object using a top-level field logs. The value of this field is an array containing JSON objects that represent the logs.

Warning

If a top-level field of logs is specified in the template, it will be overwritten.

For example, using the following template…

{
  "message": "Alert from log-store!",
  "auth-token": "hpZU7pArZq5yqW51CXQ6Ix1DWJ9KvmA+HAe8VlSzz1rbomIt8aozCf9jAWWq/7aKRFisb2kEJ+6j"
}

… and a search query that results in the following logs …

```json lines {“t”:1672365312,”ip”:”60.219.148.3”,”method”:”POST”,”path”:”/clients/zip=13560”,”req_id”:”a38fdb2e-3f2c-43cb-b5f2-11c2640ae3e1”,”resp_code”:200,”response_time”:357,”size”:96,”tier”:”web”} {“t”:1672365013,”lookup_by”:”zip”,”req_id”:”a38fdb2e-3f2c-43cb-b5f2-11c2640ae3e1”,”response_time”:334,”sql”:”SELECT * FROM clients WHERE zip = ?”,”tier”:”app”}


... the following JSON would be sent to the webhook URL:

```json
{
  "message": "Alert from log-store!",
  "auth-token": "hpZU7pArZq5yqW51CXQ6Ix1DWJ9KvmA+HAe8VlSzz1rbomIt8aozCf9jAWWq/7aKRFisb2kEJ+6j",
  "logs": [
    {"t":1672365312,"ip":"60.219.148.3","method":"POST","path":"/clients/zip=13560","req_id":"a38fdb2e-3f2c-43cb-b5f2-11c2640ae3e1","resp_code":200,"response_time":357,"size":96,"tier":"web"},
    {"t":1672365013,"lookup_by":"zip","req_id":"a38fdb2e-3f2c-43cb-b5f2-11c2640ae3e1","response_time":334,"sql":"SELECT * FROM clients WHERE zip = ?","tier":"app"},
    {"t":1672364817,"req_id":"a38fdb2e-3f2c-43cb-b5f2-11c2640ae3e1","response_time":280,"rows":1,"sql":"SELECT * FROM clients WHERE zip = '13560'","tier":"db"}
  ]
}

Advanced JSON Processing#

Adding fields and values to the JSON object which is sent to the webhook endpoint is sufficient for most services. However, some services might require further processing of payload including: adding HTTP headers, inserting secret keys, etc. The best way to handle these situations is to configure Apache’s httpd or Nginx as a proxy that can modify the payload.

Also feel free to request additional features for modifying the JSON object by contacting support.

Search Queries with Commands#

While a basic search query is sufficient to be alerted when certain conditions occur, sometimes more advanced filtering is required. The most common example is filtering the result of a command. This can be done by using the where command.

For example, if you want to be alerted when the average response time over the last 5 minutes is greater than 500ms, you could use a query like:

5m | agg mean(response_time) | where response_time:mean > 500

Remember, if any logs are returned from your query, the alert will trigger, and the logs will be sent to the endpoint.

Triggered Alerts#

The Alert tool records in log-store whenever an alert is triggered, and the result of sending it to the endpoint. This makes debugging alerts easier, and also provides a record of when alerts have fired. Because these events are recorded as logs alongside other logs, you can search for them using the Explore tools just like any other logs. You can see each time an alert has triggered in the past day by simply clicking on the alert.

Alerts that trigger are recorded using the following basic format:

json lines { "source": "log-store", "alert": "${name}", "logs": [], "http_status": 200 } { "source": "log-store", "alert": "${name}", "logs": [], "error_message": "" }

The first format is used when the HTTP POST either succeeds or fails. The status code is recorded in the http_status field. The second format is used when there is some other type of error sending the JSON object to the endpoint. A descriptive message of the failure is added to the error_message field. Both formats will replace ${name} with the name of the alert, and include a timestamp of when the alert was triggered.