Attaching Logs

ITS-Outpost-Healthchecks ping endpoints accept HTTP HEAD, GET and POST request methods.

When using HTTP POST, you can include an arbitrary payload in the request body. ITS-Outpost-Healthchecks will log the first 10 kB (10000 bytes) of the request body, so that you can inspect it later.

Logging Command Output

In this example, we run certbot renew, capture its output (both the stdout and stderr streams), and submit the captured output to ITS-Outpost-Healthchecks:

#!/bin/sh

m=$(/usr/bin/certbot renew 2>&1)
curl -fsS -m 10 --retry 5 --data-raw "$m" https://its-outpost-healthchecks.fzi.de/ping/your-uuid-here

We can extend the previous example and signal either success or failure depending on the exit code:

#!/bin/sh

m=$(/usr/bin/certbot renew 2>&1)
curl -fsS -m 10 --retry 5 --data-raw "$m" https://its-outpost-healthchecks.fzi.de/ping/your-uuid-here/$?

If the command produces a lot of output, you may run into the following error:

/usr/bin/curl: Argument list too long

In that case, one workaround is to save the output to a temporary file, then tell curl to send the file as the request body:

#!/bin/sh

/usr/bin/certbot renew > /tmp/certbot-renew.log 2>&1
curl -fsS -m 10 --retry 5 --data-binary @/tmp/certbot-renew.log https://its-outpost-healthchecks.fzi.de/ping/your-uuid-here/$?

Using Runitor

Runitor is a third-party utility that runs the supplied command, captures its output and reports to ITS-Outpost-Healthchecks. It also measures the execution time and retries HTTP requests on transient errors. Best of all, the syntax is simple and clean:

runitor -uuid your-uuid-here -- /usr/bin/certbot renew

Sending Logs Without Signalling Success or Failure

You may sometimes want to log diagnostic information without altering the check's current state. ITS-Outpost-Healthchecks provides the /log endpoint just for that. When you send an HTTP POST request to this endpoint, ITS-Outpost-Healthchecks will log the event and display it in check's "Events" section but will keep the check's state unchanged.

Handling More Than 10 kB of Logs

While ITS-Outpost-Healthchecks can store a small amount of logs in a pinch, it is not specifically designed for that. If you run into the issue of logs getting cut off, consider the following options:

• See if the logs can be made less verbose. For example, if you have a batch job that outputs a line of text per item processed, perhaps it can output a summary with the totals instead.
• If the important content is usually at the end, submit the last 10 kB instead of the first. Here is an example that submits the last 10 kB of dmesg output:

#!/bin/sh

m=$(dmesg | tail --bytes=10000)
curl -fsS -m 10 --retry 5 --data-raw "$m" https://its-outpost-healthchecks.fzi.de/ping/your-uuid-here

• Finally, if it is critical to capture the entire log output, consider using a dedicated log aggregation service for capturing the logs.

Where to See Captured Logs

In the check's details page, Events section, click on individual events to see full event details, including the captured log information.

In the dialog that opens, use the "Download Original" link to download the request body data, exactly as it was submitted to ITS-Outpost-Healthchecks: