RECLAIM THE STACK!

Log Aggregation: Qryn

Qryn is a Loki compatible log (and metrics) aggregation system using ClickHouse under the hood. Promtail is used to feed logs into qryn. Finally we configure Grafana from our Monitoring stack to use qryn as a data source.

Github: https://github.com/metrico/qryn
Documentation: https://qryn.metrico.in

Architecture Decision Record

Pros

  • Simple monolithic architecture (qryn HTTP server + ClickHouse as data store)
  • Fully Loki / LogQL compatible
  • Also supports other data sources (we don't make use of this currently)
  • ClickHouse as a backend equals excellent performance and scalability
  • ClickHouse can be configured to use S3 for storage for increased durability and "bottomless" storage

Cons

  • Requires extra configuration for ClickHouse
  • Not as widely used as Loki

Alternatives Considered

Loki

At Mynewsdesk we use Loki for our in house version of the platform. It has served us quite well but architecturally it has significantly more moving parts than Qryn which means it becomes harder to reason about. It's also dependent on S3 if running in it's "scalable" mode which makes it harder to create a good "it just works" experience for our open source platform.

We have also run into the odd performance issue with Loki, especially when it comes to querying large amounts of data. Loki version 2.8.0 introduced a new TSBD Index which promises a 4x performance improvement which may bring performance on par with ClickHouse. We have not yet had the chance to test this out.

Usage

You can tail logs from applications via k logs <application-name>. You can also filter the logs based on type of deployment by passing the types as additional arguments to the command. Eg. k logs <application-name> web sidekiq.

To do advanced search, filtering and aggregation of logs, you can use the Explore view in Grafana and selecting the Logging datasource. Run k logs:search <application-name> to quickly open the Grafana log UI for a specific application.

Read more on how to work with logs in the Grafana UI in their Logs in Explore documentation.