Automated monitoring for Kubernetes platforms

Auto-scaling requires auto-monitoring

Dynamic IT infrastructures are characterised by the fact that they can quickly scale IT resources for applications up or down according to the needs of the business departments. But no matter to what extent the resources are available: they must be properly monitored and logged.

A company in the automotive industry was faced with the challenge of keeping a dynamically scaling system transparent. The dynamics resulted from multiple autoscaling infrastructures and a complex, shifting microservice application. The manual effort for the long-term storage of observation data therefore had to be reduced to a minimum and at the same time be cost-efficient.

The requirements

A legacy monitoring and logging stack, which did not meet the requirements, as well as a lack of standards in the entire system posed central hurdles.

• The monitoring and logging system had to recognise scaling events and was not allowed to trigger false-positive alarms

• Traditional monitoring systems rely on host names or IP addresses, static information that does not change. In dynamic environments these parameters sometimes change very often, which is why this information could not be used

• The monitoring and logging systems had to be scalable across different AWS accounts. The observability systems therefore required autoscaling functions

Adjusted open source projects

Liquid Reply set up a monitoring environment that meets the requirements to monitor and log infrastructure as well as applications in the best possible way.

For the solution, the Prometheus, Thanos, Loki and Grafana projects of the Cloud Native Computing Foundation (CNCF) were used.

This revealed the lack of standards throughout the system, especially in the overall monitoring of various AWS services. Liquid Reply therefore set up a cross-account network with native AWS services.

To overcome the outdated monitoring and logging stack, Liquid Reply relied on Helm charts developed by the open source community, extending them with various functions that met the client’s requirements.

Automated monitoring of new clusters

Within a short period of time, Liquid Reply succeeded in establishing a new, stable and secure monitoring and logging approach. By integrating it, the client is now able to create new Kubernetes clusters which are automatically monitored and logged by the central monitoring cluster. This reduces the potential for manual failures of IT resources and indirectly reduces costs.

Why this matters beyond this project

Monitoring that has to be reconfigured by hand every time the infrastructure scales will always lag behind the platform it is supposed to watch. The decisive move in this engagement was therefore not the tool selection but the operating model: observability defined as code, shipped with every new cluster by default, and centralised so that long-term storage, alerting rules and dashboards exist exactly once.

The same architecture — Prometheus for collection, Thanos for long-term storage across accounts, Loki for logs and Grafana on top — has since become a common baseline for Kubernetes platforms, and it remains our default recommendation when a legacy monitoring stack cannot keep up with autoscaling: it is open source end to end, it scales with the platform, and it removes the manual toil that causes monitoring gaps in the first place.