Kubernetes on AWS - How to store your logs using Fluent Bit on S3 and later SQL querying them using AWS Athena

Here I’ll show how to systematically store all kubernetes pod logs in S3, never worrying about it getting full, having it auto deleting files after X days and using SQL to query them though AWS Athena, of course leveraging table partitioning so we don’t need to process logs from unwanted timestamps.

After many frustrated attempts to send logs to ElasticSearch, getting the ELK cluster in a red state, processing logs that have never been consumed and getting annoyed with the almost-but-no-really SQL language, I found out a great inexpensive solution:

In all the kubernetes clusters I manage, logs are shipped to AWS S3 using FluentBit, an opensource, lightweight tool written in C that ships logs to multiple places, including AWS S3. After an already merged patch of mine, it can save logs exactly in a format accepted by AWS Athena. Different days can be trivially stored in different S3 paths, which improves how logs can be processed later.

Athena is an AWS managed big data SQL query tool that process files (in our case gzipped logs) stored in AWS S3. One writes a SQL query and it runs through all logs in a specific path (per application, days, etc) to give us results. AWS only charges for data processed by Athena, so if a log is never consumed, you never pay for it.

We can also set S3 to automatically delete logs after 30 days, so the housekeeping is automatic as well. Athena also supports OBCD and JBDC, so one can easily automate periodical log processing tasks.

I plan to post on github examples of how to get everything setup, so you can reproduce the everything you see in this presentation is very few minutes.

I’ve successfully implemented the above setup in multiple k8s clusters at work, some of them generate more than 1 GB of log per hour and we are all quite satisfied with the performance, the easy to use and the cost. All one needs is a working k8s cluster and an AWS account. I am reasonably familiar with fluent bit, for I have written few pull requests to it: https://github.com/fluent/fluent-bit/pulls?q=is%3Apr+author%3Amarcosdiez. I have more than 25 pull requests merged for the kubernetes dashboard https://github.com/kubernetes/dashboard/pulls?q=is%3Apr+author%3Amarcosdiez and about 10 years of AWS experience, including an AWS certification.