Kubernetes Logging
Logging in Kubernetes isn't supposed to be complicated. As an app developer, you push to containers and forget. As an Kubernetes maintainer, you collect from the various sources and push where ever you want to maintain these logs.
The log collection and forwarding really stepped up when a common approach could be setup for all of them. This is where fluent-bit, fluentd have shined in the last 5 years. Fluent-bit offers an excellent high performant, optimised method for collection which can collect a wide variety of sources. However it is limited by the number of possible output plugins that are available. It is possible to create output plugins for them in Golang and new experimental support is being attempted for Rust.
On the other hand, fluentd offers the enormous flexibility of Ruby, and writing a custom plugin is easy enough. However, the memory utilisation of fluentd even with available optimisations, means it is not well suited for running on every node.
Add to this the fact that the recommended architecture from fluentd calls for a separation in the log aggregation and forwarding layers. A structure that was observed in Sumologic's helm charts for logging was quite interesting.
Fluent-bit for the collection running as a daemonset, Fluentd for aggregation and plugin related work running as a statefulset in cluster itself. This is an excellent way to combine the strengths of both ecosystems, which ensures that you minimise the resource consumption on each node, and are able to leverage the plugin ecosystem of fluentd.
Attaching a repo , with example configuration of this you can run in your own clusters. Just replace the fluentd output config to whichever log aggregation system you use.
Other Interesting Alternatives,
The next level of optimisation in the collection layer could be a rust based forwarder, since it offers similar benefits to fluent-bit. Vector, now owned by Datadog provides a rich ecosystem for pushing logs/metrics to various sinks like Elasticsearch, AWS Cloudwatch logs etc.