Pushing K8s Cluster Logs to S3 Bucket using Fluentd

by Prakarsh -
March 29, 2020
Pushing-K8s-Cluster-Logs-to-S3-Bucket-using-Fluentd-Devtron
Pushing K8s Cluster Logs to S3 Bucket using Fluentd

Storing logs on Elastic search can be very costly, both, in terms of cost as well as in terms of time when you’re trying to retrieve them back. So, in order to save cost, we started sending our Kubernetes cluster logs to AWS S3 bucket, where we would store them till 6 months while keeping the log’s retention policy on Elastic search to only 1 month.

Apart from Devtron Helm Charts deployment feature, you can also use Helm charts to deploy fluent-bit (to collect logs from various pods/deployments on K8s nodes) and fluentd (as an aggregator and forwarder of the logs to s3 or Elastic Search)

Installing Fluent-bit Helm Chart for forwarding K8s Cluster Logs to Fleuntd

Let’s look at the stable/fluent-bit helm chart configurations, you can use it to directly forward the logs to Elastic Search or forward the logs to fluentd for further processing/enrichment.

Edit the fluent-bit-fd-values-2.8.11.yaml below to make the changes mentioned as below.

Forwarding Logs to Fluentd (Required for forwarding logs to S3): To forward Kubernetes cluster logs to fluentd for further enrichment and then forwarding the logs to Elastic search and/or S3 bucket, specify the in-cluster fluentd service as host in the forward section and set the type of the backend to “forward

Forwarding logs to Elastic Search: To forwards Kubernetes cluster logs directly to Elastic Search, you can specify the in cluster elastic search client service name or a hosted elastic search endpoint in the es configurations and set the type of the backend to “es“.

Installing Fluent-bit using Helm
Once you’ve edited the fluent-bit values file according to your use case, use helm install command to install it.

If this is the first time you’re using Helm, please Install and configure Helm-Tiller in your cluster first.

helm install fluent-bit stable/fluent-bit --version 2.8.11 -f fluent-bit-fd-values-2.8.11.yaml

To Uninstall Fluent-bit

helm delete fluent-bit --purge 

fluent-bit-fd-values-2.8.11.yaml



Now that we have configured fluent-bit to collect logs from various pods/deployments in our Kubernetes cluster, we now need an aggregator that aggregates all the logs and writes/pushes them to the required place (files, RDBMS, NoSQL, IaaS, SaaS, Hadoop, elastic search, AWS S3).

Installing Fluentd Helm Chart for forwarding Logs to Elastic Search as well as S3

Let’s look at the stable/fluentd helm chart configurations, we will configure fluentd to send logs collected from fluentbit (or other data sources) to Elastic Search (for shorter retention) as well as to AWS S3 bucket (for longer retention/archive).

Edit the following blocks in the sample fluentd-es-s3-values-2.3.2.yaml file provided below.


Elastic Search Configurations Block
Set the Elastic search configurations in the Elastic Search configuration block. You can set the in-cluster elastic search client service name or a hosted elastic search endpoint as host. (when using in-cluster service, do append the namespace of the elastic search client service separated by a dot, for example – a service elasticsearch-client in a namespace logging will be written as elasticsearch-client.logging in the host)

output:
  host: <elasticsearch-client>.<namespace>
  port: 9200
  scheme: http
  sslVersion: TLSv1
  buffer_chunk_limit: 2M
  buffer_queue_limit: 8


Fluentd Plugins Block
Enable the fluentd plugins and import fluent-plugin-s3 and fluent-plugin-rewrite-tag-filter

plugins:
  enabled: true
  pluginsList: 
    - fluent-plugin-s3
    - fluent-plugin-rewrite-tag-filter


S3 Bucket Configurations Block
Set the S3 configurations in the S3 configurations block. Set the s3_bucket, s3_region, path.

    <match **>
       @type s3
       s3_bucket <k8s-logs-bucket>
       s3_region <ap-southeast-1>
       s3_object_key_format "${tag}/%{time_slice}-events_%{index}.%{file_extension}"
       time_slice_format %Y/%m/%d/%H
       time_slice_wait 10m
       path cluster1-logs
       # if you want to use ${tag} or %Y/%m/%d/ like syntax in path / s3_object_key_format,
       # need to specify tag for ${tag} and time for %Y/%m/%d in <buffer> argument.
       <buffer tag,time>
         @type file
         flush_mode interval
         flush_interval 30s
         path /var/log/fluent/s3
         timekey 300 # 1 hour partition
         timekey_wait 1m
         timekey_use_utc true # use utc
         chunk_limit_size 100m
       </buffer>
       <format>
         @type json
       </format>
    </match>

Installing Fluentd using Helm
Once you’ve made the changes mentioned above, use the helm install command mentioned below to install the fluentd in your cluster.

helm install fluentd-es-s3 stable/fluentd --version 2.3.2 -f fluentd-es-s3-values.yaml

Uninstalling Fluentd

helm delete fluentd-es-s3 --purge 

fluentd-es-s3-values-2.3.2.yaml

Subscribe for DevOps updates

Invalid email address
We promise not to spam you. You can unsubscribe at any time.
Recent posts