Many people are leveraging fluentd, fluent-bit or the ELK-stack in combination with filebeat to collect their logs from Kubernetes clusters. These options are often complicated to configure or not flexible enough to use for different teams.

We were looking for an easy way to collect our logs. Besides that, the customer wanted an option to do this, too. In the end we decided to use the logging-operator by banzaicloud. It is utilizing a DaemonSet of fluent-bit to collect all the logs from the containers running on the nodes. This fluent-bit sends the logs to a central fluentd instance, where the logs can be modified. In the end the logs will be sent to a central logging system like Loki, elasticsearch, AWS Cloudwatch, etc. (Output documentation: https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/)

Log stream separation

To configure the logging-operator we will have to create a few resources. I will guide you through an exemplary process to configure two different log streams:

1. The first log stream should collect all the logs from our operating namespaces (kube-system, monitoring, logging, tracing). These logs should be forwarded to our own Loki instance.
2. The second log stream should collect logs from all the other namespaces so that we will have an example to collect customer logs. These logs should be forwarded to AWS Cloudwatch.

Installing the logging-operator

The first thing we have to do is installing the logging-operator itself. We will do this with the helm-chart banzaicloud has created. It would be possible to do this with a few YAML files, but for this short tutorial we will use helm:  

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
helm repo update
helm install logging-operator banzaicloud-stable/logging-operator --set createCustomResource=false

Now we are able to see the logging-operator Pod:

kubectl get pods

NAME                                READY   STATUS    RESTARTS   AGE
logging-operator-7494f75887-62bzw   1/1     Running   0          11s

This pod will observe the Kubernetes cluster and waits for the following custom resources:

• Logging
• Flow/ClusterFlow
• Output/ClusterOutput

The Logging-Resource defines a logging system. This logging system contains the fluent-bit Daemonset and the fluentd StatefulSet. If you want to configure these systems separate to another you will have to define separate logging-systems.

Flow or ClusterFlow will define the various filters, you want to use. For example, you can define, which logs should be processed, or which keys should be removed from the log message.

Output and ClusterOutput will define, to which system the logs are sent. You are able to configure the output specific settings, e.g. username and password for the system.

Creating the logging system

In order to create the necessary CRDs we will install the second helm-chart that is available by banzaicloud:

helm install logging-operator-logging banzaicloud-stable/logging-operator-logging -f ./logging-operator-logging.yaml

We are defining some values in the separate values-file logging-operator-logging.yaml. The content of this file is defining the ClusterFlow and the ClusterOutput resource, e.g. it is selecting all the logs from the management namespaces or the URL of the Loki instance. In addition, it is adding some extra labels like the cluster name or region to all log-messages:

nameOverride: logging-operator-monitoring-stack
clusterFlows:
  - name: all-system-pods
    spec:
      match:
        - select:
            namespaces:
            - kube-system
            - monitoring
            - logging
            - tracing
      globalOutputRefs:
        - loki
clusterOutputs:
  - name: loki
    spec:
      loki:
        url: http://my-loki.default.svc.cluster.local:3100
        configure_kubernetes_labels: true
        extra_labels:
          cluster: eks-test-cluster
          region: eu-west-1       

After installing the helm-chart we can grab the pods again and we will see that the logging-operator created the fluent-bit Daemonset and the fluentd StatefulSet:

kubectl get pods

NAME                                                             READY   STATUS      RESTARTS   AGE
logging-operator-7494f75887-62bzw                                1/1     Running     0          12m
logging-operator-monitoring-stack-fluentbit-gdqsh                1/1     Running     0          44s
logging-operator-monitoring-stack-fluentbit-rfqrb                1/1     Running     0          44s
logging-operator-monitoring-stack-fluentbit-x6r8d                1/1     Running     0          44s
logging-operator-monitoring-stack-fluentd-0                      2/2     Running     0          45s

Now we are able to collect and store the log messages for our management namespaces. They are forwarded to Loki, as defined in the logging-operator-logging.yaml. The second log stream is created in the same way with the following values file. Be aware that you have to create an IAM user which is able to write into AWS Cloudwatch (Plugins documentation: https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs). The credentials for this user should be provided in a Kubernetes Secret.

nameOverride: logging-operator-customer-stack
clusterFlows:
  - name: all-customer-pods
    spec:
      match:
        - exclude:
            namespaces:
            - kube-system
            - monitoring
            - logging
            - tracing
      globalOutputRefs:
        - cloudwatch
clusterOutputs:
  - name: cloudwatch
    spec:
      cloudwatch:
        aws_key_id:
          valueFrom:
            secretKeyRef:
              name: logging-s3
              key: awsAccessKeyId
        aws_sec_key:
          valueFrom:
            secretKeyRef:
              name: logging-s3
              key: awsSecretAccessKey
        log_group_name: customer-log-group
        log_stream_name: customer-log-stream
        region: eu-west-1
        auto_create_stream true
        buffer:
          timekey: 30s
          timekey_wait: 30s
          timekey_use_utc: true

Conclusion

All in all, we are able to set up various log streams to collect our logs from applications. The customers are able to define their own log streams with little effort, too. We are flexible in modifying these logs and are able to send them to various log systems. The setup is quite easy and we are able to set it up in a few minutes.

All the code examples can be found in our GithHub repository.

---

This is the second post in the observability series. Our first post “Monitor your Multi-Cluster-Environments” can be found here. The next post which describes how you can setup and configure Loki in a distributed way will be released on 1st March.

References

Photo by Lukas Blazek on Unsplash

https://banzaicloud.com/docs/one-eye/logging-operator/

https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs

https://github.com/Liquid-Reply/logging-operator-demo