HowTo: Use the Logging-Operator
Many people are leveraging fluentd, fluent-bit or the ELK-stack in combination with filebeat to collect their logs from Kubernetes clusters. These options are often complicated to configure or not flexible enough to use for different teams.
We were looking for an easy way to collect our logs. Besides that, the customer wanted an option to do this, too. In the end we decided to use the logging-operator by banzaicloud. It is utilizing a DaemonSet of fluent-bit to collect all the logs from the containers running on the nodes. This fluent-bit sends the logs to a central fluentd instance, where the logs can be modified. In the end the logs will be sent to a central logging system like Loki, elasticsearch, AWS Cloudwatch, etc. (Output documentation: https://banzaicloud.com/docs/one-eye/logging-operator/configuration/plugins/outputs/)
Log stream separation
To configure the logging-operator we will have to create a few resources. I will guide you through an exemplary process to configure two different log streams:
- The first log stream should collect all the logs from our operating namespaces (kube-system, monitoring, logging, tracing). These logs should be forwarded to our own Loki instance.
- The second log stream should collect logs from all the other namespaces so that we will have an example to collect customer logs. These logs should be forwarded to AWS Cloudwatch.
Installing the logging-operator
The first thing we have to do is installing the logging-operator itself. We will do this with the helm-chart banzaicloud has created. It would be possible to do this with a few YAML files, but for this short tutorial we will use helm:
helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
helm repo update
helm install logging-operator banzaicloud-stable/logging-operator --set createCustomResource=false
Now we are able to see the logging-operator Pod:
kubectl get pods
NAME READY STATUS RESTARTS AGE
logging-operator-7494f75887-62bzw 1/1 Running 0 11s
This pod will observe the Kubernetes cluster and waits for the following custom resources:
- Logging
- Flow/ClusterFlow
- Output/ClusterOutput
The Logging-Resource defines a logging system. This logging system contains the fluent-bit Daemonset and the fluentd StatefulSet. If you want to configure these systems separate to another you will have to define separate logging-systems.
Flow or ClusterFlow will define the various filters, you want to use. For example, you can define, which logs should be processed, or which keys should be removed from the log message.
Output and ClusterOutput will define, to which system the logs are sent. You are able to configure the output specific settings, e.g. username and password for the system.
Creating the logging system
In order to create the necessary CRDs we will install the second helm-chart that is available by banzaicloud:
helm install logging-operator-logging banzaicloud-stable/logging-operator-logging -f ./logging-operator-logging.yaml
We are defining some values in the separate values-file logging-operator-logging.yaml. The content of this file is defining the ClusterFlow and the ClusterOutput resource, e.g. it is selecting all the logs from the management namespaces or the URL of the Loki instance. In addition, it is adding some extra labels like the cluster name or region to all log-messages:
nameOverride: logging-operator-monitoring-stack
clusterFlows:
- name: all-system-pods
spec:
match:
- select:
namespaces:
- kube-system
- monitoring
- logging
- tracing
globalOutputRefs:
- loki
clusterOutputs:
- name: loki
spec:
loki:
url: http://my-loki.default.svc.cluster.local:3100
configure_kubernetes_labels: true
extra_labels:
cluster: eks-test-cluster
region: eu-west-1
After installing the helm-chart we can grab the pods again and we will see that the logging-operator created the fluent-bit Daemonset and the fluentd StatefulSet:
kubectl get pods
NAME READY STATUS RESTARTS AGE
logging-operator-7494f75887-62bzw 1/1 Running 0 12m
logging-operator-monitoring-stack-fluentbit-gdqsh 1/1 Running 0 44s
logging-operator-monitoring-stack-fluentbit-rfqrb 1/1 Running 0 44s
logging-operator-monitoring-stack-fluentbit-x6r8d 1/1 Running 0 44s
logging-operator-monitoring-stack-fluentd-0 2/2 Running 0 45s
Now we are able to collect and store the log messages for our management namespaces. They are forwarded to Loki, as defined in the logging-operator-logging.yaml. The second log stream is created in the same way with the following values file. Be aware that you have to create an IAM user which is able to write into AWS Cloudwatch (Plugins documentation: https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs). The credentials for this user should be provided in a Kubernetes Secret.
nameOverride: logging-operator-customer-stack
clusterFlows:
- name: all-customer-pods
spec:
match:
- exclude:
namespaces:
- kube-system
- monitoring
- logging
- tracing
globalOutputRefs:
- cloudwatch
clusterOutputs:
- name: cloudwatch
spec:
cloudwatch:
aws_key_id:
valueFrom:
secretKeyRef:
name: logging-s3
key: awsAccessKeyId
aws_sec_key:
valueFrom:
secretKeyRef:
name: logging-s3
key: awsSecretAccessKey
log_group_name: customer-log-group
log_stream_name: customer-log-stream
region: eu-west-1
auto_create_stream true
buffer:
timekey: 30s
timekey_wait: 30s
timekey_use_utc: true
Conclusion
All in all, we are able to set up various log streams to collect our logs from applications. The customers are able to define their own log streams with little effort, too. We are flexible in modifying these logs and are able to send them to various log systems. The setup is quite easy and we are able to set it up in a few minutes.
All the code examples can be found in our GithHub repository.
This is the second post in the observability series. Our first post “Monitor your Multi-Cluster-Environments” can be found here. The next post which describes how you can setup and configure Loki in a distributed way will be released on 1st March.
References
Photo by Lukas Blazek on Unsplash
https://banzaicloud.com/docs/one-eye/logging-operator/
https://github.com/fluent-plugins-nursery/fluent-plugin-cloudwatch-logs