Comprehensive Guide to Trivy and Trivy-Operator: From Scanning to Monitoring with Prometheus and Grafana

Introduction

Set Up a Kubernetes Cluster

Install Minikube

Start Minikube

minikube start

minikube start --driver=<NameOfDriver>

What is Trivy-Operator

Install Trivy-Operator

1. Add Aqua Security Helm repository:

helm repo add aqua https://aquasecurity.github.io/helm-charts/

2. Update all registries:

helm repo update

3. Install Trivy-Operator:

helm install trivy-operator aqua/trivy-operator --namespace trivy-system --create-namespace --values values.yaml --version 0.25.0

trivy:   
  ignoreUnfixed: true

helm install trivy-operator aqua/trivy-operator --namespace trivy-system --create-namespace --set trivy.ignoreUnfixed=true --version 0.25.0

kubectl get all -n trivy-system

Automated Vulnerability Scanning

1. Create a Namespace

kubectl create namespace nginx

2. Deploy Nginx:

kubectl create deployment nginx --image=nginx:1.16 -n nginx 

3. View the reports

kubectl get vulnerabilityreports -n nginx -o wide
kubectl get configauditreport -n nginx -o wide

Visualize Trivy-Operator Metrics in Grafana

Installing Prometheus and Grafana

1. Create a Namespace:

kubectl create namespace monitoring

2. Add and Update the Helm Chart Repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

3. Customize Prometheus to Discover ServiceMonitors:

prometheus:
  prometheusSpec:
    # Allows overriding the default Helm value
    serviceMonitorSelectorNilUsesHelmValues: false

    # Select all ServiceMonitors regardless of label
    serviceMonitorSelector: {}
    # Watch for ServiceMonitors in all namespaces
    serviceMonitorNamespaceSelector: {}

4. Install Prometheus and Grafana

Install the kube-prometheus-stack using the customized values file:

helm upgrade --install prom prometheus-community/kube-prometheus-stack -n monitoring --values values.yaml

5. Configure Trivy-Operator to Expose Metrics

serviceMonitor:
  enabled: true
trivy:
  ignoreUnfixed: true

6. Upgrade Trivy-Operator with updated values:

helm upgrade trivy-operator aqua/trivy-operator --namespace trivy-system --create-namespace --values values.yaml --version 0.25.0

7. Access the Prometheus and the Grafana Dashboard

kubectl port-forward service/prom-kube-prometheus-stack-prometheus -n monitoring 9090:9090

kubectl port-forward service/prom-grafana -n monitoring 3000:80

Accessing and Exploring Metrics in Grafana

• Username: admin
• Password: prom-operator (or extract the password using kubectl get secret -n monitoring prom-grafana -o jsonpath="{.data.admin-password}" | base64 --decode)

1. In Grafana, go to Dashboards > New Dashboard.
2. Click Add visualization
3. Select Prometheus as the data source.
4. In the Query Editor, use the following queries to get useful insights.

sum(trivy_image_vulnerabilities)

sum(trivy_resource_configaudits)

sum(trivy_image_exposedsecrets)

topk(5, sum by (namespace) (trivy_image_vulnerabilities))

kubectl delete deploy nginx -n nginx

Trivy's Scanning Capabilities

trivy image [YOUR_IMAGE_NAME]

trivy image --scanners vuln [YOUR_IMAGE_NAME]

trivy image --image-config-scanners misconfig [YOUR_IMAGE_NAME]

trivy image --image-src podman, containerd alpine:3.7.3

trivy fs /path/to/project

trivy fs /path/to/file

trivy fs --scanners misconfig /path/to/project

trivy repo (REPO_PATH | REPO_URL)

trivy repo ./

trivy repo https://github.com/aquasecurity/trivy-ci-test

Features, Security, and Compliance Mapping

• Vulnerability Report: Automated vulnerability scanning to detect issues in workloads.
• Configuration Audit Report: Automated configuration audits for detecting flag misconfiguration or custom rules.
• Exposed Secret Report: Automated secrets scan to find and report leaked sensitive data within the cluster.
• RBAC Assessment Report: RBAC scan to evaluate and provide information about the installed resources permissions.
• Core Infrastructure Assessment Report: Analyses and scans core Kubernetes components (etcd, apiserver, scheduler, controller-manager, etc.).

• NSA & CISA Kubernetes Hardening Guidance: Provides cybersecurity recommendations and report.
• CIS Kubernetes Benchmark:  Ensures cluster configuration aligns with security best practices.
• Kubernetes Pod Security Standards: Workload validation against restricted policy.

Avoiding Common Pitfalls in Container Security Scanning

• Ignoring Low and Medium Severity Issues 
  Focusing only on critical vulnerbailities and ignoring low and medium severity issues could end badly. Attackers often chain together several small vulnerabilities to create a serios exploit. It is recommended that you track all vulnerabilites using severity-based alerts, and not only the critical ones.
• Forgetting to Scan Base Images
  Most vulnerabilites are in base image, so avoid just focusing on the application layer. Scanning the base image should be part of the pipeline. This will ensures that images are scanned and updated regularly.
• Not Acting on Findings
  Even with a scanning pipeline in place, findings often end up being ignored if there isn't a clear process in place. There should be a clear vulnerability management workflow, such as how findings are tracked, who ist notified if there is a finding, what is the SLA for fixing the issue etc.
• Minimal Runtime Protection
  Image scanning only protects when containers are deployed. Containers are left unprotected at runtime and do not catch unauthorized changes. To avoid this, use content-aware firewalls, monitor containers for anomalous actions, regularly check the configs for misconfigurations and use runtime threat detection tools like Falco.
• Vulnerable Container Registry
  Unprotected registries are rewarded targets and put the entire container pipeline at risk. If one compromises the registry, the key to infect each and every image is in her hand. To avoid this, prevent unauthorized access, avoid unencrypted network and use TLS certificates.

Conclusion

References: