Monitoring

Kubebuilder projects use controller-runtime to implement controllers and admission webhooks. controller-runtime instruments several key metrics related to controllers and webhooks by default using kubernetes instrumentation guidelines. and makes them available via HTTP endpoint in prometheus metric format.

Following metrics are instrumented by default:

• Total number of reconcilation errors per controller
• Length of reconcile queue per controller
• Reconcilation latency
• Usual resource metrics such as CPU, memory usage, file descriptor usage
• Go runtime metrics such as number of Go routines, GC duration

To quickly examine metrics in your development environment, you can run the following:

# launch manager
$ make run

# in another terminal, access the metrics

$ curl http://localhost:8080/metrics
# HELP controller_runtime_reconcile_errors_total Total number of reconcile errors per controller
# TYPE controller_runtime_reconcile_errors_total counter
controller_runtime_reconcile_errors_total{controller="mysql-controller"} 10
# HELP controller_runtime_reconcile_queue_length Length of reconcile queue per controller
# TYPE controller_runtime_reconcile_queue_length gauge
controller_runtime_reconcile_queue_length{controller="mysql-controller"} 0
# HELP controller_runtime_reconcile_time_seconds Length of time per reconcile per controller
# TYPE controller_runtime_reconcile_time_seconds histogram
controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="0.005"} 10
controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="0.01"} 10
controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="0.025"} 10
controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="10"} 10
controller_runtime_reconcile_time_seconds_bucket{controller="mysql-controller",le="+Inf"} 10
controller_runtime_reconcile_time_seconds_sum{controller="mysql-controller"} 2.3416e-05
controller_runtime_reconcile_time_seconds_count{controller="mysql-controller"} 10
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 7.69e-05
go_gc_duration_seconds{quantile="0.25"} 0.0001225
go_gc_duration_seconds{quantile="0.5"} 0.000124351
go_gc_duration_seconds{quantile="0.75"} 0.000236344
go_gc_duration_seconds{quantile="1"} 0.000262102
go_gc_duration_seconds_sum 0.000822197
go_gc_duration_seconds_count 5
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 39
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.9.4"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
.....
....

Is the metrics endpoint protected ?

Yes. By default, kubebuilder generated YAML manifests (under config/ dir) ensures that the access to metrics endpoint is authenticated and authorized using an auth proxy which is deployed as sidecar container in the manager pod. You can read more details about the auth proxy based approach here.

If you want to disable the auth proxy, which is not recommended, you can follow the instructions in the Kustomization file located in config/default/kustomization.yaml

If your project was created using 1.0.5 or older kubebuilder, you need to modify the following files as show in PR #513.

• cmd/manager/main.go
• config/default/kustomization.yaml
• config/default/manager_auth_proxy_patch.yaml
• config/rbac/auth_proxy_role.yaml
• config/rbac/auth_proxy_role_binding.yaml
• config/rbac/auth_proxy_service.yaml

How do I configure Prometheus Server to access the metrics?

Kubebuilder generated manifests for manager have annotations such as prometheus.io/scrape, prometheus.io/path on the metrics service so that it can be easily discovered by the prometheus server deployed in your kubernetes cluster.

Assuming auth is enabled, which is by default, you will have to add the following to the job which is configured to scrap kubernetes service endpoints.

tls_config:
    insecure_skip_verify: true

bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token