kilo/docs/monitoring.md
leonnicolas bb3554a3c6
Apply suggestions from code review
Co-authored-by: Lucas Servén Marín <lserven@gmail.com>
2021-10-20 22:25:05 +02:00

101 lines
3.6 KiB
Markdown

# Monitoring
The following assumes that you have applied the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) monitoring stack onto your cluster.
## Kilo
Monitor the Kilo DaemonSet with:
```shell
kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/podmonitor.yaml
```
## WireGuard
Monitor the WireGuard interfaces with:
```shell
kubectl create ns kilo
kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/wg-exporter.yaml
```
The manifest will deploy the [Prometheus WireGuard Exporter](https://github.com/MindFlavor/prometheus_wireguard_exporter) as a DaemonSet and a [PodMonitor](https://docs.openshift.com/container-platform/4.8/rest_api/monitoring_apis/podmonitor-monitoring-coreos-com-v1.html).
By default the kube-prometheus stack only monitors the `default`, `kube-system` and `monitoring` namespaces.
In order to allow Prometheus to monitor the `kilo` namespace, apply the Role and RoleBinding with:
```shell
kubectl apply -f https://raw.githubusercontent.com/squat/kilo/main/manifests/wg-exporter-role-kube-prometheus.yaml
```
## Metrics
### Kilo
Kilo exports some standard metrics with the Prometheus GoCollector and ProcessCollector.
It also exposes some Kilo-specific metrics.
```
# HELP kilo_errors_total Number of errors that occurred while administering the mesh.
# TYPE kilo_errors_total counter
# HELP kilo_leader Leadership status of the node.
# TYPE kilo_leader gauge
# HELP kilo_nodes Number of nodes in the mesh.
# TYPE kilo_nodes gauge
# HELP kilo_peers Number of peers in the mesh.
# TYPE kilo_peers gauge
# HELP kilo_reconciles_total Number of reconciliation attempts.
# TYPE kilo_reconciles_total counter
```
### WireGuard
The [Prometheus WireGuard Exporter](https://github.com/MindFlavor/prometheus_wireguard_exporter) exports the following metrics:
```
# HELP wireguard_sent_bytes_total Bytes sent to the peer
# TYPE wireguard_sent_bytes_total counter
# HELP wireguard_received_bytes_total Bytes received from the peer
# TYPE wireguard_received_bytes_total counter
# HELP wireguard_latest_handshake_seconds Seconds from the last handshake
# TYPE wireguard_latest_handshake_seconds gauge
```
## Display some Metrics
If your laptop is a Kilo peer of the cluster you can access the Prometheus UI by navigating your browser directly to the cluster IP of the `prometheus-k8s` service.
Otherwise use `port-forward`:
```shell
kubectl -n monitoring port-forward svc/prometheus-k8s 9090
```
and navigate your browser to `localhost:9090`.
Check if you can see the PodMonitors for Kilo and the WireGuard Exporter under **Status** -> **Targets** in the Prometheus web UI.
If you don't see them, check the logs of the `prometheus-k8s` Pods; it may be that Prometheus doesn't have the permission to get Pods in the `kilo` namespace.
In this case, you need to apply the Role and RoleBinding from above.
Navigate to **Graph** and try to execute a simple query, e.g. type `kilo_nodes` and click on `execute`.
You should see some data.
## Using Grafana
Let's navigate to the Grafana dashboard.
Again, if your laptop is not a Kilo peer, use `port-forward`:
```shell
kubectl -n monitoring port-forward svc/grafana 3000
```
Now navigate your browser to `localhost:3000`.
The default user and password is `admin` `admin`.
An example configuration for a dashboard displaying Kilo metrics can be found [here](https://raw.githubusercontent.com/squat/kilo/main/docs/grafana/kilo.json).
You can import this dashboard by hitting **+** -> **Import** on the Grafana dashboard.
The dashboard looks like this:
<img src="./graphs/kilo.png" />