Introduction
In this document, variables primed for substitution before execution are highlighted as such:
For example:
becomes:
In this document commands to execute in a shell are shown as code and each block of code is designed to be a single command that can be copy and pasted:
Overview
This document covers the deployment of various monitoring services which allow a Kubernetes Administrator to monitor the health, metrics, and logs for all cluster services including Reach.
List of services to be deployed:
Prometheus Stack (health, metrics)
- Grafana
- Prometheus
- Alertmanager
ELK Stack (logging)
- Elasticsearch
- Kibana
- Filebeat
Set Up the Deployment Shell
Export the following environment variables which will be used throughout the installation:
Export the following if Grafana will be configured to use AAD for authentication:
Create a new namespace:
The next command below uses the pwgen package to generate a random string of 30 alphanumeric characters.
Before proceeding make sure pwgen is installed on your machine or use a different package to generate the string replacing the command inside the brackets:
Storage
Express
If you only want to modify the storage class and leave all other parameters such as size as default, export these variables out:
Custom Parameters
Here is a list of different monitoring services and how to customize their storage.
Elasticsearch
Please refer to the <inline-code>volumeClaimTemplate<inline-code>: section found in the values.yaml file in the elasticsearch helm chart repository for a list of available parameters to customize such as size, access modes and so on.
These values can be added/tweaked in the following files:
- k8s/misc/elk/elasticsearch/values-prod.yaml
- k8s/misc/elk/elasticsearch/values-common.yaml
Prometheus Stack
Please refer to the <inline-code>volumeClaimTemplate<inline-code>: sections found in the values.yaml file in the prometheus-stack helm chart repository for a list of available parameters to customize such as size, access modes and so on.
These values can be added/tweaked in the following files:
- k8s/misc/elk/prometheus/values-prod.yaml
- k8s/misc/elk/prometheus/values-common.yaml
Monitoring TLS
Manually create a TLS secret from a TLS key and cert or use the LetsEncrypt integration with cert-manager.
Manually Creating a TLS Cert Secret
LetsEncrypt with Cert-manager
Export the following:
Create Secrets
Create a secret which will store Grafana credentials:
Installing Grafana, Alertmanager, and Prometheus
Add these repos to Helm and update:
If you have chosen to configure Grafana to use AAD for authentication, follow thisguide to register an app in AAD https://grafana.com/docs/grafana/latest/auth/azuread/#create-the-azure-ad-application and then create a secret by filling out it with the information of the app youhave created.
Export the following:
Install:
Check the status of deployed pods:
Scraping Virtalis Reach Services Using Prometheus
Export the following:
Run the upgrade:
Accessing the Grafana frontend
Retrieve the Grafana admin user:
Retrieve the Grafana admin password:
Grafana can now be accessed at https://${MONITORING_DOMAIN}/grafana/ from a web-browser using the admin user and admin password
Installing Elasticsearch, Kibana, Filebeat
Add this helm repo and update:
Export this variable:
Install Elasticsearch:
Install Kibana:
Patch Kibana and Elasticsearch:
Get the elasticsearch admin password:
Open up kibana in a web browser, log in using the elasticsearch admin password and the username “elastic” and add any additional underprivileged users that you want to have access to the logging system:
Install Filebeat:
Optionally, configure filebeat log index to tweak the data retention period and index max size.
You can read more about ILM here.
Clean-up Post Monitoring Installation
Unset environment variables:
Clear bash history:
This will clean up any secrets exported in the system.
