Grafana

LegGrafana

Check the link Grafana setup to setup grafana

Once its up you can run it locally http://localhost:3000/

Go to add data source and add prometheus as a data source as follows

Once data source is added create a dashboard with the required metrics .

Add an alert 

You can use the templating feature in grafana to provide an option to filter based on certain criteria.

Click on settings -> templating -> New 

A form like the one below would appear. 

1. Choose the name data source label_values(label) - Returns a list of values for the label in every metric.
2. Regex. helps filter based on the expression.
3. Multivalued to choose all option.
4. Preview shows a set of outputs. 

This helps filter results in the result graph based on the selection. The variables can be used in the query in graphs. 

Prometheus is very useful in the following scenarios

1. Track API wise failures. Current day yesterday, 7 days before so on.
1. Use offset 1d to get results for the previous day. 
2. Sample Query : sum by (job,__name__)(irate({__name__=~"$apierrors", job=~"$service", instance=~"$instance"} [2m] offset 1d)) * 60.
1. Here The irate get the per second rate *60 converts it to mins. 
2. Offset 1d fetches results 1 day before.
3. $service $instance are variables in the templating.
2. Grafana also provides an option to display results in a table. 
3. Alerting - In case a particular threshold is exceeded grafana allows configuration of alerts. You need to first setup a notification channel say slack or email and then setup alerts by specifiying the query and the notification channel.

   

   Select the query condition you can add multiple conditions using and or operators

   
   

   

   Add slack channel or email notification created.

   
   

   

   Create notification channel here