Monitoring Spring boot application using Prometheus and Grafana

In continuation with previous article ,we will see how we can monitor spring boot application developed in earlier post with actuator. We have already seen prometheus tool for monitoring application but as prometheus doesn’t provide better visualization experience, we have to go with other tools which has rich UI for better visualization of application metrics.

You can use Grafana for the better visualization .

Grafana

While Prometheus does provide some crude visualization, Grafana offers a rich UI where you can build up custom graphs quickly and create a dashboard out of many graphs in no time. You can also import many community built dashboards for free and get going.

Grafana can pull data from various data sources like Prometheus, Elasticsearch, InfluxDB, etc. It also allows you to set rule-based alerts, which then can notify you over Slack, Email, Hipchat, and similar.

Let’s start with running Grafana using Doc

You can download grafana for windows using below link –

https://grafana.com/grafana/dashboards

If you visit http://localhost:3000, you will be redirected to a login page:

Login page

Default credentials to login are admin/admin .

Since Grafana works with many data sources, we need to define which one we’re relying on. In our case we have to select Prometheus as our data source:

Datasource configuration

Select Prometheus from below provided options –

Now, add the URL that Prometheus is running on, in our case http://localhost:9090 and select Access to be through a browser.

At this point, we can save and test to see if the data source is working correctly:

You can search for JVM (Micrometer) dashboard on grafana website and provide its ID in next step to import that dashboard into grafana –

You can find the dashboard ID as highlighted in below screenshot –

As i mentioned earlier, Grafana has lots of of pre-built dashboards. For Spring Boot projects, the JVM dashboard is popular. We are going to use JVM micrometer (ID = 4701) dashboard in this example .

Click on highlighted plus icon and then import –

Select the Prometheus datasource name created in earlier steps as highlighted in below screenshot-

Once you click on import , your dashboard is ready . You can have a look at all the metrics exposed by prometheus on a single dashboard –

I hope this tutorial is helpful for you to configure your spring boot application with Grafana

In this article, we used Micrometer to reformat the metrics data provided by Spring Boot Actuator and expose it in a new endpoint. This data was then regularly pulled and stored by Prometheus, which is a time-series database. Ultimately, we’ve used Grafana to visualize this information with a user-friendly dashboard.

Monitoring an application’s health and metrics helps us manage it better, notice unoptimized behavior, and better understand its performance. This especially holds true when we’re developing a system with many microservices, where monitoring each service can prove to be crucial when it comes to maintaining our system.

Based on this information, we can draw conclusions and decide which microservice needs to scale if further performance improvements can’t be achieved with the current setup.

Github Downlod Link:

Spring mvc integration with prometheus Example

In this article ,we will see how we can monitor spring MVC applications using prometheus tool . You wont get much documentation for this setup on web. Prometheus is mostly configured with spring boot easily. But in order to configure it with spring MVC based application , it requires some additional configurations –

Prometheus

Prometheus is a time-series monitoring application written in Go. It can run on a server, in a docker container, or as part of a Kubernetes cluster (or something similar). Prometheus collects, stores, and visualizes time-series data so that you can monitor your systems. You can tell Prometheus exactly where to find metrics by configuring a list of “scrape jobs”. Applications that are being monitored can provide metrics endpoints to Prometheus using any one of the many client libraries available; additionally, separate exporters can gather metrics from applications to make them available in Prometheus. Metrics get stored locally for 15 days, by default, and any Prometheus server can scrape another one for data. Additionally, remote storage is another option for Prometheus data – provided there is a reliable remote storage endpoint.

Benefits:

  • The option of “service discovery” allows Prometheus to keep track of all current endpoints effortlessly.
  • Outages are quickly detected .
  • The PromQL query language is incredibly flexible and Turing-complete.
  • There’s also a very low load on the services monitored (metrics get stored in memory as they get generated), allowing fewer resources to get used.
  • Additionally, Prometheus users can control traffic volumes, access metrics in the browser, and allow for easy reconfiguration.

Step 1 : Spring MVC application pom.xml configuration

Below prometheus dependencies are required in pom.xml for project –

<prometheus.version>0.6.0</prometheus.version>
<dependency>
    <groupid>io.prometheus</groupid>
    <artifactid>simpleclient</artifactid>
    <version>${prometheus.version}</version>
</dependency>
<!-- Hotspot JVM metrics-->
<dependency>
    <groupid>io.prometheus</groupid>
    <artifactid>simpleclient_hotspot</artifactid>
    <version>${prometheus.version}</version>
</dependency>
<!-- Exposition servlet-->
<dependency>
    <groupid>io.prometheus</groupid>
    <artifactid>simpleclient_servlet</artifactid>
    <version>${prometheus.version}</version>
</dependency>
<!-- Pushgateway exposition-->
<dependency>
    <groupid>io.prometheus</groupid>
    <artifactid>simpleclient_pushgateway</artifactid>
    <version>${prometheus.version}</version>
</dependency>
<dependency>
    <groupid>io.prometheus</groupid>
    <artifactid>simpleclient_spring_web</artifactid>
    <version>${prometheus.version}</version>
</dependency>
<dependency>
    <groupid>com.fasterxml.jackson.core</groupid>
    <artifactid>jackson-core</artifactid>
    <version>2.5.2</version>
</dependency>

Step 2 : Spring MVC application web.xml configuration

We need configure MetricsServlet to capture the metrics of our spring mvc application as below –

<servlet>
    <servlet-name>PrometheusServlet</servlet-name>
    <servlet-class>io.prometheus.client.exporter.MetricsServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>PrometheusServlet</servlet-name>
    <url-pattern>/metrics</url-pattern>
</servlet-mapping>

Step 3: Add an interceptor class

This will intercept all the requests coming to application and capture the metrics to be exposed to prometheus –

package com.myjavablog.config;

import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;
import io.prometheus.client.Histogram;
import io.prometheus.client.Summary;
import org.apache.log4j.Logger;
import org.springframework.web.method.HandlerMethod;
import org.springframework.web.servlet.ModelAndView;
import org.springframework.web.servlet.handler.HandlerInterceptorAdapter;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;


/**
 * @author anupb
 */

public class PrometheusMetricsInterceptor extends HandlerInterceptorAdapter {

    private static Logger logger = Logger.getLogger(PrometheusMetricsInterceptor.class);

    private static final Histogram requestLatency = Histogram.build()

            .name("service_requests_latency_seconds")

            .help("Request latency in seconds.")

            .labelNames("systemId", "appId", "type", "name", "method").register();


    private ThreadLocal<Histogram.Timer> timerThreadLocal;


    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {

        return super.preHandle(request, response, handler);

    }

    @Override
    public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {

        String name = this.getName(request, handler).toLowerCase();
        String method = request.getMethod().toUpperCase();

        timerThreadLocal = new ThreadLocal<>();
        timerThreadLocal.set(requestLatency.labels(name, method).startTimer());
        super.postHandle(request, response, handler, modelAndView);
    }

    @Override
    public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {

        super.afterCompletion(request, response, handler, ex);

        if (timerThreadLocal.get() != null) {
            timerThreadLocal.get().observeDuration();
        }
    }

    @Override
    public void afterConcurrentHandlingStarted(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {

        super.afterConcurrentHandlingStarted(request, response, handler);
    }

    private String getName(HttpServletRequest request, Object handler) {
        String name = "";

        try {
            if (handler != null && handler instanceof HandlerMethod) {

                HandlerMethod method = (HandlerMethod) handler;
                String className = ((HandlerMethod) handler).getBeanType().getName();
                name = className + "." + method.getMethod().getName();
            } else {
                name = request.getRequestURI();
            }

        } catch (Exception ex) {
            logger.error("getName", ex);
        } finally {
            return name;
        }
    }
}

Step 4: Add prometheus initialization configuration

This will expose the metrics to prometheus server –

package com.myjavablog.config;
 public class PrometheusConfig {

private static Logger logger = Logger.getLogger(PrometheusConfig.class);

 

@PostConstruct

public void initialize() {

logger.info("prometheus init...");

DefaultExports.initialize();

logger.info("prometheus has been initialized...");

}

}

Step 5: Add an interceptor to spring-mvc.xml

You need to first add the schema location as below –

http://www.springframework.org/schema/mvc

http://www.springframework.org/schema/mvc/spring-mvc-3.0.xsd

Then you need to add below tag –

<mvc:interceptors>

<bean class="com.xx.config.PrometheusMetricsInterceptor"/>

</mvc:interceptors>

Step 6: Add configuration to applicationcontext.xml

<bean id="prometheusConfig" class="com.myjavablog.config.PrometheusConfig" init-method="initialize" />

Once all this configuration is done , you can add the application URL in prometheus.

These parameters are useful to monitor your spring MVC application.

Monitoring Spring boot applications using actuator and prometheus

In this article ,we will see how we can monitor spring boot applications using actuator spring boot project and prometheus tool .

Prometheus

Prometheus is a time-series monitoring application written in Go. It can run on a server, in a docker container, or as part of a Kubernetes cluster (or something similar). Prometheus collects, stores, and visualizes time-series data so that you can monitor your systems. You can tell Prometheus exactly where to find metrics by configuring a list of “scrape jobs”. Applications that are being monitored can provide metrics endpoints to Prometheus using any one of the many client libraries available; additionally, separate exporters can gather metrics from applications to make them available in Prometheus. Metrics get stored locally for 15 days, by default, and any Prometheus server can scrape another one for data. Additionally, remote storage is another option for Prometheus data – provided there is a reliable remote storage endpoint.

Benefits:

  • The option of “service discovery” allows Prometheus to keep track of all current endpoints effortlessly.
  • Outages are quickly detected .
  • The PromQL query language is incredibly flexible and Turing-complete.
  • There’s also a very low load on the services monitored (metrics get stored in memory as they get generated), allowing fewer resources to get used.
  • Additionally, Prometheus users can control traffic volumes, access metrics in the browser, and allow for easy reconfiguration.

Part 1: Spring boot application configuration

We will create a simple spring boot REST application which will expose the metrics to prometheus. Please find below project structure of application –

Project Structure

Below is the pom.xml required for project –

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.db</groupId>
    <artifactId>spring-boot-prometheus</artifactId>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.9.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-prometheus</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

We need spring-boot-starter-actuator dependency ,this would expose the endpoints of our application and makes it available to prometheus server. You can run the application and check the endpoints exposed by actuator by hitting below link in browser –

http://localhost:8081/actuator

Micrometer

micrometer-registry-prometheus dependency is required to register metrics with prometheus server. It exposes Actuator metrics to external monitoring systems such as Prometheus, Netflix Atlas, AWS Cloudwatch, and many more.

Similarly, Micrometer automatically exposes /actuator/metrics data into something your monitoring system can understand. All you need to do is include that vendor-specific micrometer dependency in your application.

Micrometer is a separate open-sourced project and is not in the Spring ecosystem, so we have to explicitly add it as a dependency. Since we will be using Prometheus, let’s add it’s specific dependency in our pom.xml:

Part 2: Prometheus Server Configuration

For this example , i will be installing portable prometheus on local system . You can download the prometheus setup from official prometheus website as below –

https://prometheus.io/download/

You need to make below changes in prometheus.yml file of prometheus configuration to scrape the metrices your application .Please add below job to scrape the metrices of your application –

 # Details to connect Prometheus with Spring Boot actuator end point to scrap the data
  # The job name is added as a label `job=` to any time series scraped from this config.
  - job_name: 'spring-actuator'
   
    # Actuator end point to collect the data. 
    metrics_path: '/actuator/prometheus'

    #How frequently to scape the data from the end point
    scrape_interval: 5s

    #target end point. We are using the Docker, so local host will not work. You can change it with
    #localhost if not using the Docker.
    static_configs:
    - targets: ['192.168.43.33:8081']

Run the prometheus server by double clicking prometheus.exe flie .

Now prometheus console will open and it will strat gathering the metrices of your application. You can access the prometheus server in browser using http://localhost:9090

Targets for metrics
Classes loaded in JVM prometheus metrics
Number of Threads in different states

You can gather the metrices of your application by selecting the various parameters provided by prometheus like –

  • jvm_classes_loaded_classes
  • jvm_threads_live_threads
  • jvm_threads_states_threads
  • jvm_threads_states_threads
  • tomcat_global_request_seconds_count

These parameters are useful to monitor your systems. In the next article, we will create a prometheus server using docker and expose the metrices.

Github Downlod Link:

Bitnami