Using Prometheus for Monitoring

This tutorial is adapted from Web Age course Microservices Development Bootcamp with Immersive Project.

In this tutorial, you will use Prometheus to monitor a Spring Boot application. You will configure the Spring Boot application to enable metrics gathering via Spring Boot Actuators and expose Prometheus endpoint. Then, you will run Prometheus in a Docker container and configure it to monitor your Spring Boot application.

Part 1 – Setting the Stage

Before you start working on this tutorial:

  1. Ensure Docker is installed and configured.
  2. Run the following commands to pull Docker images required to use Prometheus 
    1. docker pull prom/prometheus:v2.10.0
    2. docker pull grafana/grafana:6.2.2

In this part, you will create a directory and copy an existing project to it.

1. Open a new Terminal window by clicking Applications > Terminal.

2. Create a directory where you will copy the Spring Boot application:

mkdir -p /home/wasadmin/TutorialWorks

3. Switch to the directory:

cd /home/wasadmin/TutorialWorks

4. Download the application from here. Unzip the existing application:

unzip /home/wasadmin/TutorialFiles/employees-rest-service

5. Switch to the project directory:

cd employees-rest-service

Part 2 – Explore an Existing Spring Boot Application

In this part, you will explore an existing Spring Boot application.

1. Build and run the sample project:

mvn clean install spring-boot:run

Wait until the application is started.

2. From Application menu, open Firefox web browser.

3. Navigate to the following URL:

http://localhost:8080/employees/

Note: Don’t forget to type in the / at the end.

The URL accesses Customers services implemented in the sample application. The service returns data in JSON format and it looks like this:

4. In the terminal where Spring Boot application is running, press Ctrl+C to stop the application.

5. Using gedit or vi/nano text editors, browse the code in the following files:

src/main/java/com/webage/rest/model/Employee.java
src/main/java/com/webage/rest/model/Employees.java
src/main/java/com/webage/rest/dao/EmployeeDAO.java
src/main/java/com/webage/rest/controller/EmployeeController.java

Note:

model folder contains the models (Employee and Employees).

dao folder contains Data Access Object which populates the data.

controller folder contains the REST service(s).

Part 3 – Add Prometheus Dependencies to the Project

In this part, you will modify pom.xml and add Prometheus dependencies to the project.

1. Open pom.xml:

gedit pom.xml

2. In the dependencies tag, add Actuator dependency:

<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency>

To enable Spring Boot Actuator, you add the spring-boot-actuator dependency. Actuator dependency enables various features, such as monitoring app, gathering metrics, understanding traffic or the state of database.

3. In the dependencies tag, add Prometheus dependency:

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

To integrate actuator with Prometheus, you need to add the micrometer-registry-prometheus dependency. Spring Boot uses Micrometer, an application metrics facade to integrate actuator metrics with external monitoring systems. It supports several monitoring systems like Netflix Atlas, AWS Cloudwatch, Datadog, InfluxData, SignalFx, Graphite, Wavefront, Prometheus etc.

4. Save and close pom.xml.

5. In the terminal, run the following command to start up your application:

mvn clean install spring-boot:run

6. In the web browser window, type the following URL:

http://localhost:8080/actuator

Notice the default endpoints exposed over the HTTP protocol look like this:

Prometheus endpoints are not exposed by default.

7. In the terminal, press Ctrl+C to stop your application.

Part 4 – Modify Spring Boot Application Properties

In this part, you will modify Spring Boot application properties to enable monitoring of various actuators.

1. Create a directory where you will store application properties:

mkdir src/main/resources

2. Create application properties file:

gedit src/main/resources/application.properties

3. Enter the following content in the editor:

management.security.enabled=false
management.metrics.export.prometheus.enabled: true
management.endpoints.web.exposure.include=*
management.endpoints.web.exposure.exclude=
management.endpoints.jmx.exposure.include=*
management.endpoints.jmx.exposure.exclude=

These lines enable Prometheus metrics that you can view over HTTP and also JMX.

4. Save the file and exit gedit.

5. Rebuild and run your application:

mvn clean install spring-boot:run

6. In the web browser, enter the following URL:

http://localhost:8080/actuator

Notice there’s a huge list of endpoints available.

7. Browse through the list and notice Prometheus is also available.

8. In the web browser, enter the following URL:

http://localhost:8080/actuator/prometheus

Notice there are various metrics available which can be monitored. You will do so later in this tutorial.

Part 5 – Create a Prometheus Configuration File

In this part, you will configure Prometheus to scrape metrics data from Spring Boot Actuator’s /prometheus endpoint.

1. Open a new Terminal.

2. Before you create Prometheus configuration file, option the host IP address. Run the following command and make a note of your host’s IP address:

ifconfig | grep netmask | grep broadcast 

3. Create a folder to save Prometheus configure file:

mkdir /home/wasadmin/TutorialWorks/prometheus

4. Use gedit to create a file named prometheus.yml:

gedit /home/wasadmin/TutorialWorks/prometheus/prometheus.yml

5. Enter the following contents to the file: (Note: Don’t forget to substitute the HOST_IP which made a note of in step 5. Also, ensure you indent the code properly)

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'spring-actuator'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
    - targets: ['HOST_IP:8080']

Note: These lines specify the following:

* Prometheus will run on port 9090

* It’s going to allow you to monitor Spring Boot actuator at /actuator/prometheus

* The application you want to run is located at HOST_IP:8080

6. Save the file and exit gedit.

Part 6 – Run Prometheus and Monitor your Application

In this part, you will run Prometheus. Prometheus is available in one of the docker images, which is preloaded on your machines.

1. Run the following command to start up Prometheus in a container: (Note: It’s a single statement)

sudo docker run -d --name=prometheus -p 9090:9090 -v /home/wasadmin/TutorialWorks/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml

2. If you are prompted for a password, enter wasadmin

3. Ensure there are no errors in the terminal, wait for a few seconds, and run the following command to verify Prometheus container has started:

sudo docker ps -a

4. It should show you Prometheus container like this:

5. In the web browser window, open a new tab, and enter the following URL:

http://localhost:9090

It should look like this:

6. In Expression textbox, enter the following:

process_uptime_seconds

Note: It’s one of the metrics which is exposed over /actuator/prometheus endpoint. It shows you how long the application has been running. You can navigate to http://localhost:8080/actuator/prometheus and search for it. There are also several other metrics which can be monitored.

7. Press Execute button

Notice a graph shows up under Graph tab. You can customize the time-period by pressing + and – buttons

8. Click Console tab to see the text version of the metrics.

9. Switch back to Graph tab.

10. In Expression text box, enter the following, and press Execute:

system_cpu_usage

11. It will show you CPU usage.

12. In Expression text box, enter the following, and press Execute:

http_server_requests_seconds_max{method="GET",status="200",uri="/employees/"}

13. It will show REST service response latency. It can be helpful to find slow APIs/REST services.

Note: If you don’ see any graph, open a tab in the web browser, and enter the URL: http://localhost:8080/employees/ and refresh the page a few times.

Part 7 – Clean-Up

In this part, you will stop Prometheus and your application.

1. Run the following command to stop Prometheus container:

sudo docker stop prometheus

If prompted for a password, enter wasadmin

2. Remove Prometheus container:

sudo docker rm prometheus 

If prompted for a password, enter wasadmin

3. Switch to the terminal where Spring Boot application is running and press Ctrl+C to stop the application.

4. Close web browser tabs where Prometheus is running.

5. Close the Terminal windows.

Part 8 – Review

In this tutorial, you used Prometheus to monitor a Spring Boot application.

Using Jaegar for Tracing

This tutorial is adapted from Web Age course Microservices Development Bootcamp with Immersive Project.

In this tutorial, you will implement instrumentation using Jaegar. You will modify an existing Spring Boot application to add tracing to it.

Part 1 – Setting the Stage

Before you start working on this tutorial:

  1. Ensure Docker is installed and configured.
  2. Run the following commands to pull Docker images required to use Jaeger 
    1. docker pull jaegertracing/all-in-one:1.12.0
    2. docker pull grafana/grafana:6.2.2

In this part, you will create a directory and copy an existing project to it.

1. Open a new Terminal window by clicking Applications > Terminal.

2. Create a directory where you will copy the Spring Boot application:

mkdir -p /home/wasadmin/TutorialWorks

3. Switch to the directory:

cd /home/wasadmin/TutorialWorks

4.  Download the application from here. Unzip the existing application:

unzip /home/wasadmin/TutorialFiles/employees-rest-service

5. If request to replace the files Press A and hit Enter to replace all the files.

6. Switch to the project directory:

cd employees-rest-service

Part 2 – Explore an Existing Spring Boot Application

In this part, you will explore an existing Spring Boot application.

1. Build and run the sample project:

mvn clean install spring-boot:run

2. From Application menu, open Firefox web browser.

3. Navigate to the following URL:

http://localhost:8080/employees/

Note: Don’t forget to type in the / at the end.

The URL accesses Customers services implemented in the sample application. The service returns data in JSON format and it looks like this:

4. In the terminal where Spring Boot application is running, press Ctrl+C to stop the application.

5. Using gedit or vi/nano text editors, browse the code in the following files:

src/main/java/com/webage/rest/model/Employee.java
src/main/java/com/webage/rest/model/Employees.java
src/main/java/com/webage/rest/dao/EmployeeDAO.java
src/main/java/com/webage/rest/conroller/EmployeeController.java

Note:

model folder contains the models (Employee and Employees)

dao folder contains Data Access Object which populates the data.

controller folder contains the REST service(s)

Part 3 – Add Jaegar Dependency and to the Project and Configure It

In this part, you will modify pom.xml and add Jaegar dependency to the project and configure a bean in the main application file which returns a tracing instance.

1. Open a Terminal window and switch to the project folder:

cd /home/wasadmin/TutorialWorks/employees-rest-service

2. Open the project’s main application file:

gedit src/main/java/com/webage/rest/SpringBootDemoApplication.java

3. After the existing import statements, add the following imports:

import io.jaegertracing.Configuration; import io.jaegertracing.internal.JaegerTracer; import org.springframework.context.annotation.Bean;

4. Inside SpringBootDemoAppliction class, after the main function, add the following code:

@Bean
public static JaegerTracer getTracer() {
   Configuration.SamplerConfiguration samplerConfig = Configuration.SamplerConfiguration.fromEnv().withType("const").withParam(1);
   Configuration.ReporterConfiguration reporterConfig = Configuration.ReporterConfiguration.fromEnv().withLogSpans(true);
   Configuration config = new Configuration("jaeger tutorial").withSampler(samplerConfig).withReporter(reporterConfig);
   return config.getTracer();
}

Note: These lines configure a bean which returns a JaegarTracing instance which will be used for tracing purpose. Also note, your application will traces will show up as “jaegar tutorial” in Jaeger. You can customize this name to whatever application name you want.

5. Save the file and exit gedit.

6. Open pom.xml:

gedit pom.xml

7. In the dependencies tag, add a new dependency:

	<dependency>
  	  <groupId>io.jaegertracing</groupId>
	  <artifactId>jaeger-client</artifactId>
	  <version>0.35.5</version>
	</dependency>

8. Save and close the file.

9. Run the following command to build your application:

mvn clean install

10. Ensure there are no errors. If there any, resolve them before proceeding to the next part of this tutorial.

Part 4 – Modify the Existing REST Service

In this part, you will modify the existing REST service/controller and write code to trace messages.

1. Open the REST service/controller file:

gedit src/main/java/com/webage/rest/controller/EmployeeController.java

2. After the existing import statements, add the following imports:

import io.opentracing.Span;
import io.opentracing.Tracer;

3. Inside EmployeeController class, add the following as the first statement:

@Autowired
private Tracer tracer;

4. Inside getEmployees method, delete the existing return statement, and add the following code:

Span span = tracer.buildSpan("get employees").start();
span.setTag("http.status_code", 201);
Employees data = employeeDao.getAllEmployees(); 
span.finish();
return data;

Note: Your find getEmployees method should look like this:

public Employees getEmployees()

{

Span span = tracer.buildSpan(“get employees”).start();

span.setTag(“http.status_code”, 201);

Employees data = employeeDao.getAllEmployees();

span.finish();

return data;

}

Note: These lines use the JaegarTracing object, which you configured in the main application file, to obtain a Span object. Span allows you to start the trace, optionally add some additional data in the form tags, and stop the trace

5. Save the file and close gedit.

6. Run the following command to build your application.

mvn clean install

7. Ensure there are no errors. If there any, resolve them before proceeding to the next part of this tutorial.

Part 5 – Verify Tracing in Jaegar

In this part, you will run your application, start jaegar, and verify tracing is working properly .

1. Run the following command to start up your Spring Boot application:

mvn  spring-boot:run

2. Open a web browser and navigate to the following URL:

http://localhost:8080/employees/

Note: It’s the same data you saw previously. Next, you will start Jaegar and view the traces.

3. Open a new terminal.

4. Run the following command to run Jaegar in a Docker container:

sudo docker run -d –name jaegar –rm -it –network=host jaegertracing/all-in-one:1.12.0

5. Enter wasadmin as password.

6. In the web browser window, open a new tab, and navigate to the following URL:

http://localhost:16686

7. On the Jaegar web console, click Service dropdown and select jaegar tutorial:

Note: If Service dropdown is blank, refresh the page in the tab where http://localhost:8080/employees/ page is open, and then refresh Jaegar web console.

Jaegar tutorial is your custom application. Jaegar query is available OOB as part of Jaegar. It allows you to make REST API calls to query Jaegar in case if you don’t want to use the web console.

8. Click Find Traces button:

Notice the trace looks like this:

9. Click the trace ‘jaegar tutorial‘ to view details.

It should show up like this:

10. In Service & Operation section, click jaegar tutorial again.

Notice it looks like this:

Notice Tags is showing your custom label http.status_code with value 201.

11. Go back to the Jaegar web console main page.

12. In the web browser tab where you have http://localhost:8080/employees/ page open, refresh the page.

13. Go back to Jaegar web console and click Find Traces.

14. Notice the traces show up like this:

 

Part 6 – Clean-Up

In this part, you will stop Prometheus and your application.

1. Run the following command to stop Jaegar container:

sudo docker stop jaegar

If prompted for a password, enter wasadmin

2. Switch to the terminal where Spring Boot application is running and press Ctrl+C to stop the application.

3. Close web browser tabs where Jaegar is running.

4. Close the Terminal windows.

Part 7 – Review

In this tutorial, you implemented instrumentation using Jaegar.

Create a Jenkins Pipeline

In this tutorial, you will explore the Pipeline functionality.

At the end of this tutorial, you will be able to:

  1. Create a simple pipeline

  2. Use a ‘Jenkinsfile’ in your project

  3. Use manual input steps in a pipeline

Part 1 – Create a Simple Pipeline

We can create a pipeline job that includes the pipeline script in the job configuration, or the pipeline script can be put into a ‘Jenkinsfile’ that’s checked-in to version control.

To get a taste of the pipeline, we’ll start off with a very simple pipeline defined in the job configuration.

Prerequisite: We need to have a project in source control to check-out and build. For this example, we’ll be using the ‘SimpleGreeting’ project that  cloned to a ‘Git’ repository at ‘C:\Software\repos\SimpleGreeting.git’.

For the above setup, refer to Part 2 of the blog Creating a Jenkins Job.

1. To connect to Jenkins, open Firefox and enter the following URL:

http://localhost:8080/

2. Login with your user/password.

3. Click on the New Item link.

4. Enter ‘SimpleGreetingPipeline‘ as the new item name, and select ‘Pipeline‘ as the item type.

5. When the input looks as above, click on OK to create the new item.

6. Scroll down to the Pipeline section and enter the following in the Script text window.

node {
    stage 'Checkout'
    git url: 'C:\\Software\\repos\\SimpleGreeting.git'
    
    stage 'Maven build'
    bat 'mvn install'
    
    stage 'Archive Test Results'
    step([$class: 'JUnitResultArchiver',
      testResults: '**/target/surefire-reports/TEST-*.xml'])
}

This pipeline is divided into three stages. First, we checkout the project from our ‘git’ repository. Then we use the ‘bat’ command to run ‘mvn install’ as a Windows batch file. Finally, we use the ‘step’ command to utilize a step from a standard Jenkins plugin – in this case, the JUnitResultArchiver, to save and display the results of the unit tests.

All of the above is wrapped inside the ‘node’ command, to indicate that we want to run these commands in the context of a workspace running on one of Jenkins execution agents (or the master node if no agents are available).

7. Click on Save to save the changes and return to the project page.

8. Click on Build Now to start up a pipeline instance.

9. After a few moments, you should see the Stage View appear, and successive stages will appear as the build proceeds, until all three stages are completed.

Part 2 – Pipeline Definition in a ‘Jenkinsfile’

For simple pipelines or experimentation, it’s convenient to define the pipeline script in the web interface. But one of the common themes of modern software development is “If it isn’t in version control, it didn’t happen”. The pipeline definition is no different, especially as you build more and more complex pipelines.

You can define the pipeline in a special file that is checked out from version control. There are several advantages to doing this. First, of course, is that the script is version-controlled. Second, we can edit the script with the editor or IDE of our choice before checking it in to version control. In addition, we can employ the same kind of “SCM Polling” that we would use in a more traditional Jenkins job.

In the following steps, we’ll create a Jenkinsfile and create a pipeline job that uses it.

1. Open the Eclipse editor. If this lab is completed in the normal sequence, you should have the ‘SimpleGreeting’ project already in Eclipse’s workspace. If not, check out the project from version control (consult your instructor for directions if necessary).

2. In the Project Explorer, right-click on the root node of the SimpleGreeting project, and then select New → File.

3. Enter ‘Jenkinsfile‘ as the file name.

4. Click Finish to create the new file.

5. Enter the following text into the new file (Note: this is the same script that we used above, so you could copy/paste it from the Jenkins Web UI if you want to avoid some typing):

node {
    stage 'Checkout'
    git url: 'C:\\Software\\repos\\SimpleGreeting.git'
    
    stage 'Maven build'
    bat 'mvn install'
    
    stage 'Archive Test Results'
    step([$class: 'JUnitResultArchiver',
      testResults: '**/target/surefire-reports/TEST-*.xml'])
}

6. Save the Jenkinsfile by selecting File → Save from the main menu, or by hitting Ctrl-S.

7. In the Project Explorer, right-click on the SimpleGreeting node, and then select Team → Commit…

8. Eclipse will display the Commit Changes dialog. Click the check box next to Jenkinsfile (to include that file in the commit) and enter a commit message.

9. Click Commit and Push, and then click OK to dismiss the status dialog.

Now we have a Jenkinsfile in our project, to define the pipeline. Next, we need to create a Jenkins job to use that pipeline.

10. In the Jenkins user interface, navigate to the root page, and then click on New Item.

11. Enter ‘SimpleGreetingPipelineFromGit‘ as the name of the new item, and select Pipeline as the item type.

12. Click OK to create the new item.

13. Scroll down to the Build Triggers section.

14. Click on Poll SCM and enter ‘* * * * *’ as the polling schedule. This entry will cause Jenkins to poll once per minute.

15. Scroll down to the Pipeline section, and change the Definition entry to ‘Pipeline Script from SCM’

16. Enter the following:

SCM: Git

Repository URL: C:\Software\repos\SimpleGreeting.git

(Press the tab key)

17. The Pipeline section should look similar to:

18. Click Save to save the new configuration.

19. Click Build Now to launch the pipeline.

20. You should see the pipeline execute, similar to the previous section.

Part 3 – Try out a Failing Build

The pipeline that we’ve defined so far appears to work perfectly. But we haven’t tested it with a build that fails. In the following steps, we’ll insert a test failure and see what happens to our pipeline.

1. In Eclipse, go to the Project Explorer and locate the file ‘Greeting.java’. It will be under src/main/java in the package ‘com.simple’.

2. Open ‘Greeting.java’.

3. Locate the line that reads ‘return “GOOD”;’. Change it to read ‘return “BAD”;’

4. Save the file.

5. In the Project Explorer, right-click on Greeting.java and then select Team → Commit… (This is a shortcut for committing a single file).

6. Enter an appropriate commit message and then click Commit and Push.

7. Click OK in the results box, to close it.

8. Switch back to Jenkins.

9. In a minute or so, you should see a build launched automatically. Jenkins has picked up the change in the ‘Git’ repository and initiated a build. If nothing happens then click Build Now.

This time, the results are a little different. The ‘Maven Build’ stage is showing a failure, and the ‘Archive Test Results’ stage was never executed.

What’s happened is that the unit tests have failed, and Maven exited with a non-zero result code because of the failure. As a result, the rest of the pipeline was canceled. This behavior probably isn’t what you want or what you expect in most cases. We’d like to go ahead and archive the test results, even when there’s a failure. That way, we can see the trend including failed tests.

The solution here is to add a command-line parameter to the Maven invocation. If we add ‘-Dmaven.test.failure.ignore’ to the Maven command line, then Maven will continue with the build even if the tests fail.

10. Go back to Eclipse and open the ‘Jenkinsfile‘ if necessary.

11. Alter the ‘bat “mvn…” line to read as follows:

   bat 'mvn -Dmaven.test.failure.ignore install'

12. Save the ‘Jenkinsfile’. Commit and push the changes using the same technique as above.

13. After a minute or so, you should see a new Pipeline instance launched. If nothing happens then click Build Now.

This time, the pipeline runs to completion, and the test results are archived as expected. Notice that the build is now flagged as ‘unstable’ (indicated by the yellow color and the icon). The JUnit archiver noticed the failures and flagged the build unstable, even though Maven exited normally.

Part 4 – Add a Manual Approval Step

One of the interesting features of the Pipeline functionality is that we can include manual steps. This is very useful when we’re implementing a continuous deployment pipeline. For example, we can include a manual approval step (or any other data collection) for cases like ‘User Acceptance Testing’ that might not be fully automated.

In the steps below, we’ll add a manual step before a simulated deployment.

1. Go to ‘Eclipse’ and open the ‘Jenkinsfile‘ if necessary.

2. Add the following to the end of the file after the closing }:

stage 'User Acceptance Test'
def response= input message: 'Is this build good to go?',
 parameters: [choice(choices: 'Yes\nNo', 
 description: '', name: 'Pass')]
if(response=="Yes") {
 node {
  stage 'Deploy'
  bat 'mvn -Dmaven.test.failure.ignore install'
 }
}	

This portion of the script creates a new stage called ‘User Acceptance Test’, then executes an ‘input’ operation to gather input from the user. If the result is ‘Yes’, the script executes a deploy operation in a new ‘node’ step. (In this case, we’re repeating the ‘mvn install’ that we did previously. Only because we don’t actually have a deployment repository setup)

3. Save and commit ‘Jenkinsfile’ as previously.

When the pipeline executes, watch for a “paused” stage called ‘User Acceptance Test”. If you move your mouse over this step, you’ll be able to select “Yes” or “No”.

4. Move your mouse over the ‘User Acceptance Test’ step, select Yes and click Proceed.

You should see the job run to completion.

5. Run the pipeline again (click Build Now), but this time, select No on the ‘User Acceptance Test’ and click Proceed, you’ll see that the pipeline exits early and doesn’t run the ‘deploy’ stage.

What’s happened is that the final ‘Deploy’ stage was only executed when we indicated that the ‘User Acceptance Test’ had passed.

6. Close all.

Part 5 – Review

In this tutorial, we explored the Pipeline functionality in Jenkins. We built a simple pipeline in the Jenkins web UI, and then used a ‘Jenkinsfile’ in the project. Lastly, we explored how to gather user input, and then take different build actions based on that user input.