devLap

Monitoring Java Microservices with OpenTelemetry and OpenObserve

java dev.to

Monitoring microservices is hard.

When a user request fans out across multiple services, each with its own database, logs, and failure modes, traditional monitoring tools often give you a fragmented picture. You can tell something is slow, but not exactly where or why.

Distributed tracing solves this.

In this tutorial, we'll implement distributed tracing for a Java Spring Boot microservices application using two open-source tools: OpenTelemetry and OpenObserve.

If your stack includes other languages, check out these guides too:

What you'll build

By the end of this guide, you'll have:

  • A working Spring Boot microservices setup with cross-service HTTP calls
  • Zero-code instrumentation using the OpenTelemetry Java Agent
  • End-to-end traces in OpenObserve with flamegraph and Gantt chart views

What is distributed tracing?

In microservices, one user action can trigger a chain of calls across many services. If a request takes 3 seconds, tracing helps answer:

  • Which service caused the delay?
  • Which operation failed?
  • Where exactly time was spent?

Distributed tracing works by attaching context (trace_id, span_id) at request entry and propagating it across service boundaries (usually with traceparent headers). This gives you one complete request journey.

A trace is made up of spans. Each span records:

  • Service + operation
  • Start time + duration
  • HTTP details (method, URL, status)
  • DB query metadata
  • Errors/exceptions
  • Parent-child relationships

For deeper fundamentals: Distributed Tracing Basics to Beyond

Why OpenTelemetry + OpenObserve?

OpenTelemetry

OpenTelemetry is a CNCF standard for traces, metrics, and logs.

For Java, the OpenTelemetry Java Agent can auto-instrument Spring Boot, JDBC, and HTTP clients with no code changes.

OpenObserve

OpenObserve is an open-source backend for logs, metrics, and traces.

  • OTLP-native ingest
  • SQL-powered analytics
  • Unified observability in one interface
  • Lightweight and storage-efficient

Architecture used in this tutorial

We'll run four services:

Service Port Responsibility
discovery-service 8761 Eureka registry
user-service 8081 User CRUD (MySQL)
order-service 8082 Order management; calls user-service
payment-service 8083 Payment processing; calls order-service

The key trace path is:

payment-service -> order-service -> user-service -> MySQL

Prerequisites

  • Java 17+
  • Maven 3.8+
  • Docker + Docker Compose
  • MySQL 8 (or use Dockerized MySQL from compose)

Step 1: Clone the project 

git clone https://github.com/openobserve/java-distributed-tracing.git
cd java-distributed-tracing
Enter fullscreen mode Exit fullscreen mode

Step 2: Start OpenObserve and MySQL 

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

This starts:

  • OpenObserve: http://localhost:5080
  • MySQL: localhost:3306 (tracingdb)

Login to OpenObserve with:

  • Email: admin@example.com
  • Password: Admin123!

Step 3: Download OpenTelemetry Java Agent 

mkdir agents
curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar \
  -o agents/opentelemetry-javaagent.jar
Enter fullscreen mode Exit fullscreen mode

Step 4: Configure agent export to OpenObserve

Example from user-service/scripts/start.sh:

export OTEL_SERVICE_NAME=user-service
export OTEL_RESOURCE_ATTRIBUTES=service.name=user-service,deployment.environment=dev
export OTEL_TRACES_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=none
export OTEL_LOGS_EXPORTER=none
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:5080/api/default/traces
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="Authorization=Basic {token}"

java \
  -Xms256m \
  -Xmx512m \
  -javaagent:../agents/opentelemetry-javaagent.jar \
  -jar target/user-service-0.0.1-SNAPSHOT.jar
Enter fullscreen mode Exit fullscreen mode

Get {token} from OpenObserve UI:

Step 5: Start discovery-service 

cd discovery-service
mvn clean install -Dmaven.test.skip
sh scripts/start.sh
Enter fullscreen mode Exit fullscreen mode

Open: http://localhost:8761

Step 6: Start user/order/payment services

Run each in a separate terminal.

cd user-service
mvn clean install -Dmaven.test.skip
sh scripts/start.sh
Enter fullscreen mode Exit fullscreen mode 
cd order-service
mvn clean install -Dmaven.test.skip
sh scripts/start.sh
Enter fullscreen mode Exit fullscreen mode 
cd payment-service
mvn clean install -Dmaven.test.skip
sh scripts/start.sh
Enter fullscreen mode Exit fullscreen mode

Verify registration in Eureka:

Step 7: Generate traces

1) Create user 

curl -X POST http://localhost:8081/api/users \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Priya Sharma",
    "email": "priya@example.com",
    "phone": "+91-9876543210"
  }'
Enter fullscreen mode Exit fullscreen mode

2) Create order 

curl -X POST http://localhost:8082/api/orders \
  -H "Content-Type: application/json" \
  -d '{
    "userId": 1,
    "productName": "Mechanical Keyboard",
    "quantity": 1,
    "totalAmount": 4999.00
  }'
Enter fullscreen mode Exit fullscreen mode

3) Process payment (full distributed trace) 

curl -X POST http://localhost:8083/api/payments/process \
  -H "Content-Type: application/json" \
  -d '{
    "userId": 1,
    "orderId": 1,
    "amount": 4999.00,
    "currency": "INR",
    "paymentMethod": "UPI"
  }'
Enter fullscreen mode Exit fullscreen mode

4) Trigger an error trace 

curl -X POST http://localhost:8082/api/orders \
  -H "Content-Type: application/json" \
  -d '{
    "userId": 9999,
    "productName": "Test Product",
    "quantity": 1,
    "totalAmount": 100.00
  }'
Enter fullscreen mode Exit fullscreen mode

Expected: 400 Bad Request

Visualize in OpenObserve

Go to http://localhost:5080 -> Traces

Trace Explorer

You'll see:

  • Trace ID
  • Root span
  • Service
  • Duration
  • Span count
  • Status

Filter examples

  • service_name = payment-service
  • status = ERROR
  • Duration range
  • operation_name for specific endpoints

Flamegraph + Gantt chart

Click a POST /api/payments/process trace.

  • Flamegraph: nested span timing hierarchy
  • Gantt: timeline-aligned span bars

Query traces with SQL

OpenObserve supports SQL over trace data.

Slowest payment traces 

SELECT trace_id, duration, service_name, operation_name
FROM "default"
WHERE service_name = 'payment-service'
  AND operation_name LIKE '%payments/process%'
ORDER BY duration DESC
LIMIT 10;
Enter fullscreen mode Exit fullscreen mode

Error count by service 

SELECT service_name, COUNT(*) as error_count
FROM "default"
WHERE span_status = 'ERROR'
GROUP BY service_name
ORDER BY error_count DESC;
Enter fullscreen mode Exit fullscreen mode

Avg/max latency by service 

SELECT service_name,
       AVG(duration) as avg_duration_us,
       MAX(duration) as max_duration_us,
       COUNT(*) as request_count
FROM "default"
GROUP BY service_name;
Enter fullscreen mode Exit fullscreen mode

What the Java agent captured automatically

Without adding tracing code, the OpenTelemetry Java Agent instrumented:

  • Spring Web incoming HTTP requests
  • RestTemplate outbound calls (traceparent injected)
  • JDBC/MySQL queries
  • Context propagation across service boundaries

See supported libraries: OpenTelemetry Java Instrumentation

Final takeaway

You now have end-to-end distributed tracing for a Java microservices app with:

  • Zero-code instrumentation
  • Full request path visibility
  • Visual root-cause analysis (flamegraph/Gantt)
  • SQL-based troubleshooting in OpenObserve
  • A path to production scaling without vendor lock-in

Further reading

Source: dev.to

arrow_back Back to Tutorials