devLap

OrderHub Day 39: distributed tracing + Loki — one trace-id across four services, into every log line, 135 tests

java dev.to

Day 38 gave OrderHub a Grafana dashboard, so I can see the p99 climb. But a metric is an aggregate over ALL requests — it says what and when, never which hop. When one order crawls through four services (order → inventory → payment → shipping), its story is scattered across four log streams with nothing tying them together. Day 39 fixes that: give every request a trace-id, make each step a timed span, propagate that id across every HTTP and Kafka hop, and stamp it into every log line shipped to Loki.

Three jars, one feature gate

micrometer-tracing-bridge-brave adapts Micrometer's Tracer API onto Brave (OpenZipkin's tracer) — it creates spans and auto-instruments Feign/RestTemplate/WebClient/Kafka to propagate context. zipkin-reporter-brave exports finished spans so the waterfall can be assembled. loki-logback-appender ships JSON logs (with the traceId) to Loki. Being on the classpath does nothing yet — Boot's tracing auto-config is governed by management.tracing.enabled, which I bind straight to my own flag:

# application.yml
orderhub:
  observability:
    tracing:
      enabled: false          # the SINGLE feature gate (default OFF)
management:
  tracing:
    enabled: ${orderhub.observability.tracing.enabled:false}
    sampling: { probability: 1.0 }
    baggage:
      correlation:
        enabled: true
        fields: orderId       # copy trace ctx + orderId baggage into the MDC
Enter fullscreen mode Exit fullscreen mode

One property is the whole switch. Off by default, Brave/Zipkin auto-config never activates and the jars sit present-but-dark — which keeps all 131 prior tests byte-for-byte green.

Propagation is the whole trick

A trace stays distributed only if the id crosses process boundaries. Over HTTP, the instrumentation INJECTS the context as headers (traceparent / X-B3-TraceId) on the outbound Feign call; inventory EXTRACTS them and continues the same trace. Over Kafka, that context is written to record headers on publish and the @KafkaListener reads it on consume — so a hop consumed seconds later still attaches to the original trace, not a new one. I hand-code none of it; the bridge wraps all four clients automatically.

The MDC bridge to Loki

The link from traces to logs is the MDC — a per-thread bag of key/values that Micrometer populates with traceId/spanId. The console pattern reads them with %X{...}, and the :- default renders them empty (not an error) when tracing is off, so it's harmless with the gate disabled:

<!-- logback-spring.xml -->
%d %-5level [${appName:-},%X{traceId:-},%X{spanId:-},%X{orderId:-}] %logger{36} - %msg%n

<springProfile name="loki">          <!-- dark unless the loki profile is active -->
  <appender name="LOKI" class="com.github.loki4j.logback.Loki4jAppender">
    <http><url>${lokiUrl}</url></http>
    <format><message><pattern>
      {"level":"%level","traceId":"%X{traceId:-}","spanId":"%X{spanId:-}","msg":"%msg"}
    </pattern></message></format>
  </appender>
</springProfile>
Enter fullscreen mode Exit fullscreen mode

The Loki appender lives inside <springProfile name="loki">, so without that profile it's never instantiated — another reason every test stays green.

Closing the loop: logs ↔ traces

Loki is the log database — like Day-37's Prometheus, except Prometheus pulls metrics while Loki receives logs pushed by the appender. I add a grafana/loki:3.1.0 service to compose and provision a Grafana Loki datasource, same provisioning-as-code as Day 38. The payoff is the derivedField: a regex runs over each log line, pulls the traceId out, and turns it into a clickable link to the matching trace.

# grafana/provisioning/datasources/loki.yml
jsonData:
  derivedFields:
    - name: traceId
      matcherRegex: '"traceId":"(\w+)"'
      url: 'http://localhost:9411/zipkin/traces/${__value.raw}'   # -> the trace
Enter fullscreen mode Exit fullscreen mode

A log line jumps to its trace; a slow trace jumps back to its logs. All three pillars on one request.

Guard it with a test

Wiring drifts silently — delete the MDC fields and correlation dies at 3am, not in CI. A plain JUnit test proves both halves. An ApplicationContextRunner (a throwaway sliced context that never touches the shared @SpringBootTest one) loads TracingConfig with the flag on and asserts the orderId BaggageField bean exists — and with the flag absent asserts it does NOT. Then it reads logback-spring.xml and loki.yml off disk and asserts they reference the tracing fields and the derived field.

new ApplicationContextRunner()
  .withUserConfiguration(TracingConfig.class)
  .withPropertyValues("orderhub.observability.tracing.enabled=true")
  .run(ctx -> assertThat(ctx).hasSingleBean(BaggageField.class));
Enter fullscreen mode Exit fullscreen mode

The reactor goes 131 → 135 tests, BUILD SUCCESS.

The mental model is three questions, one id: metrics say something's wrong, traces say where, logs say why — all correlated by trace-id. Day 37 made OrderHub emit, Day 38 made it legible; Day 39 makes it navigable, from a red p99 to the exact slow span to the logs that explain it. Next, Day 40 tightens authorization with method-level security.

Live: https://dev48v.infy.uk/orderhub/day39-distributed-tracing-loki.html
Repo: https://github.com/dev48v/order-hub-from-zero

Source: dev.to

arrow_back Back to Tutorials