Getting Started with Grafana Tempo

Learn the fundamentals of Grafana Tempo, a distributed tracking backend.

Distributed monitoring is the way to get detailed information about system performance. It is a way to visualize the life cycle of a request that goes through the application. An application can consist of multiple services that are on a single node or distributed across nodes.

So by using distributed tracing, you can get an aggregate view of all services. And Grafana Tempo is about that.

What is Grafana Tempo?

There were some big updates from Grafana Labs at this year’s ObservabilityCon conference, and Grafana Tempo was one of them. Grafana Labs has added one more project, “Grafana Tempo”, to its open source portfolio.

Grafana Tempo is an open source, highly extensible and easy to use distributed monitoring backend. Tempo is fully compatible with other tracking protocols such as Zipkin, Jaeger, OpenTelemetry, and OpenCensus. It currently supports the Tempo data discovery engine in Loki, which monitors platforms like Prometheus and Grafana. Grafana 7.3+ offers a seamless experience between Grafana and Tempo.

Why use time?

Tempo is used to correlate metrics, traces, and logs. There are cases where users get the same type of error more than once. If I want to understand what’s going on, I’ll have to look at the exact traces. But due to downsampling, you lose valuable information you might be looking for. With Tempo, we now don’t need to sample distributed tracking data. We can store the entire trace on object storage like S3 or GCS, which makes Tempo very cost effective.

Additionally, Tempo allows you to debug/fix issues faster by allowing you to quickly jump from metrics to the relevant traces of the specific log that logged some issues.

Below are the setup options used in Tempo.

• Distributor: These are used to set the receive options to receive beats and then send them to the feeders.
• Ingester – Used to generate trace batches and send them to TempoDB for storage.
• Compressor: Streams blocks from storage like S3 or GCS, combines them, and writes them back to storage.
• Storage: This is used to configure TempoDB. You need to mention the storage backend name (S3 or GCS) with other parameters in this configuration.
• Member List: Used to coordinate between tempo components.
• Authentication/Server: Tempo uses Weaworks/Common Server. It is used to configure the server.

progress architecture

The above diagram shows the working architecture of Grafana Tempo.

First, the distributor receives legs in various formats from Zipkin, Jaeger, OpenTelemetry, OpenCensus and sends these legs to the gate person using the tracking ID hash. Ingester then creates batches of traces called blocks.

It then sends those blocks to back-end storage (S3/GCS). Once you have the tracking ID you want to troubleshoot, you’ll use the Grafana UI and place the tracking ID in the search bar. The queryer is now responsible for getting object or object storage details about the tracking ID you entered.

It first checks if that tracking ID is present in the digest; if it can’t find it, check the hosting server. It uses a simple HTTP endpoint to display the trace. Meanwhile, the compressor takes blocks from the file, combines them, and sends them back to the file to reduce the number of blocks in the file.

Configuring Tempo with Docker

In this section, I will configure Grafana Tempo step by step using Docker. You need a Tempo backend first, so set up a docker network.

[geekflare@geekflare ~]$ red docker create docker-tempo

Download the Tempo setup file.

[geekflare@geekflare ~]$ curl -o tempo.yaml

Here is a list of the protocol options you get:

Using the tempo configuration file, run the docker container. Here I choose the Jaeger – Thrift Compact (port 6831) format to send the trace.

[geekflare@geekflare ~]$ docker run -d –rm -p 6831:6831/udp –name tempo -v $(pwd)/tempo-local.yaml:/etc/tempo-local.yaml –network docker-tempo grafana/tempo:latest -config.file=/etc/tempo-local.yaml

Now you need to run the Tempo query container. First, download the tempo query configuration file.

[geekflare@geekflare ~]$ curl -o tempo-query.yaml

Using the tempo query configuration file, run the docker container.

[geekflare@geekflare ~]$ docker run -d –rm -p 16686:16686 -v $(pwd)/tempo-query.yaml:/etc/tempo-query.yaml –network docker-tempo grafana/tempo-query:latest –grpc-storage-plugin.configuration-file=/etc/tempo-query.yaml

The Jaeger user interface will now be accessible on As shown below.

In the search bar, you can add the trace id of the log you want to fix and it will create a trace for you.

Run demo app in Tempo

It’s time to run a demo example provided by Grafana Tempo. I’ll run a docker-compose example, so if you’re trying the same thing, you should have docker-compose installed on your machine.

Download the Grafana Tempo zip file:

Extract it to the home directory and the docker-compose directory. You’ll find plenty of docker-compose examples; I am using the example where the application data is stored locally.

[geekflare@geekflare ~]$ cd tempo-master/examples/docker-compose/
[geekflare@geekflare docker-compose]$ ls docker-compose.loki.yaml docker-compose.s3.minio.yaml docker-compose.yaml etc example-data readme.md tempo-link.png

Run the following command to start the stack.

[geekflare@geekflare docker-compose]$ docker-compose up -d Starting docker-compose_prometheus_1 … finished Starting docker-compose_tempo_1 … finished Starting docker-compose_grafana_1 … finished Starting docker-compose_tempo-query_1 … finished Starting docker-compose_synthetic-load-generator_1 … finished

As you can see, it started containers for Grafana, Loki, Tempo, Tempo-query, and Prometheus.

[geekflare@geekflare docker-compose]$ docker ps ID CONTAINER IMAGE COMMAND STATUS GENERAL PORT NAME STATUS 84cd557ce07b grafana/tempo-query:latest “/go/bin/query-linux…” 10 minutes ago Up 4 seconds 0.0.0.0:16686->16686/tcp docker-compose_tempo-0.25-starterator: sh” 10 minutes ago Up 4 seconds docker-compose_synthetic-load-generator_1 6d9d9fbdb8f1 grafana/grafana:7.3.0-beta1 “/run.sh” 10 minutes ago Up 6 seconds 0.0.0.0:3000->3000/tcp docker-compose_grafana_1 d8574ea25028 graf ana/0.0.cp, 0. 0.0.0:49172->14268/tcp docker-compose_tempo_1 5f9e53b5a09c prom/prometheus:latest “/bin/prometh eus –c…” 10 minutes ago Up to 6 seconds 0.0.0.0:9090->9090/tcp docker-compose_pro metheus_1

You can also go to your browser and check if Grafana, Jaeger UI, Prometheus are running.

Now the load add generator running inside the container is generating the id of the track you are putting inside the tempo. Run the following command and you will be able to see the log.

[geekflare@geekflare docker-compose]$ docker-compose logs -f synthesize-load-generator Attach docker-compose_synthetic-load-generator_1 synthesize-load-generator_1 | usage parameter: –jaegerCollectorUrl added-load-generator_1 | 04/21/17 14:24:34 INFO INFO ScheduledTraceGenerator: Start creating traces for service UI, routes/products, 2880 traces/hours added-load-generator_1 | 04/21/17 14:24:34 INFO ScheduledTraceGenerator: Start creating traces for service UI, route/cart, 14400 traces/hour aggregate-load-generator_1 | 04/21/17 14:24:34 INFO ScheduledTraceGenerator: Start creating traces for service UI, routes/payments, 480 traces/hours added-load-generator_1 | 04/21/17 14:24:37 INFO ScheduledTraceGenerator: TraceId 17867942c5e161f2 issued for service/currency UI route load-generator_1 | 04/21/17 14:24:37 INFO ScheduledTraceGenerator: TraceId 3d9cc23c8129439f Issued for service UI route synthetic-load-generator_1 send/service

These are the tracking IDs you need to pass in order to create a tracking.

I’m copying one of the tracking IDs and placing it in the Jaeger UI search bar.

You can see that it successfully created the traces relative to the trace id I provided.

Conclusion

This is Grafana Tempo. Go ahead and start using Tempo to create a trace to understand the metrics and issues in your logs in detail.

Everything is recorded in Tempo and no detail will be lost due to downsampling, which has happened before. Tempo helps the developer or production team to understand the root cause of possible errors or warnings in the registry in an easy way.

Let the article source
Getting Started with Grafana Tempo of website

Categories: How to