A Hands-on Guide to OpenTelemetry - Better Tracing with Automatic Instrumentation

 Are you ready to start your journey on the road to collecting telemetry data from your applications? Great observability begins with great instrumentation! 

In this series you'll explore how to adopt OpenTelemetry (OTel) and how to instrument an application to collect tracing telemetry. You'll learn how to leverage out-of-the-box automatic instrumentation tools and understand when it's necessary to explore more advanced manual instrumentation for your applications. By the end of this series you'll have an understanding of how telemetry travels from your applications, to the OpenTelemetry Collector, and be ready to bring OpenTelemetry to your future projects. Everything discussed here is supported by a hands-on, self-paced workshop authored by Paige Cruz. 

The previous article we took our first steps in generating telemetry data using automatic instrumentation. In this article, we explore how to gain better insights by adding manual instrumentation to our application leveraging the existing auto-instrumentation.

It is assumed that you followed the previous articles in setting up both OpenTelemetry and the example Python application project, but if not, go back and see the previous articles as it's not covered here.

We saw that automatic instrumentation was very broadly scoped with our Python application, generating things like trace_id, span_id, telemetry.sdk.name, and more. To get a deeper level of insight into our application we need to manually create new spans or modify existing ones.

Manually instrumenting application

With our auto-instrumentation set up previously, one of the libraries opentelemetry-bootstrap installed was opentelemetry-instrumentation-flask. This library instruments web requests to the Flask application, so let's modify that a bit by adding the number of times the homepage has been loaded as an attribute on the auto-instrumented span.

Below is the current telemetry data output from loading the homepage, showing a single span from our console log with auto-instrumentation generating default attributes such as http.method, http.host, and more:

10.88.0.19 - - [11/Jul/2024 10:37:20] "GET / HTTP/1.1" 200 -
{
    "name": "GET /",
    "context": {
        "trace_id": "0xdd1f48a2740d5c133bd288333e99b176",
        "span_id": "0xa27a71b94664503f",
        "trace_state": "[]"
    },
    "kind": "SpanKind.SERVER",
    "parent_id": null,
    "start_time": "2024-07-11T10:37:20.904983Z",
    "end_time": "2024-07-11T10:37:20.917104Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "http.method": "GET",
        "http.server_name": "0.0.0.0",
        "http.scheme": "http",
        "net.host.name": "localhost:8001",
        "http.host": "localhost:8001",
        "net.host.port": 8000,
        "http.target": "/",
        "net.peer.ip": "10.88.0.19",
        "net.peer.port": 37982,
        "http.user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...",
        "http.flavor": "1.1",
        "http.route": "/",
        "hits": 1,
        "http.status_code": 200
    },
    "events": [],
    "links": [],
    "resource": {
        "attributes": {
            "telemetry.sdk.language": "python",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.version": "1.25.0",
            "service.name": "hello-otel",
            "telemetry.auto.version": "0.46b0"
        },
        "schema_url": ""
    }
}

Let's refine our insights by adding some telemetry data collection using the OpenTelemetry SDK which was installed by the call to opentelemetry-distro. We access this by importing the trace module as show below as found in the project file automatic/app.py:

import random
import re
import urllib3

import requests
from opentelemetry import trace

from flask import Flask, render_template, request
...

This gives us access to the auto-instrumented spans through the tracer, which has been created by our agent wrapper opentelemetry-instrument.  We just access it programmatically as shown below a little farther down in the project file:

import random
import re
import urllib3

import requests

from opentelemetry import trace

from flask import Flask, render_template, request

from breeds import breeds

app = Flask(__name__)

tracer = trace.get_tracer(app.name)
...

Through the auto-instrumentation our Flask routes are all instrumented, so we want to access those through existing spans and not create a new one. We continue down in the file where an attribute named hits in the method named index() under the homepage route is added to track page loads as shown below:

@app.route('/')
def index():

    span = trace.get_current_span()
    global HITS
    HITS = HITS + 1

    span.set_attribute("hits", HITS)
    msg = f'This webpage has been viewed {HITS} times'
    return msg

Once all these changes have been verified in the file automatic/app.py, then we build a new container image with the following command:

$ podman build -t hello-otel:auto-manual -f automatic/Buildfile-auto

Successfully tagged localhost/hello-otel:auto-manual  \
516c5299a32b68e7a4634ce15d1fd659eed2164ebe945ef1673f7a55630e22c8

When we run this container image we are again wrapping it with the OpenTelemetry agent, known as opentelemetry-instrument. Configures a global tracer, which we set with a flag to output to the console, and provide a service name for our application with the following command:

$ podman run -i -p 8001:8000 -e FLASK_RUN_PORT=8000 hello-otel:auto-manual \

    opentelemetry-instrument \

    --traces_exporter console \

    --metrics_exporter none   \

    --service_name hello-otel \

    flask run --host=0.0.0.0

* Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. 

Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:8000
 * Running on http://10.88.0.19:8000
Press CTRL+C to quit

Now generate telemetry data (tracing) in your console to verify the new manual attribute in the existing span generated by auto-instrumentation. We do this by opening the main page at http://localhost:8001 in our browser and examining the trace in our running container console log and locating the hits entry:

10.88.0.20 - - [12/Jul/2024 11:12:49] "GET / HTTP/1.1" 200 -
{
    "name": "GET /",
    "context": {
        "trace_id": "0x6dc306ffe9c6add8df597e8a18878bd6",
        "span_id": "0x4018ff1f2b51431d",
        "trace_state": "[]"
    },
    "kind": "SpanKind.SERVER",
    "parent_id": null,
    "start_time": "2024-07-12T11:12:49.077685Z",
    "end_time": "2024-07-12T11:12:49.081377Z",
    "status": {
        "status_code": "UNSET"
    },
    "attributes": {
        "http.method": "GET",
        "http.server_name": "0.0.0.0",
        "http.scheme": "http",
        "net.host.name": "localhost:8001",
        "http.host": "localhost:8001",
        "net.host.port": 8000,
        "http.target": "/",
        "net.peer.ip": "10.88.0.20",
        "net.peer.port": 37058,
        "http.user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ...",
        "http.flavor": "1.1",
        "http.route": "/",

        "hits": 1, 
        "http.status_code": 200

These examples use code from a Python application that you can explore in the provided hands-on workshop. There is more reading available for you on learning about the basics of OpenTelemetry.

What's next?

This article explored the use of OpenTelemetry auto-instrumentation and how to combine that automation with manual instrumentation to enhance our insights into the functioning of our application.

Next up, diving into programmatic instrumentation with OpenTelemetry.