Skip to content

mgodse/opentelemetry-dotnet

 
 

Repository files navigation

OpenTelemetry .NET SDK - distributed tracing and stats collection framework

.NET Channel: Gitter chat

Community Channel: Gitter chat

We hold regular meetings. See details at community page.

Approvers (@open-telemetry/dotnet-approvers):

Find more about the approver role in community repository.

Maintainers (@open-telemetry/dotnet-maintainers):

Find more about the maintainer role in community repository.

Build Status

OpenTelemetry is a toolkit for collecting application performance and behavior data.

The library is in Alpha stage. The library is expected to move to GA stage after v1.0.0 major release.

Please join gitter for help or feedback on this project.

We encourage contributions. Use tags up-for-grabs and good first issue to get started with the project. Follow CONTRIBUTING guide to report issues or submit a proposal.

Packages

Nightly builds

Myget feeds:

API and implementation

Package MyGet (CI) NuGet (releases)
OpenTelemetry MyGet Nightly NuGet Release
OpenTelemetry.Api MyGet Nightly NuGet Release

Data Collectors

Package MyGet (CI) NuGet (releases)
Asp.Net Core MyGet Nightly NuGet Release
.Net Core HttpClient & Azure SDKs MyGet Nightly NuGet Release
StackExchange.Redis MyGet Nightly NuGet Release

Exporters Packages

Package MyGet (CI) NuGet (releases)
Zipkin MyGet Nightly NuGet release
Prometheus MyGet Nightly NuGet release
Application Insights MyGet Nightly NuGet release
Stackdriver MyGet Nightly NuGet release
Jaeger MyGet Nightly NuGet release
LightStep MyGet Nightly NuGet release
NewRelic NuGet release

OpenTelemetry Tracing QuickStart: collecting data

You can use OpenTelemetry API to instrument code and report data. Check out Tracing API overview to learn more about distributed tracing.

In the examples below we demonstrate how to create and enrich spans though OpenTelemetry API.

OpenTelemetry also provides auto-collectors for ASP.NET Core, HttpClient calls (.NET Core) and Azure SDKs - configuration section demonstrates how to enable it.

Obtaining tracer

Applications should follow configuration section to find out how to create/obtain tracer.

Libraries must take dependency on OpenTelemetry API package only and should never instantiate tracer or configure OpenTelemetry. Libraries will be able to obtain global tracer that may either be noop (if user application is not instrumented with OpenTelemetry) or real tracer implemented in the SDK package.

Create basic span

To create the most basic span, you only specify the name. OpenTelemetry SDK collects start/end timestamps, assigns tracing context and assumes status of this span is OK.

var span = tracer.StartSpan("basic span");
// ...
span.End();

Create nested spans

In many cases you want to collect nested operations. You can propagate parent spans explicitly in your code or use implicit context propagation embedded into OpenTelemetry and .NET.

Explicit parent propagation and assignment

var parentSpan = tracer.StartSpan("parent span");

// explicitly assigning parent here
var childSpan = tracer.StartSpan("child span", parentSpan);

childSpan.End();
parentSpan.End();

Implicit parent propagation and assignment

// calling StartActiveSpan starts a span and puts parentSpan into the ambient context
// that flows in async calls.   When child is created, it implicitly becomes child of current span
using (tracer.StartActiveSpan("parent span", out _))
{
    var childSpan = tracer.StartSpan("child span");

    childSpan.End();
}

// parent span is ended when StartActiveSpan result is disposed

Span with attributes

Attributes provide additional context on span specific to specific operation it tracks such as HTTP/DB/etc call properties.

// spans have Client, Server, Internal, Producer and Consumer kinds to help visualize them
var span = tracer.StartSpan("span with attributes", SpanKind.Client);

// attributes specific to the call
span.SetAttribute("db.type", "redis");
span.SetAttribute("db.instance", "localhost:6379[0]");
span.SetAttribute("db.statement", "SET");
span.End();

Span with links

Links allow to create relationships between different traces i.e. allow spans to have multiple relatives. They are typically used to tracer batching scenarios where multiple traces are merged into another one.

Links affect sampling decision and should be added before sampling decision is made (i.e. before span starts).

SpanContext link1 = ExtractContext(eventHubMessage1);
SpanContext link2 = ExtractContext(eventHubMessage2);

var span = tracer.StartSpan("span with links", SpanKind.Server, DateTime.UtcNow, new [] {link1, link2});
span.End();

Span with events

Events are timed text (with optional attributes) annotations on the span. Events can be added to current span (or any running span).

using (tracer.StartActiveSpan("incoming HTTP request", SpanKind.Server, out var span))
{
    span.AddEvent("routes resolved");
}

// span is ended when StartActiveSpan result is disposed

Context propagation out of process

When instrumenting transport-layer operations, instrumentation should support context propagation.

// this extracts W3C trace-context from incoming HTTP request
// context may be valid if present and correct in the headers
// or invalid if there was no context (or it was not valid)
// instrumentation code should not care about it
var context = tracer.TextFormat.Extract(incomingRequest.Headers, (headers, name) => headers[name]);

var incomingSpan = tracer.StartSpan("incoming http request", context, SpanKind.Server);

var outgoingRequest = new HttpRequestMessage(HttpMethod.Get, "http://microsoft.com");
var outgoingSpan = tracer.StartSpan("outgoing http request", SpanKind.Client);

// now that we have outgoing span, we can inject it's context
// Note that if there is no SDK configured, tracer is noop -
// it creates noop spans with invalid context. we should not propagate it.
if (outgoingSpan.Context.IsValid)
{
    tracer.TextFormat.Inject(
        outgoingSpan.Context,
        outgoingRequest.Headers,
        (headers, name, value) => headers.Add(name, value));
}

// make outgoing call
// ...

outgoingSpan.End();
incomingSpan.End();

Auto-collector implementation for Activity/DiagnosticSource

System.Diagnostics.Activity is similar to OpenTelemetry Span. HttpClient, ASP.NET Core, Azure SDKs use them to expose diagnostics events and context.

Leaving aside subscription mechanism, here is an example how you may implement callbacks for Start/Stop Activity

void StartActivity()
{
    var span = tracer.StartSpanFromActivity("GET api/values", Activity.Current);

    // extract other things from Activity and set them on span (tags to attributes)
    // ...
}

void StopActivity()
{
    var span = tracer.CurrentSpan;
	
	span.End();
    if (span is IDisposable disposableSpan)
    {
        disposableSpan.Dispose();
    }
}

Configuration

Configuration is done by user application: it should configure exporter and may also tune sampler and other properties.

Basic Configuration

  1. Install packages to your project: OpenTelemetry OpenTelemetry.Exporter.Zipkin

  2. Create TracerFactory

    using (TracerFactory.Create(builder => builder
            .UseZipkin()
            .SetResource(new Resource(new Dictionary<string, string>() { { "service.name", "http-client-test" } })))
    {
        // ...
    }

Configuration with Microsoft.Extensions.DependencyInjection

  1. Install packages to your project: OpenTelemetry.Hosting to provide AddOpenTelemetry helper method OpenTelemetry.Collector.AspNetCore to collect incoming HTTP requests OpenTelemetry.Collector.Dependencies to collect outgoing HTTP requests and Azure SDK calls

  2. Make sure TracerFactory, is registered in DI.

    services.AddOpenTelemetry(builder =>
    {
        builder
            .SetSampler(new AlwaysSampleSampler())
            .UseZipkin()
    
            // you may also configure request and dependencies collectors
            .AddRequestCollector()
            .AddDependencyCollector()
            .SetResource(new Resource(new Dictionary<string, string>() { { "service.name", "my-service" } }))
    });

Using StackExchange.Redis collector

Outgoing http calls to Redis made using StackExchange.Redis library can be automatically tracked.

  1. Install package to your project: OpenTelemetry.Collector.StackExchangeRedis

  2. Configure Redis collector

    // connect to the server
    var connection = ConnectionMultiplexer.Connect("localhost:6379");
    
    using (TracerFactory.Create(b => b
                .SetSampler(new AlwaysSampleSampler())
                .UseZipkin()
                .SetResource(new Resource(new Dictionary<string, string>() { { "service.name", "my-service" } }))
                .AddCollector(t =>
                {
                    var collector = new StackExchangeRedisCallsCollector(t);
                    connection.RegisterProfiler(collector.GetProfilerSessionsFactory());
                    return collector;
                })))
    {
    
    }

You can combine it with dependency injection as shown in previous example.

Custom samplers

You may configure sampler of your choice

 using (TracerFactory.Create(b => b
            .SetSampler(new ProbabilitySampler(0.1))
            .UseZipkin()
            .SetResource(new Resource(new Dictionary<string, string>() { { "service.name", "my-service" } })))
{

}

You can also implement custom sampler by implementing ISampler interface

class MySampler : Sampler
{
    public override string Description { get; } = "my custom sampler";

    public override Decision ShouldSample(SpanContext parentContext, ActivityTraceId traceId, ActivitySpanId spanId, string name,
        IDictionary<string, object> attributes, IEnumerable<Link> links)
    {
        bool sampledIn;
        if (parentContext != null && parentContext.IsValid)
        {
            sampledIn = (parentContext.TraceOptions & ActivityTraceFlags.Recorded) != 0;
        }
        else
        {
            sampledIn = Stopwatch.GetTimestamp() % 2 == 0;
        }

        return new Decision(sampledIn);
    }
}

OpenTelemetry QuickStart: exporting data

Using the Jaeger exporter

The Jaeger exporter communicates to a Jaeger Agent through the compact thrift protocol on the Compact Thrift API port. You can configure the Jaeger exporter by following the directions below:

  1. Get Jaeger.
  2. Configure the JaegerExporter
    • ServiceName: The name of your application or service.
    • AgentHost: Usually localhost since an agent should usually be running on the same machine as your application or service.
    • AgentPort: The compact thrift protocol port of the Jaeger Agent (default 6831)
    • MaxPacketSize: The maximum size of each UDP packet that gets sent to the agent. (default 65000)
  3. See the sample for an example of how to use the exporter.
using (var tracerFactory = TracerFactory.Create(
    builder => builder.UseJaeger(o =>
    {
        o.ServiceName = "jaeger-test";
        o.AgentHost = "<jaeger server>";
    })))
{
    var tracer = tracerFactory.GetTracer("jaeger-test");
    using (tracer.StartActiveSpan("incoming request", out var span))
    {
        span.SetAttribute("custom-attribute", 55);
        await Task.Delay(1000);
    }
}

Using Zipkin exporter

Configure Zipkin exporter to see traces in Zipkin UI.

  1. Get Zipkin using getting started guide.
  2. Configure ZipkinTraceExporter as below:
  3. See sample for example use.
using (var tracerFactory = TracerFactory.Create(
    builder => builder.UseZipkin(o =>
        o.ServiceName = "my-service";
        o.Endpoint = new Uri("https://<zipkin-server:9411>/api/v2/spans"))))
{
    var tracer = tracerFactory.GetTracer("zipkin-test");
    var span = tracer
        .SpanBuilder("incoming request")
        .StartSpan();

    await Task.Delay(1000);
    span.End();
}

Using Prometheus exporter

Configure Prometheus exporter to have stats collected by Prometheus.

  1. Get Prometheus using getting started guide.
  2. Start PrometheusExporter as below.
  3. See sample for example use.
var exporter = new PrometheusExporter(
    new PrometheusExporterOptions()
    {
        Url = "http://+:9184/metrics/"
    },
    Stats.ViewManager);

exporter.Start();

try
{
    // record metrics
    statsRecorder.NewMeasureMap().Put(VideoSize, values[0] * MiB).Record();
}
finally
{
    exporter.Stop();
}

Using the NewRelic exporter

The New Relic OpenTelemetry Trace Exporter is a OpenTelemetry Provider that sends data from .NET applications to New Relic. It uses the NewRelic SDK to send Traces to the New Relic backend

Please refer to the New Relic Exporter Documentation

Using Stackdriver Exporter

This sample assumes your code authenticates to Stackdriver APIs using service account with credentials stored in environment variable GOOGLE_APPLICATION_CREDENTIALS. When you run on GAE, GKE or locally with gcloud sdk installed - this is typically the case. There is also a constructor for specifying path to the service account credential. See sample for details.

  1. Add Stackdriver Exporter package reference.
  2. Enable Stackdriver Trace API.
  3. Enable Stackdriver Monitoring API.
  4. Instantiate a new instance of StackdriverExporter with your Google Cloud's ProjectId
  5. See sample for example use.

Advanced configuration

You may want to filter on enrich spans and send them to multiple destinations (e.g. for debugging or telemetry self-diagnostics purposes). You may configure multiple processing pipelines for each destination like shown in below example.

In this example

  1. First pipeline sends all sampled in spans to Zipkin
  2. Second pipeline sends spans to ApplicationInsights, but filters them first with custom built FilteringSpanProcessor
  3. Third pipeline adds custom DebuggingSpanProcessor that simply logs all calls to debug output
using (var tracerFactory = TracerFactory.Create(builder => builder
    .UseZipkin(o =>
    {
        o.Endpoint = new Uri(zipkinUri);
    })
    .UseApplicationInsights(
        o => o.InstrumentationKey = "your-instrumentation-key",
        p => p.AddProcessor(nextProcessor => new FilteringSpanProcessor(nextProcessor)))
    .AddProcessorPipeline(pipelineBuilder => pipelineBuilder.AddProcessor(_ => new DebuggingSpanProcessor()))))
    .SetResource(new Resource(new Dictionary<string, string>() { { "service.name", "test-zipkin" } }))

{
    // ...
}

Traces

using (var tracerFactory = TracerFactory.Create(builder => builder
    .AddProcessorPipeline(c => c.SetExporter(new StackdriverTraceExporter("YOUR-GOOGLE-PROJECT-ID")))))
{
    var tracer = tracerFactory.GetTracer("stackdriver-test");
    var span = tracer
        .SpanBuilder("incoming request")
        .StartSpan();

    await Task.Delay(1000);
    span.End();
}

Metrics

var metricExporter = new StackdriverExporter(
    "YOUR-GOOGLE-PROJECT-ID",
    Stats.ViewManager);
metricExporter.Start();

Using Application Insights exporter

  1. Create Application Insights resource.
  2. Set instrumentation key via telemetry configuration object (new TelemetryConfiguration("iKey")). This object may be injected via dependency injection as well.
  3. Instantiate a new instance of ApplicationInsightsExporter.
  4. See sample for example use.
using (var tracerFactory = TracerFactory.Create(builder => builder
    .UseApplicationInsights(config => config.InstrumentationKey = "instrumentation-key")))
{
    var tracer = tracerFactory.GetTracer("application-insights-test");
    var span = tracer
        .SpanBuilder("incoming request")
        .StartSpan();

    await Task.Delay(1000);
    span.End();
}

Using LightStep exporter

Configure LightStep exporter to see traces in LightStep.

  1. Setup LightStep using getting started guide
  2. Configure LightStepTraceExporter (see below)
  3. See sample for example use
using (var tracerFactory = TracerFactory.Create(
    builder => builder.UseLightStep(o =>
        {
            o.AccessToken = "<access-token>";
            o.ServiceName = "lightstep-test";
        })))
{
    var tracer = tracerFactory.GetTracer("lightstep-test");
    using (tracer.StartActiveSpan("incoming request", out var span))
    {
        span.SetAttribute("custom-attribute", 55);
        await Task.Delay(1000);
    }
}

Implementing your own exporter

Tracing

Exporters should subclass SpanExporter and implement ExportAsync and Shutdown methods. Depending on user's choice and load on the application ExportAsync may get called concurrently with zero or more spans. Exporters should expect to receive only sampled-in ended spans. Exporters must not throw. Exporters should not modify spans they receive (the same span may be exported again by different exporter).

It's a good practice to make exporter IDisposable and shut it down in IDispose unless it was shut down explicitly. This helps when exporters are registered with dependency injection framework and their lifetime is tight to the app lifetime.

class MyExporter : SpanExporter
{
    public override Task<ExportResult> ExportAsync(IEnumerable<Span> batch, CancellationToken cancellationToken)
    {
        foreach (var span in batch)
        {
            Console.WriteLine($"[{span.StartTimestamp:o}] {span.Name} {span.Context.TraceId.ToHexString()} {span.Context.SpanId.ToHexString()}");
        }

        return Task.FromResult(ExportResult.Success);
    }

    public override Task ShutdownAsync(CancellationToken cancellationToken)
    {
        return Task.CompletedTask;
    }
}

Users may configure the exporter similarly to other exporters. You should also provide additional methods to simplify configuration similarly to UseZipkin extension method.

var exporter = new MyExporter();
using (var tracerFactory = TracerFactory.Create(
    builder => builder.AddProcessorPipeline(b => b.SetExporter(new MyExporter())))
{
    // ...
}

Versioning

This library follows Semantic Versioning.

GA: Libraries defined at a GA quality level are stable, and will not introduce backwards-incompatible changes in any minor or patch releases. We will address issues and requests with the highest priority. If we were to make a backwards-incompatible changes on an API, we will first mark the existing API as deprecated and keep it for 18 months before removing it.

Beta: Libraries defined at a Beta quality level are expected to be mostly stable and we're working towards their release candidate. We will address issues and requests with a higher priority. There may be backwards incompatible changes in a minor version release, though not in a patch release. If an element is part of an API that is only meant to be used by exporters or other OpenTelemetry libraries, then there is no deprecation period. Otherwise, we will deprecate it for 18 months before removing it, if possible.

About

OpenTelemetry .NET SDK

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • C# 100.0%