Aspire Dashboard: Traces: Span links aren't shown in span details

[samsp-msft]

When dealing with messaging systems, you typically don't want spans trying to encapsulate the processing of messages that you don't have control over, so rather than having a parent/child relationship, links can be used instead.
The dashboard does not currently display links in the details view for spans.
Also linked spans are not shown unless they have the same TraceID and it's unclear if that will always happen in a message based system.

We should :-

1. Add a section for linked spans to the details of a span- and make it transitive, so show the links to the currrent span, and from the current span (they may not be the same)
2. Consider adding linked spans to the trace view to show the other spans that current ones are linked to.

[samsp-msft]

Sample code for the API Service to create an unparented linked span:

var apiActivitySource = new ActivitySource("CustomActivitySource");

app.MapGet("/weatherforecast", () =>
{
    var parent = Activity.Current;
    Activity.Current = null;
    var links = new List<ActivityLink>
        {
            new ActivityLink(parent.Context),
        };

    var pContext = new ActivityContext(parent.Context.TraceId, new ActivitySpanId(), ActivityTraceFlags.Recorded);
    var activity = apiActivitySource.StartActivity("LinkTest", ActivityKind.Internal, pContext, null, links);

WeatherForecast[] forecast;
    using (activity)
    {
        forecast = Enumerable.Range(1, 5).Select(index =>
             new WeatherForecast
             (
                 DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
                 Random.Shared.Next(-20, 55),
                 summaries[Random.Shared.Next(summaries.Length)]
             ))
             .ToArray();
    }
    Activity.Current = parent;
    return forecast;
});

Add the following to ConfigureOpenTelemetry(...) in the ServiceDefaults to add the activitySource and console output for diagnostic purposes.

                tracing.AddSource("CustomActivitySource");
                tracing.AddConsoleExporter();

You want a trace with the following properties to test with:

Activity.TraceId:            5f27e3762a87be1eea8bc65a70ec2893
Activity.SpanId:             f77c6ad8a3d3cedc
Activity.TraceFlags:         Recorded
Activity.ActivitySourceName: CustomActivitySource
Activity.DisplayName:        LinkTest
Activity.Kind:               Internal
Activity.StartTime:          2024-03-02T00:15:59.6368281Z
Activity.Duration:           00:00:00.0014354
Activity.Links:
    5f27e3762a87be1eea8bc65a70ec2893 e66a00a9ee3f676c
Resource associated with Activity:
    telemetry.distro.name: Azure.Monitor.OpenTelemetry.AspNetCore
    service.name: apiservice
    service.instance.id: apiservice-32dv1ro
    telemetry.sdk.name: opentelemetry
    telemetry.sdk.language: dotnet
    telemetry.sdk.version: 1.7.0

It doesn't have an Activity.ParentId but does have the Activity.Links

[samsp-msft]

@JamesNK

[JamesNK]

Sure, we can do this. Links are not must have and no one has asked for it yet so it probably won't happen before V1.

[davidfowl]

Moving to backlog

[martinjt]

I only just figured out that this wasn't supported.

I had people mentioning it in my talk today, and found out I can't use it for my workshop since messaging is normally span links.

For another datapoint, ServiceBus's instrumentation OOTB (the experimental flagged one) creates links for the Receive spans, which makes the dashboard very confusing.

I think this is going to become a bigger issue as the Messaging semantic conventions will be being stablised in the next few months, making it non-experimental in the .NET Azure SDKs.

[slang25]

I've been working on some tracing of a message bus library (AWS based) I look after and discovered this same issue, I was (and still am) expecting to see linked spans displayed in a meaningful way in the Aspire dashboard. This is an important feature for successful adoption.

[danmoseley]

Any interest in a contribution? 🙂

[JamesNK]

Request to people who are interested in this:

• How do other UIs represent span links? Screenshots would be helpful if it's not too much trouble.
• Are span links typically between spans inside the same trace, or can they link to different traces?
• What do you expect to happen when clicking on a span link?
• What happens if a span link is to a span that's not in the system? For example, I imagine sampling could mean a linked span isn't available. Alternatively, data can be removed from Aspire once limits are hit. The linked span could have be removed to make room for new data.

We already capture span links. The work involved is displaying it in the UI. It would be great to see how systems represent links between spans.

Ideas for displaying span link:

• Span ID as clickable link. Simple but not user friendly.
• Span name as clickable link. Would include application name.
• Mini summary of span as a link. Application name + span name + start time + duration + status?

When you click on a span link I expect that it would:

1. Navigate to the correct trace detail view (if the span link is on a different trace)
2. Open the details view for the linked span. This would highlight the span in the trace detail UI.
3. If a trace has many spans, then the UI should scroll the trace detail UI so the selected span is visible. I don't think we have an easy way to do that right now. Maybe a follow up feature.

[martinjt]

Span links are multiple per span, and can be attached to any span (root or
descendent) on either side, with attributes for each link. So I'd expect to see a list of the span links being able to see the details of each link and
it's attributes.

In terms of navigation, I'd expect to be taken to the details view of the
span inside the trace it's been linked to.

Span links are most useful when they link between spans in different
traces. The big usecase is in long running contexts across multiple
services in event driven systems. It's not something I'd expect to see in
aspire in production (mainly due to the timescale of the trace linking back
to something from minutes or hours before), so it's mainly a dev workflow.

Happy to send some honeycomb/jaeger screenshots, but it'll be next week.

Bonus points if you backlink automatically (which is probably easier with the inmemory model).

[JamesNK]

Backlinking shouldn't be difficult.

Although, the UI would be slow if a span has thousands of backlinks to display. Simple fix is to put a limit on rendered backlinks (e.g. 100). If there is feedback that people have spans with lots of backlinks and they want to see them all then the UI could be virtualized.

[martinjt]

The rendering is the same problem either way really.

A span could have a link to 100 other spans.

A span could be linked to from 100 other spans.

If you're putting in backlinks then you need to distinguish them from outbound/forward links in the UI somehow. Which does make it complicated and there aren't any solutions right now doing it that I know about.

The main thing is how the display works with 100+ outbound links.

[JamesNK]

PR: #4805 cc @martinjt

[Barsonax]

We are actually using span links to break up our traces in more managable chunks while still retaining the ability to see the big picture. The full picture is just too big to render all at the same time. I noticed app insights renders the links but the aspire dashboard indeed doesn't.

Would be nice to see this working in the aspire dashboard soon so kudos to JamesNK.