Idea "Mask Urls" for Url transformation in response

[dimitridaras]

New Feature

It would be great if Ocelot could include built-in support for URL transformation in response bodies and the location header (called, for instance, Mask URLs in content in Transformation | Azure API Management). I.e., a downstream service URL in the response is converted to the upstream gateway URL.

Motivation for New Feature

Restful APIs should include HATEOAS - useful links to other endpoints within the response. Most people agree that these should be absolute. The service only knows about itself - it shouldn't know anything about the gateway, but the client should only know about the gateway; therefore the gateway needs to convert service URLs to gateway URLs.

[davidni]

On a related note, there are cases where a service would still have to know its public endpoint that the client used to reach it through the Gateway (Ocelot). A common mechanism is that the gateway would add the non-standardized headers X-Forwarded-Proto. X-Forwarded-Host, X-Forwarded-Port, etc. More recently there have been attempts at standardizing a newer form of those as the Forwarded header (RFC 7239).

Popular packages such as Swashbuckle would magically just work behind Ocelot if the X-Forwarded-* headers were added by Ocelot (relevant code in Swashbuckle that uses those headers) and perhaps a similar approach would also work for the @dimitridaras.

[raman-m]

Indeed, for this solution, we need to append additional X-Forwarded-* headers to the request for the downstream service using a delegating handler, as there is no official support for this beyond the X-Forwarded-For feature. Subsequently, the downstream service will be capable of generating the subsequent workflow URL.
Our team believes that the existing Headers Transformation feature addresses 95% of user scenarios. The only limitation lies in the Placeholders which is a predefined list of transformation artifacts.

[TomPallister]

@dimitridaras Ocelot can already transform the Location header, see the docs here.

@dimitridaras @davidni I agree this would be a good feature, one day I might get round to it but for now I don't have time :(.

I am so busy with work, life and just fixing bugs and answering questions. At the moment my main focus is just to improve the code quality of Ocelot and make it more modular. A lot of the code isn't very good and this was mainly because when I started the project I was just excited to have users requesting features so a lot of features were created without much though for other scenarios (such as the one requested here).

However let's keep this open and we can do it one day.

[dimitridaras]

I am still quite new to Ocelot. In the meantime, what is the best workaround? A delegating handler?

[raman-m]

Have you tried Delegating Handlers to mask URLs?

[TomPallister]

@dimitridaras you can either use a delegating handler, inject middleware or just a middleware before the Ocelot pipeline, I think you can manipulate the HttpContext object Ocelot will return. Though I'm not 100% sure of that.

[raman-m]

Hi @dimitridaras !
How are you?
Still want this feature?

[dimitridaras]

Hi. I don't need this feature at the moment, as I have not been working on anything that uses Ocelot for a few years now. To be honest, I thought that Ocelot was no longer being maintained, but I'm delighted to see that it is still active.

[raman-m]

Sad to hear that! 😢

[raman-m]

serviceCollection.AddTransient<IRestClient>((serviceProvider) => new RestClient("http://api.worldbank.org/v2"));

http://api.worldbank.org/v2 💩 😝

[alexbereznikov]

For those who look for workaround, here is something to start with

internal sealed class UrlRewriteDelegatingHandler : DelegatingHandler
{
    private static readonly JsonSerializerOptions SerializerOptions = CreateSerializerOptions();

    private readonly IHttpContextAccessor _httpContextAccessor;

    public UrlRewriteDelegatingHandler(IHttpContextAccessor httpContextAccessor)
    {
        _httpContextAccessor = httpContextAccessor;
    }

    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        using var responseMessage = await base.SendAsync(request, cancellationToken).ConfigureAwait(false);

        var responseStream = await responseMessage.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false);
        await using var responseStreamDisposable = responseStream.ConfigureAwait(false);

        var dictionary = await JsonSerializer.DeserializeAsync<IDictionary<string, object?>>(responseStream, SerializerOptions, cancellationToken).ConfigureAwait(false);

        if (dictionary is not null)
        {
            RewriteDictionary(dictionary);
        }

        var newResponseStream = new MemoryStream();

        await JsonSerializer.SerializeAsync(newResponseStream, dictionary, SerializerOptions, cancellationToken).ConfigureAwait(false);

        newResponseStream.Seek(0, SeekOrigin.Begin);

        var newResponseMessage = new HttpResponseMessage(responseMessage.StatusCode)
        {
            ReasonPhrase = responseMessage.ReasonPhrase,
            Version = responseMessage.Version,
        };

        foreach (var (key, value) in responseMessage.Headers)
        {
            newResponseMessage.Headers.TryAddWithoutValidation(key, value);
        }

        newResponseMessage.Content = new StreamContent(newResponseStream);

        foreach (var (key, value) in responseMessage.Content.Headers)
        {
            newResponseMessage.Content.Headers.TryAddWithoutValidation(key, value);
        }

        return newResponseMessage;
    }

    private static JsonSerializerOptions CreateSerializerOptions()
    {
        var options = new JsonSerializerOptions();

        options.Converters.Add(new JsonDictionaryConverterFactory());

        return options;
    }

    private static string MakePathRegex(string pathTemplate)
    {
        return pathTemplate
            .Replace("{", "(?<", StringComparison.OrdinalIgnoreCase)
            .Replace("}", ">.+)", StringComparison.OrdinalIgnoreCase);
    }

    private void RewriteDictionary(IDictionary<string, object?> dictionary)
    {
        foreach (var (key, value) in dictionary)
        {
            if (value is string url && Uri.TryCreate(url, UriKind.Absolute, out var uri))
            {
                dictionary[key] = RewriteUrl(uri);
            }
            else if (value is IDictionary<string, object?> nested)
            {
                RewriteDictionary(nested);
            }
        }
    }

    private string RewriteUrl(Uri uri)
    {
        var route = FindMatchingRoute(uri);

        if (route is null)
        {
            return uri.ToString();
        }

        var pathTemplate = MakePathRegex(route.DownstreamPathTemplate.Value);

        var match = Regex.Match(uri.PathAndQuery, pathTemplate);
        var groups = ((IEnumerable<Group>)match.Groups).Skip(1).Select(group => (group.Name, group.Value));

        var upstreamTemplate = route.UpstreamPathTemplate.OriginalValue;

        foreach (var (name, value) in groups)
        {
            upstreamTemplate = upstreamTemplate.Replace($"{{{name}}}", value, StringComparison.OrdinalIgnoreCase);
        }

        var request = _httpContextAccessor.HttpContext?.Request ?? throw new InvalidOperationException("No request present.");
        var uriBuilder = new UriBuilder(
            request.Scheme,
            request.Host.Host);

        if (request.Host.Port.HasValue)
        {
            uriBuilder.Port = request.Host.Port.Value;
        }

        uriBuilder.Path = upstreamTemplate;

        return uriBuilder.ToString();
    }

    private DownstreamRoute? FindMatchingRoute(Uri uri)
    {
        var configuration = _httpContextAccessor.HttpContext?.Items.IInternalConfiguration();

        if (configuration is null)
        {
            return null;
        }

        var routes = configuration.Routes;

        foreach (var route in routes)
        {
            foreach (var downstreamRoute in route.DownstreamRoute)
            {
                if (MatchesRoute(downstreamRoute, uri))
                {
                    return downstreamRoute;
                }
            }
        }

        return null;

        static bool MatchesRoute(DownstreamRoute routeToMatch, Uri uriToMatch)
        {
            var uriPath = uriToMatch.PathAndQuery;
            var pathTemplate = MakePathRegex(routeToMatch.DownstreamPathTemplate.Value);

            return Regex.IsMatch(uriPath, pathTemplate);
        }
    }
}

[raman-m]

Hello Alex!
What does your code do actually?
Could you write some test cases to present this solution plz?

[alexbereznikov]

Hello Raman, this handler solves one particular issue we faced.

So we have a controller action that returns some json. In this json, some fields are links to another controller/action.
Since our service is behind the gateway, we can't just return these links to client. So we need to turn them to gateway links somehow.
I thought about passing some header to the service first and do the conversion there, but routing rules might be different for different actions, so, in my opinion, best place to do this is the gateway.

This hander deserializes downstream response json to IDictionary<string, object?> using a custom converter. Then, it looks into every string field, and tries to create an Uri instance (so that it affects only links). Then, for each found link, it tries to find a matching route in Ocelot config using regex. If the route is found - it replaces placeholders with groups from matched regex. After that, the dictionary is serialized again, and a new response is returned to the client.

I don't think this code is ready to be integrated in Ocelot, as it might not cover some edge cases and might not be optimal. If you think it makes sense to add something like this to Ocelot - I can create a branch with tests when I have time.

[raman-m]

Hello Raman, this handler solves one particular issue we faced.

I've realized that URL Rewrite is practical only for BFF mode (UI clients). For data-driven integrations, such as those in a microservices architecture, URL Rewrite is nearly redundant.

---

So we have a controller action that returns some json. In this json, some fields are links to another controller/action.
Since our service is behind the gateway, we can't just return these links to client. So we need to turn them to gateway links somehow.

The question is, "Who is the developer of the downstream service?" If it's you, a variety of solutions can be implemented. If it's a third-party service, your options for such conversions are limited to the client's side or the gateway's side. If you are the developer of the downstream service, you have more freedom. However, the challenge is that your downstream service is not aware of the gateway's routing, necessitating a mapping of service/controller routes to gateway routes.

I thought about passing some header to the service first and do the conversion there, but routing rules might be different for different actions, so, in my opinion, best place to do this is the gateway.

You're on the right track! 😉 Have you explored, experimented with, and implemented a solution based on the official Headers Transformation feature, especially Handling 302 Redirects one? For a sequential workflow, you would need to forward the subsequent (rollback) URL or routing information, correct? In the case of a parallel workflow, it appears that a similar approach could be employed based on a custom header.
Moreover, have you examined this #1821 (comment)? We would like to understand what might be lacking in this official Headers Transformation feature.

---

This hander deserializes downstream response json to IDictionary<string, object?> using a custom converter. Then, it looks into every string field, and tries to create an Uri instance (so that it affects only links). Then, for each found link, it tries to find a matching route in Ocelot config using regex. If the route is found - it replaces placeholders with groups from matched regex. After that, the dictionary is serialized again, and a new response is returned to the client.

When using a third-party downstream service your solution could be a viable option. However, scanning and analyzing the response body is not ideal as it can degrade the gateway's performance. Solutions that involve body manipulations should be avoided. In the worst-case scenario of consuming third-party services, it appears these services are not aware of your workflow and are unable to generate workflow URLs.

[raman-m]

I don't think this code is ready to be integrated in Ocelot, as it might not cover some edge cases and might not be optimal.

Fair enough! As previously mentioned, your solution involves body manipulations that could impair the overall performance of the gateway. I believe this approach is not suitable. Consequently, we cannot incorporate this feature design into the Ocelot kernel as an official feature.
However, we do see potential in transforming this design solution into an official extension package. The possible delivery routes should be explored and sanctioned by the team.

If you think it makes sense to add something like this to Ocelot - I can create a branch with tests when I have time.

Make sense! It's feasible to develop an additional Ocelot package, like Ocelot.BFFUtils or Ocelot.DelegatingHandlers. However, this would necessitate a distinct delivery milestone and an essential presentation sample app.
If you're keen to contribute, we can commence development. Initially, you should fork the Ocelot repository into your repositories because our Dev Process requires forking. Subsequently, when you have spare time, contact me, and we'll strategize our next steps.