The RPS decodes URL encoded characters, leading to 404 responses when querying the Gitlab Project API

RPS Service Release: R12

Actual Behavior

The Gitlab Project API has endpoints including the project path (namespace and name) as an URL encoded path segment:

• /api/v4/projects/foo%2Fbar%2Fbaz
• /api/v4/projects/foo%2Fbar%2Fbaz/archive
• ...

However, NGINX decodes those URLs when it should not:

• /api/v4/projects/foo/bar/baz
• /api/v4/projects/foo/bar/baz/archive

Which leads to 404 when trying to reach those API endpoints.

On another note, the WAF Layer (powered by Apache2 + ModSecurity2) also reject such URLs.

Desired Behavior

The URLs should not be decoded.

Relevant Logs

N/A

added SDLC label

assigned to @delasda

Solution

WAF Layer

On the Apache2 side, we need to add at the top-level of the configuration:

AllowEncodedSlashes NoDecode

This will prevent Apache2 from decoding the URL encoded characters.

Then in each VirtualHost, we need to add the nocanon attribute to the ProxyPass directives, so that mod_proxy does not try to decode them either:

<VirtualHost *:443>
  # ...

  ProxyPass ... nocanon

  # ...
</VirtualHost>

Proxy Layer

In NGINX, we need to remove the rewrite directive, and replace it with an NJS script that will create a variable $proxy_uri from the variable $request_uri.

The rewrite directives act on the normalized $uri variable (URL decoded) and then the proxy_pass directive use the rewritten URI.

It is possible to give to proxy_pass the $request_uri instead (which contains the original, untouched non-decoded URI), like this:

proxy_pass http://upstream$request_uri;

However, we loose the subpath rewriting that the RPS Service supports, which is done at the moment with:

rewrite ^{{ src_path }}(.*) {{ tgt_path }}$1 break;

To map /foo/a to /bar/a for example.

Using an NJS script to create the $proxy_uri variable from the $request_uri variable is easier to avoid the URI normalization that NGINX does everywhere. Once that variable is created in the NJS script, it can be used like this:

proxy_pass http://upstream$proxy_uri;

mentioned in merge request config-controller!25 (merged)

mentioned in merge request waf!13 (merged)

mentioned in merge request proxy!4 (merged)

mentioned in merge request config-controller!26 (merged)

Deployment of v0.44.3 done in acceptance environment.

Job schedules have been disabled to avoid deploying it automatically to production.

Tests have been running for a while in both acceptance and production.

This seems to be definitely fixed.

closed

mentioned in issue #4 (closed)