304 Not Modified Broken with Response Compression Enabled

[dimitribouniol]

Fixed an issue where HTTP/2 connections would disconnect if response compression were used and a 304 Not Modified response was returned.

Motivation:

I discovered in the Vapor Discord that initially, Safari was having trouble loading content from my site https://jiiiii.moe, which uses both HTTP/2 and response compression. Loads seemed to be sporadic at best, dropping images and other resources seemingly randomly.

We discovered that this was only ever an issue for 304 Not Modified responses after we were able to reproduce with Chromium, which surfaced an issue understanding the HTTP/2 traffic, along with the headers it attempted and received:

Note that vapor not NIO ever logged any issues even with .trace logging enabled.

Vapor configuration:

app.http.server.configuration.supportPipelining = true
app.http.server.configuration.responseCompression = .enabled
app.http.server.configuration.hostname = "0.0.0.0"
app.http.server.configuration.port = 443
app.http.server.configuration.tlsConfiguration = ...
app.http.server.configuration.supportVersions = [.one, .two]
try await app.execute()

To reproduce in Safari:

1. Load a vapor site configured with HTTP/2 and response compression (https://jiiiii.moe will work until this fix is in)
2. Scroll down so everything loads
3. Force refresh with ⌥⌘R
4. Scroll down just a little (first green section on my site is a good point) to load some images from server, then back to the top
5. Refresh the usual way using ⌘R
6. Scroll past that point, all the way to the following section where resources are not fresh in memory or disk, and the browser sends cache headers to the server. Notice these resources no longer load.

To reproduce in Chromium:

1. Load the same site
2. Wait several minutes for the hot memory/disk cache to invalidate
3. Refresh, which causes the browser to ask the server if the resources changed.
   (I could not get Chromium to ask for them sooner, so the Safari steps are more consistent to trigger the bug)

For what its worth, cURL doesn't seem to have any issues:

% curl 'https://jiiiiiii.moe/images/jiiiii-chan.png' \
  -H 'accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8' \
  -H 'accept-language: en-US,en;q=0.6' \
  -H 'if-modified-since: Sun, 09 Jun 2024 10:27:55 GMT' \
  -H 'if-none-match: "1717928875-46689"' \
  -H 'priority: i' \
  -H 'referer: https://jiiiiiii.moe/' \
  -H 'sec-ch-ua: "Brave";v="125", "Chromium";v="125", "Not.A/Brand";v="24"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: image' \
  -H 'sec-fetch-mode: no-cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-gpc: 1' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36' -vv
* Host jiiiiiii.moe:443 was resolved.
* IPv6: (none)
* IPv4: 99.2.158.101
*   Trying 99.2.158.101:443...
* Connected to jiiiiiii.moe (99.2.158.101) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=jii.moe
*  start date: May 28 10:38:46 2024 GMT
*  expire date: Aug 26 10:38:45 2024 GMT
*  subjectAltName: host "jiiiiiii.moe" matched cert's "jiiiiiii.moe"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://jiiiiiii.moe/images/jiiiii-chan.png
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: jiiiiiii.moe]
* [HTTP/2] [1] [:path: /images/jiiiii-chan.png]
* [HTTP/2] [1] [accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8]
* [HTTP/2] [1] [accept-language: en-US,en;q=0.6]
* [HTTP/2] [1] [if-modified-since: Sun, 09 Jun 2024 10:27:55 GMT]
* [HTTP/2] [1] [if-none-match: "1717928875-46689"]
* [HTTP/2] [1] [priority: i]
* [HTTP/2] [1] [referer: https://jiiiiiii.moe/]
* [HTTP/2] [1] [sec-ch-ua: "Brave";v="125", "Chromium";v="125", "Not.A/Brand";v="24"]
* [HTTP/2] [1] [sec-ch-ua-mobile: ?0]
* [HTTP/2] [1] [sec-ch-ua-platform: "macOS"]
* [HTTP/2] [1] [sec-fetch-dest: image]
* [HTTP/2] [1] [sec-fetch-mode: no-cors]
* [HTTP/2] [1] [sec-fetch-site: same-origin]
* [HTTP/2] [1] [sec-gpc: 1]
* [HTTP/2] [1] [user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36]
> GET /images/jiiiii-chan.png HTTP/2
> Host: jiiiiiii.moe
> accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8
> accept-language: en-US,en;q=0.6
> if-modified-since: Sun, 09 Jun 2024 10:27:55 GMT
> if-none-match: "1717928875-46689"
> priority: i
> referer: https://jiiiiiii.moe/
> sec-ch-ua: "Brave";v="125", "Chromium";v="125", "Not.A/Brand";v="24"
> sec-ch-ua-mobile: ?0
> sec-ch-ua-platform: "macOS"
> sec-fetch-dest: image
> sec-fetch-mode: no-cors
> sec-fetch-site: same-origin
> sec-gpc: 1
> user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36
> 
< HTTP/2 304 
< last-modified: Sun, 09 Jun 2024 10:27:55 GMT
< etag: "1717928875-46689"
< content-length: 46689
< date: Sun, 09 Jun 2024 16:11:53 GMT
< 
* Connection #0 to host jiiiiiii.moe left intact

--

After these hints, I did more digging and found that HTTPResponseCompressor seemed to be skipping over this early exit:

Which seemed to cause an empty body to become 8 bytes before Chromium (and Safari) gave up trying to decompress and reported failure.

(It predictably skips straight to this end state in the compressor:)

Modifications:

In as straightforward as a fix as I could think of, I added a new case to mayHaveResponseBody to return false when the response code is 304 Not Modified, as this is prohibited by the spec anyways:

A 304 response is terminated by the end of the header section; it cannot contain content or trailers.

Ideally, we should follow up with a more holistic fix to skip compressing anything if the response has an empty body, but that should be necessary with the above fix in place.

Result:

With this fix in place, my server now returns 304 responses properly 🎉

More Info:

This investigation has been recorded to two separate live streams for posterity, as that's when I discovered the issue and the fix 😅

• 2024-06-09: https://youtube.com/live/EYQ7PSphDts
• 2024-06-10: https://youtube.com/live/1F-dXWx0Zxg

[Lukasa]

Great find! This looks like a correct fix, but we'll need some unit tests for this to ensure that we don't regress the functionality. Do you mind adding some? We have some existing no-body tests that could be extended for this use-case.

[dimitribouniol]

Compression test available here:

• HTTP Compression Test with 304 Not Modified Responses swift-nio-extras#224

[Lukasa]

@swift-server-bot add to allowlist

[Lukasa]

Really nice one @dimitribouniol, thanks!