Remove support for deprecated obs-fold in header values

[MihaZupan]

RFC7230 deprecated support for line-folding in header values (emphasis mine):

Historically, HTTP header field values could be extended over
multiple lines by preceding each extra line with at least one space
or horizontal tab (obs-fold). This specification deprecates such
line folding except within the message/http media type
(Section 8.3.1). A sender MUST NOT generate a message that includes
line folding (i.e., that has any field-value that contains a match to
the obs-fold rule) unless the message is intended for packaging
within the message/http media type.

Previously headers like foo: a\r\n b were considered valid (space following new line).

This PR removes obs-fold support from header validation, instead treating all new line characters as invalid.

On the receiving side, servers MAY accept obs-fold as long as they replace new lines with spaces.

A server that receives an obs-fold in a request message that is not
within a message/http container MUST either reject the message by
sending a 400 (Bad Request), preferably with a representation
explaining that obsolete line folding is unacceptable, or replace
each received obs-fold with one or more SP octets prior to
interpreting the field value or forwarding the message downstream.

Kestrel will treat the whole request as invalid and close the connection when receiving such a header.
HttpClient today does accept them in responses - this PR does not change that.

Fixes #50597

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

RFC7230 deprecated support for line-folding in header values (emphasis mine):

Historically, HTTP header field values could be extended over
multiple lines by preceding each extra line with at least one space
or horizontal tab (obs-fold). This specification deprecates such
line folding except within the message/http media type
(Section 8.3.1). A sender MUST NOT generate a message that includes
line folding (i.e., that has any field-value that contains a match to
the obs-fold rule) unless the message is intended for packaging
within the message/http media type.

Previously headers like foo: a\r\n b were considered valid (space following new line).

This PR removes obs-fold support from header validation, instead treating all new line characters as invalid.

On the receiving side, servers MAY accept obs-fold as long as they replace new lines with spaces.

A server that receives an obs-fold in a request message that is not
within a message/http container MUST either reject the message by
sending a 400 (Bad Request), preferably with a representation
explaining that obsolete line folding is unacceptable, or replace
each received obs-fold with one or more SP octets prior to
interpreting the field value or forwarding the message downstream.

Kestrel will treat the whole request as invalid and close the connection when receiving such a header.
HttpClient today does accept it - this PR does not change that.

Fixes #50597

Author:	MihaZupan
Assignees:	MihaZupan
Labels:	area-System.Net.Http
Milestone:	6.0.0

[geoffkizer]

What's the motivation for doing this?

I'm concerned that we will break some customer that is relying on this.

[geoffkizer]

Actually, I think this is missing some relevant changes.

HttpRuleParser.GetWhitespaceLength should be updated.

[MihaZupan]

Note: the change may seem huge, but it's 95% test changes

Product changes:

• Changed ContainsInvalidNewLine to ContainsNewLine
• Removed CRLF support from HttpRuleParser.GetWhitespaceLength
• Added CRLF check to HttpRuleParser.GetExpressionLength, NameValueHeaderValue's CheckValueFormat, HttpRequestHeaders.From* and AuthenticationHeaderValue's parameter

*after #52794, setting the From property would bypass new line checks (Add("From", "Foo") would still do the check).

[MihaZupan]

/azp run runtime

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[stephentoub]

Product changes LGTM, and I skimmed the tests. Do we have tests that validate all forms of newline are still allowed with TryAddWithoutValidation and NonValidated headers?

[MihaZupan]

Yes, I added a bunch of new-line combinations to this TryAddWithoutValidation/NonValidated test

runtime/src/libraries/System.Net.Http/tests/UnitTests/Headers/HttpHeadersTest.cs

Lines 332 to 334 in 3f3e304

	[Theory]
	[MemberData(nameof(HeaderValuesWithNewLines))]
	public void TryAddWithoutValidation_AddValueContainingNewLine_Rejected(string headerValue)

Comment has been addressed

[geoffkizer]

Do we have tests that validate all forms of newline are still allowed with TryAddWithoutValidation and NonValidated headers?

I'm looking at the tests you referenced @MihaZupan and it looks like all forms of newline are NOT allowed with TryAddWithoutValidation.

In particular it looks like the HeaderValuesWithNewLines enumerable is explicitly testing that we disallow these.

Am I missing something here? I thought the idea was to continue to allow obs-fold when using TryAddWithoutValidation but to restrict it elsewhere. Is this broken now?

[MihaZupan]

@geoffkizer We continue to do no validation with TryAddWithoutValidation. The test name is bad, but we are testing that new lines added without validation will be present when we enumerate without validation.

runtime/src/libraries/System.Net.Http/tests/UnitTests/Headers/HttpHeadersTest.cs

Line 344 in 3f3e304

Assert.Equal(headerValue, headers.NonValidated["foo"].ToString());

It is also testing that accessing the header with validation will result in its removal.

[geoffkizer]

It is also testing that accessing the header with validation will result in its removal.

Why is it removed if it is a valid header with embedded obs-fold?

[MihaZupan]

If you access a header with validation and we consider it as invalid, it will be removed from the collection. This has been the existing behavior.

The change is that now we treat any new lines as invalid - even if it would have been a valid obs fold.

In other words, you can only work with obs-folds if you add them without validation and read them without validation.

What behavior did you have in mind as desired here?