Confusing behavior of Uri.IsWellFormedUriString in .Net Core and .Net Framework

[t-bzhan]

The behavior of Uri.IsWellFormedUriString is very confusing. Below code snippets:

            List<string> absoluteUrls = new List<string>()
            {
                "https://api.contoso.com/query?q=世界&accessKey=acd%3D",
                "https://api.contoso.com/query?q=%E4%B8%96%E7%95%8C&accessKey=acd%3D",
                "https://api.contoso.com/query?q=世界&accessKey=acd=",
                "https://api.contoso.com/query?q=world&accessKey=acd%3D"
            };

            foreach (var url in absoluteUrls)
            {
                Console.WriteLine($"{url}: {Uri.IsWellFormedUriString(url, UriKind.Absolute)}");
            }

            List<string> relativeUrls = new List<string>()
            {
                "/query?q=世界&accessKey=acd%3D",
                "/query?q=%E4%B8%96%E7%95%8C&accessKey=acd%3D",
                "/query?q=世界&accessKey=acd=",
                "/query?q=world&accessKey=acd%3D"
            };

            foreach (var url in relativeUrls)
            {
                Console.WriteLine($"{url}: {Uri.IsWellFormedUriString(url, UriKind.Relative)}");
            }

will output following results in both .Net core 3.1 and .Net framework 4.7.2:

URL	Result
https://api.contoso.com/query?q=世界&accessKey=acd%3D	False
https://api.contoso.com/query?q=%E4%B8%96%E7%95%8C&accessKey=acd%3D	False
https://api.contoso.com/query?q=世界&accessKey=acd=	True
https://api.contoso.com/query?q=world&accessKey=acd%3D	True
/query?q=世界&accessKey=acd%3D	False
/query?q=%E4%B8%96%E7%95%8C&accessKey=acd%3D	True
/query?q=世界&accessKey=acd=	False
/query?q=world&accessKey=acd%3D	True

The "False" result of https://api.contoso.com/query?q=世界&accessKey=acd%3D gives me the impression that Uri.IsWellFormedUriString is not able to handle percent-encodings of reserved characters defined in RFC 3986, but I could not explain why https://api.contoso.com/query?q=world&accessKey=acd%3D return "True"

I also found that there are inconsistencies between the parsing result of UriKind.Absolute and UriKind.Relative for the same URI. /query?q=%E4%B8%96%E7%95%8C&accessKey=acd%3D return "True" while https://api.contoso.com/query?q=%E4%B8%96%E7%95%8C&accessKey=acd%3D return "False".

Tagging subscribers to this area: @dotnet/ncl
Notify danmosemsft if you want to be subscribed.

[MihaZupan]

Related: #34031
Also considering this a bug from the perspective that IsWellFormed doesn't agree with AbsoluteUri/TryCreate

[MatthewDavidCampbell]

Using SDK version 3.1.8. Similar problem to this issue (as well a Blazor issue) where spaces (both before and | or after) with other non-English characters in string are not a well formed Uri when the Uri is absolute:

// Arrange
var url = "https://bioutvstanfordstorage.blob.core.windows.net/viol2/blobs/fpr/Krön .fpr.gzip";

// Act
var flag = Uri.IsWellFormedUriString(Uri.EscapeUriString(url), UriKind.Absolute);

// Assert
Assert.False(flag); 

Prefix the url with a slash and change the UriKind flag to Relative | RelativeOrAbsolute and the assertion is valid:

// Arrange
var url = "/https://bioutvstanfordstorage.blob.core.windows.net/viol2/blobs/fpr/Krön .fpr.gzip";

// Act
var flag = Uri.IsWellFormedUriString(Uri.EscapeUriString(url), UriKind.Relative);

// Assert
Assert.True(flag); 

Status on this issue?

[uabarahona]

The wording of IsWellFormedUriString indicates ...ensures that the string does not require further escaping., based on that I believe to get a true from https://api.contoso.com/query?q=世界&accessKey=acd%3D you will need to scape it:

string url = "https://api.contoso.com/query?q=世界&accessKey=acd%3D";
Uri.IsWellFormedUriString(Uri.EscapeUriString(url), UriKind.Absolute);

Now from what I have investigated, the issue is possible due the validation that is being done against _string instead of OriginalString

runtime/src/libraries/System.Private.Uri/src/System/UriExt.cs

Lines 392 to 399 in 40b8e3d

	internal unsafe bool InternalIsWellFormedOriginalString()
	{
	if (UserDrivenParsing)
	throw new InvalidOperationException(SR.Format(SR.net_uri_UserDrivenParsing, this.GetType()));
	fixed (char* str = _string)
	{
	int idx = 0;

If you see how OriginalString is calculated, you will see that is not as simple as taking the _string

runtime/src/libraries/System.Private.Uri/src/System/Uri.cs

Lines 1119 to 1121 in 40b8e3d

	//
	public string OriginalString => _originalUnicodeString ?? _string;

Details

When you have similar Relative and Absolute URIs

string url = "/a?q=世界 a";
Uri.IsWellFormedUriString(Uri.EscapeUriString(url), UriKind.Relative);

string url = "https://api.contoso.com/a?q=世界 a";
Uri.IsWellFormedUriString(Uri.EscapeUriString(url), UriKind.Absolute);

⚠ Notice I am using Uri.EscapeUriString(url)

Then if you inspect URI internals you found:

	Relative (value)	Absolute(value)
_string	"/a?q=%E4%B8%96%E7%95%8C%20a"	"https://api.contoso.com/a?q=世界 a"
OriginalString	"/a?q=%E4%B8%96%E7%95%8C%20a"	"https://api.contoso.com/a?q=%E4%B8%96%E7%95%8C%20a"

As you can see OriginalString is the only value that is consistent in both Relative and Absolute, so I believe the validations should be done against it

[captainsafia]

@barahonajmc Thanks for taking the time to dig into the code here and report back. It does seem like a bug that InternalIsWellFormedOriginalString computes on _string instead of OriginalString. Mind cooking up a PR for one of the collaborators here to review?

[uabarahona]

@captainsafia sure I am going to prepare a PR, thanks for review it

[uabarahona]

Apparently, using _string instead of OriginalString is not the issue, at least not the main one, looks like absolute URI and relative URI have different ways to calculate if they are well formed or not:

For absolute URI

• The validation depends on the parse process
• The user provided string is being unescaped here and here
• An important calculation happens "here" against the mutated (unescaped) string
• ⭐ Conclusion: The calculation of Uri.IsWellFormedUriString for absolute uri happens against the "Flags" calculated on the parse process which are calculated based on the mutated(unescaped) string

For relative URI

• The validation does not depend on the parse process
• The user provided string is never scaped
• The important part of the validations happens here against the user provided string
• ⭐ Conclusion: The calculation of Uri.IsWellFormedUriString for relative uri happens against the user provided string, no Unescape happens.

Extra

Based on the explanation of the method [...ensures that the string does not require further escaping.] and the implementation of it I believe:

ℹ Uri.IsWellFormedUriString does not say you wheter URL is valid or not but instead if it needs to be escaped (false) or not (true). If we need to validate a URL then we should use TryCreate

//For relative
bool isValid = Uri.TryCreate(url, UriKind.Relative, out Uri result);

//For absolute we can add additional checks
bool isValid = Uri.TryCreate(url, UriKind.Absolute, out Uri result) 
                && (result.Scheme == Uri.UriSchemeHttp || result.Scheme == Uri.UriSchemeHttps);

With it all above examples are true

URL	Result
https://api.contoso.com/query?q=世界&accessKey=acd%3D	true
https://api.contoso.com/query?q=%E4%B8%96%E7%95%8C&accessKey=acd%3D	true
https://api.contoso.com/query?q=世界&accessKey=acd=	true
https://api.contoso.com/query?q=world&accessKey=acd%3D	true
/query?q=世界&accessKey=acd%3D	true
/query?q=%E4%B8%96%E7%95%8C&accessKey=acd%3D	true
/query?q=世界&accessKey=acd=	true
/query?q=world&accessKey=acd%3D	true

[MihaZupan]

Closing in favour of #72632