Some DateTimeTest format tests fail on the upcoming Fedora 39

[tmds]

System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"f\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"F\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"g\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"G\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"t\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"T\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"U\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"f\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"F\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"g\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"G\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"t\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"T\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"U\")
System.Tests.DateTimeOffsetTests.TryFormat_ToString_EqualResults
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"f\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"F\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"g\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"G\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"t\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"T\")

It's likely there was a regression in #83589 which was not caught in CI.

The test failures are from our CI machine. I will reproduce the issue locally and share more information.

cc @tarekgh @omajid

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

---

System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"f\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"F\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"g\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"G\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"t\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"T\")
System.Tests.DateTimeTests.TryFormat_MatchesToString(format: \"U\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"f\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"F\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"g\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"G\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"t\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"T\")
System.Tests.DateTimeTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"U\")
System.Tests.DateTimeOffsetTests.TryFormat_ToString_EqualResults
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"f\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"F\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"g\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"G\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"t\")
System.Tests.DateTimeOffsetTests.ParseExact_ToStringThenParseExactRoundtrip_Success(standardFormat: \"T\")

It's likely there was a regression in #83589 which was not caught in CI.

The test failures are from our CI machine. I will reproduce the issue locally and share more information.

cc @tarekgh @omajid

Author:	tmds
Assignees:	-
Labels:	area-System.Globalization, untriaged, needs-area-label
Milestone:	-

[tarekgh]

The problem is in DateTime.TryFormat with Utf-8. This assumes all date/time pattern characters are ascii which is not the case as CLDR started to use U+202F as the space character. The following line should convert the character to Utf-8 and store it to the buffer instead of just casting it.

runtime/src/libraries/System.Private.CoreLib/src/System/Globalization/DateTimeFormat.cs

Line 730 in 9aefa9d

result.Append(TChar.CastFrom(ch));

[tmds]

I had tried to reproduce the issue in a Fedora 38, and a Fedora 39 container yesterday and to my surprise it didn't reproduce.

The problem is in DateTime.TryFormat with Utf-8. This assumes all date/time pattern characters are ascii which is not the case as CLDR started to use U+202F as the space character. The following line should convert the character to Utf-8 and store it to the buffer instead of just casting it.

Thanks for pointing me to the problem.
This matches with the failures: they show / (0x2F) which is the lower byte of U+202F:

Expected: ···ril 18, 2023 5:10:51 AM
Actual:   ···ril 18, 2023 5:10:51/AM

[tarekgh]

I had tried to reproduce the issue in a Fedora 38, and a Fedora 39 container yesterday and to my surprise it didn't reproduce.

I am able to produce the issue on edora 39. I just used Fedora:39 docker image

[tmds]

I am able to produce the issue on Fedora 39. I just used Fedora:39 docker image

Let me try again.

I'm using the image from registry.fedoraproject.org/fedora , which has a sha ae8f1aaa5e2b for Fedora 39.
Are we using the same one?

[tarekgh]

I just did docker pull fedora:39 and then manually added all needed dependencies to build the runtime repo.

[tmds]

We're using a different image:

registry.fedoraproject.org/fedora            39     ae8f1aaa5e2b  33 hours ago  197 MB
docker.io/library/fedora                     39     9526ca27af97  5 weeks ago   196 MB

I'm going to try building in the Docker Hub image, and see if that reproduces the issue for me as well.