Implement c++20 std::chrono::duration subsecond formatting by matrackif · Pull Request #2623 · fmtlib/fmt

matrackif · 2021-11-28T21:27:39Z

Hello,

I have returned with a solution to #2207, after dealing with licensing issues in PR #2554. This time I am 100% the author of the code. The solution is simplified, and could be adapted to use if-constexpr if C++17 is detected at compile-time. In fact, the solution I have proposed is working better than the current MSVC implementation which leads to template overflow when std::atto is used as a precision. I will open an issue there.

Subsecond values that are not representable in std::intmax_t (the standard type for chrono tick counts) cause program termination with FMT_ASSERT(). The standard requires us to support up to 18 decimal digits, and std::int64_t can store a maximum value of 9,223,372,036,854,775,807 or, 9.2e+18,. Therefore it can represent all 18 digit numbers. However the following assert test had to be removed:

(void)fmt::format("{:%S}",
                    std::chrono::duration<float, std::atto>(1.79400457e+31f));

The decimal point is still not formatted according to the locale but that is a general problem.

…g to the c++20 standard

* Allow proper Duration::rep type to propagate via template argument deduction

matrackif · 2021-11-29T01:00:25Z

@vitaut Hi could you rerun the CI tests.

Thanks

…um_digits()

matrackif · 2021-12-04T16:52:44Z

@vitaut I locally used -Wconversion instead of -Wsign-converison so line of code still had to be fixed. If you could run the workflow one more time it would be much appreciated.

Thanks

vitaut

Thanks for the PR!

vitaut · 2021-12-04T16:46:27Z

+  }
+  using nanoseconds_dbl = std::chrono::duration<double, std::nano>;
+  EXPECT_EQ(fmt::format("{:%S}", nanoseconds_dbl{-123456789.123456789}),
+            "-00.123456789");


Why is the integral part 00 here?

According to the spec:

If the number of seconds is less than 10, the result is prefixed with 0

The number of seconds is 0, which is less than 10, hence we write 00.

The MSVC implementation of std::format also interprets the standard this way. The check below passes locally in my instance of Visual Studio 2022

std::format("{:%S}", std::chrono::duration<double, std::nano{-123456789.123456789}) == std::string("-00.123456789")

Oh, I misread the test case and thought that the integral part 123456789. is seconds rather than nanoseconds. I suggest changing the fractional part to something different to prevent confusion.

vitaut · 2021-12-04T16:47:09Z


+TEST(chrono_test, cpp20_duration_subsecond_support) {
+  using attoseconds = std::chrono::duration<std::intmax_t, std::atto>;
+  // Check that 18 digits of subsecond precision are supported


nit: Please add punctuation (. in the end of the sentence) here and elsewhere.

vitaut · 2021-12-04T16:48:55Z

+    EXPECT_EQ(fmt::format("{:%H:%M:%S}", dur), formatted_dur);
+  }
+  // Check that durations with precision greater than std::chrono::seconds have
+  // fixed precision and empty zeros


"empty zeros"? I suggest rephrasing this as "zero fractional part" or smth like that.

vitaut · 2021-12-04T16:50:34Z

+      "40",
+      fmt::format("{:%S}", std::chrono::duration<double>(1e20)).substr(0, 2));


Why change this?

According to the spec, the decimal point should be printed only if the duration "period" type is more precise than seconds:

If the precision of the input cannot be exactly represented with seconds, then the format is a decimal floating-point number with a fixed format and a precision matching that of the precision of the input

This can be determined statically at compile time and does not depend on runtime values of the duration. The test I modified has seconds precision, so I removed the decimal point.

In my local instance of Visual Studio 2022, the following test passes:

std::format("{:%S}", std::chrono::duration<double>{1.234}) == std::string("01")

Shall we check that there is no decimal point then?

vitaut · 2021-12-04T17:06:49Z

  }

-  void write(Rep value, int width) {
+  template <typename RepType> void write(RepType value, int width) {


Why introduce a template parameter instead of using Rep?

Rep is currently assumed to be an unsigned type. I wanted the proper type to propagate through the code and be deduced in the template. Otherwise there could be an unsigned to signed conversion here which would lead to a -Wconversion error.

Rep is currently assumed to be an unsigned type.

I don't think this is correct. Rep is the representation type of a duration and it can be signed (in fact it's normally signed).

Sorry, lowercase rep is used to store the internal duration value, and rep is unsigned to avoid overflow. At first I wanted to use the existing logic in the write() function to format a duration value of type lowercase rep. This is irrelevant now as I reverted this function to its previous state

vitaut · 2021-12-04T17:12:01Z

+    uint32_or_64_or_128_t<std::intmax_t> n =
+        to_unsigned(to_nonnegative_int(value, max_value<std::intmax_t>()));


Let's use long long instead of intmax_t because it's a built-in type and can also handle 18 digits of precision.

As you wish, std::chrono::duration types use std::intmax_t internally that's why I went along with it.

vitaut · 2021-12-04T17:15:58Z

@@ -1319,20 +1319,20 @@ inline bool isfinite(T) {
 }

 // Converts value to int and checks that it's in the range [0, upper).


vitaut · 2021-12-04T17:21:08Z

-                1000);
-  return std::chrono::duration<Rep, std::milli>(static_cast<Rep>(ms));
-}
+template <class Duration> class subsecond_helper {


Having a utility struct seems unnecessary. I suggest converting it into a function (e.g. write_fractional_seconds) or merging the logic into write where it is used.

Co-authored-by: Victor Zverovich <victor.zverovich@gmail.com>

…ubsecond_helper class with a function

…th name write_fractional_seconds() * Revert write(Rep value, int width) function to previous state

…write_fractional_seconds()

matrackif · 2021-12-06T12:24:05Z

@vitaut I have implemented the suggestions you have requested. If you have any further remarks I would gladly hear them. The workflows still need to be run :)

vitaut · 2021-12-06T16:17:36Z

+  // No decimal point is printed so size() is 2.
+  EXPECT_EQ(value.size(), 2);
+  EXPECT_EQ("40", value.substr(0, 2));


Let's simplify this:

EXPECT_EQ(value, "40");

Then we don't need a comment because it's clear that we don't emit a decimal point.

Of course. I recall the size of this string being platform dependent but after the changes in this PR it should always be consistent. Done

vitaut

A few more, mostly minor comments.

vitaut · 2021-12-09T00:38:59Z

-template <typename Rep, typename Period,
-          FMT_ENABLE_IF(std::is_floating_point<Rep>::value)>
-inline std::chrono::duration<Rep, std::milli> get_milliseconds(
+// Returns the amount of digits according to the c++ 20 spec


nit: amount -> number

vitaut · 2021-12-09T00:39:31Z

+// Returns the amount of digits according to the c++ 20 spec
+// In the range [0, 18], if more than 18 fractional digits are required,
+// then we return 6 for microseconds precision.
+static constexpr int num_digits(long long num, long long den, int n = 0) {


I don't think static is needed here (and similarly in a functions below).

Good point, static made sense when it was a member function, now it is no longer is needed.

vitaut · 2021-12-09T00:42:48Z

-inline std::chrono::duration<Rep, std::milli> get_milliseconds(
+// Returns the amount of digits according to the c++ 20 spec
+// In the range [0, 18], if more than 18 fractional digits are required,
+// then we return 6 for microseconds precision.


Is returning 6 actually required by the standard? Can we turn it into a compile-time error instead?

From the standard:

...and a precision matching that of the precision of the input (or to a microseconds precision if the conversion to floating-point decimal seconds cannot be made within 18 fractional digits)

Microseconds precision is 6 digits. The MSVC implementation of std::format also behaves this way.

vitaut · 2021-12-09T00:45:07Z

  }

+  template <class Duration> void write_fractional_seconds(Duration d) {
+    static constexpr auto fractional_width =


This doesn't need to be static because it's just an int.

Good point, lightweight integral types probably do not need to be static, even if constexpr

* Remove static from detail functions, they are no longer member functions of a class and static is unnecessary. * Change comment from "amount" to "number"

vitaut · 2021-12-09T14:38:11Z

+
+template <class Rep, class Period,
+          FMT_ENABLE_IF(!std::numeric_limits<Rep>::is_signed)>
+static constexpr std::chrono::duration<Rep, Period> abs(


One more static to remove here.

Removed in a follow-up commit.

vitaut · 2021-12-09T15:00:12Z

Merged, thanks!

vitaut · 2021-12-22T23:09:37Z

A fuzzer has discovered a UB which seems to be introduced by this PR:

fmt::print("{:%S}", std::chrono::duration<long double, std::ratio<1, 1000000> >(-1.7591538808701083784E+4865L));

% c++ test.cc -std=c++11 -I include src/format.cc -fsanitize=undefined
% ./a.out
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/chrono:906:28: runtime error: 1.75915e+4859 is outside the range of representable values of type 'long long'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior

matrackif · 2021-12-23T00:54:58Z

Subseconds are always an integral value no longer than 18 digits. The implementation will cast all values to a long long. If the value is not representable by a long long like this huge double UBsan will report this error.

It is not clear how to handle values not representable by intmax_t/long long, the spec doesn't seem to mention what to do. I guess it's up to the implementation how to truncate such values.

vitaut · 2021-12-23T01:06:27Z

I think we should add a range check when converting from an FP number.

matrackif · 2021-12-24T15:47:06Z

And throw an exception if the value is out of range? What method do you propose for range checking floats. I assume such checks exist already in the codebase

vitaut · 2021-12-24T17:13:45Z

I implemented a fix for the UB in 784e2a7.

matrackif · 2022-01-07T17:40:11Z

I implemented a fix for the UB in 784e2a7.

This commit does not work. I believe it can be safely reverted because currently the above example throws an exception "invalid value". If exceptions are disabled then we need an alternative approach.

fmt::print("{:%S}\n", std::chrono::duration<double, std::milli >(1.0)); correctly prints 00.001

Whereas if we change the type to long double then it incorrectly prints 00.000

The offending line is here

detail::abs(d) - std::chrono::duration_cast<sec>(d) where sec is std::chrono::duration<long double> will subtract away the subsecond part which we want to keep. The integer cast to std::chrono::seconds was intentional in the implementation exactly for this reason.

Link to a godbolt example

Also the commit addresses only long double, but doesn't the undefined behavior problem exist with other floating point types such as float, and double?

vitaut · 2022-01-07T18:29:42Z

Could you provide a PR with a better workaround for the UB?

matrackif · 2022-01-07T21:47:07Z

At first I thought of the following solution:

Check if the duration type is floating_point and if it is greater than max_value<long long>

a. If yes, then cast it to its seconds representation, and truncate it. This way we don't subtract away the valuable subsecond part of the number. Something like this:

detail::abs(d) - Duration{std::trunc(std::chrono::duration_cast<std::chrono::duration<typename Duration::rep>>(d).count())}

b. If not, then cast to std::chrono::seconds as before and call it a day.

But then I realized that in practice such code will never be reached because

uint32_or_64_or_128_t<long long> n = to_unsigned(to_nonnegative_int(subseconds, max_value<long long>()));

Will thrown an exception if the value is larger than max_value<long long> anyways. The simple code suddenly becomes much more complicated if we want to handle floating point values in the range: (max_value<long long>, max_value<long double>]

A possible workaround would be to avoid using detail::count_digits() to count the number of digits for floating point types and simply delegate the work to the fmt floating point formatter and specifying precision to be equal to num_fractional_digits:

Any thoughts?

vitaut · 2022-01-07T22:15:43Z

A possible workaround would be to avoid using detail::count_digits() to count the number of digits for floating point types and simply delegate the work to the fmt floating point formatter and specifying precision to be equal to num_fractional_digits

This sounds reasonable. We can always optimize this path later and correctness is more important.

matrackif · 2022-01-08T00:45:30Z

Ok, I am trying to format floating point subsecond values using the floating point formatter, but as always there are issues :).

If the duration type is floating point, then we do our calculations using floating point arithmetic. Let's say we calculate the subseconds remaining to be std::chrono::duration<double, std::micro>(123.456). We call:

format_to(out, "{:{}.{}f}", subseconds, num_fractional_digits, num_fractional_digits);, but this yields 123.456000 which is not what we want. We want the result to be simply 123, because we don't care about the fractional part of the subseconds themselves. After calling std::trunc() on the value we obtain 123.000000, still doesn't help because all std::trunc() did was zero out the fractional part.

So we could cast to an integral type to remove the fractional part, but then we end up with the same problem that we started with, since some floating point values are not representable by integral types and we get UB.

In other words, how can we remove the fractional part of floating point numbers without casting to an integral type? Or how can we format a FP number without its fractional part (and decimal point), using existing facilities?

matrackif · 2022-01-08T00:51:24Z

The easy way out would be to simply throw an exception if the formatted value is larger than max_value<long long>. Then the code would technically be correct and there would be no UB, the downside is that floating point values larger than max_value<long long> would not be supported by the facility.

vitaut · 2022-01-09T23:21:42Z

I think we should format integral and fractional parts together. I took a stab at this as part of fixing another UB in d9f045f but improvements are welcome.

matrackif · 2022-01-10T16:32:09Z

I am a fan of this approach. We simply format the duration using the floating point formatting facility, otherwise we use the integral implementation.

matrackif added 4 commits November 28, 2021 22:00

Add support for subsecond printing for std::chrono::duration accordin…

c379573

…g to the c++20 standard

Remove assert test that overflows intmax_t

57d35cc

* Hopefully fix int64_t to int32_t conversion errors.

863d2ff

* Allow proper Duration::rep type to propagate via template argument deduction

* Hopefully fix int64_t to int32_t conversion errors.

beacfd3

* Allow proper Duration::rep type to propagate via template argument deduction

matrackif added 2 commits December 4, 2021 17:46

Fix sign conversion (-Wsign-conversion) warning treated as error in n…

b8e3e33

…um_digits()

Format chrono.h with clang-format

5619e87

vitaut requested changes Dec 4, 2021

View reviewed changes

matrackif and others added 5 commits December 4, 2021 19:41

Remove extra forward slash in doxygen style comment

7544467

Co-authored-by: Victor Zverovich <victor.zverovich@gmail.com>

Apply all suggestions from GitHub, except for replacing the utility s…

3d860b7

…ubsecond_helper class with a function

* Move logic of handling subseconds from utility class to function wi…

0fcdb01

…th name write_fractional_seconds() * Revert write(Rep value, int width) function to previous state

Fix -Wshadow warning

55f62cd

Remove unsued get_subseconds() function, its logic has been moved to …

a0abbed

…write_fractional_seconds()

vitaut reviewed Dec 6, 2021

View reviewed changes

Comment thread include/fmt/chrono.h

matrackif added 2 commits December 6, 2021 17:58

Change comment from lowercase int to uppercase Int

38fb7a6

Simplify test check

89c8198

vitaut requested changes Dec 9, 2021

View reviewed changes

Integrate suggested changes

9344c47

* Remove static from detail functions, they are no longer member functions of a class and static is unnecessary. * Change comment from "amount" to "number"

vitaut reviewed Dec 9, 2021

View reviewed changes

vitaut merged commit 0bbc970 into fmtlib:master Dec 9, 2021

vitaut mentioned this pull request Dec 9, 2021

Printed seconds precision does not match C++20 spec #2207

Closed

phprus referenced this pull request Dec 26, 2021

Update changelog

3110ec5

		"40",
		fmt::format("{:%S}", std::chrono::duration<double>(1e20)).substr(0, 2));

		uint32_or_64_or_128_t<std::intmax_t> n =
		to_unsigned(to_nonnegative_int(value, max_value<std::intmax_t>()));

		@@ -1319,20 +1319,20 @@ inline bool isfinite(T) {
		}

		// Converts value to int and checks that it's in the range [0, upper).

Uh oh!

Conversation

matrackif commented Nov 28, 2021

Uh oh!

matrackif commented Nov 29, 2021

Uh oh!

matrackif commented Dec 4, 2021

Uh oh!

vitaut left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

matrackif commented Dec 6, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vitaut left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

matrackif commented Jan 7, 2022 •

edited

Loading

matrackif commented Jan 7, 2022 •

edited

Loading

matrackif commented Jan 8, 2022 •

edited

Loading

vitaut commented Jan 9, 2022 •

edited

Loading