Skip to content

Fix x**y, x.power(y, 0) and x.sqrt(0) calculates huge digits if precision limit is huge - #445

Merged
tompng merged 1 commit into
ruby:masterfrom
tompng:fix_power_limit
Oct 16, 2025
Merged

tompng merged 1 commit into
ruby:masterfrom
tompng:fix_power_limit

Conversation

@tompng

@tompng tompng commented Oct 15, 2025 •

Copy link
Copy Markdown
Member

Fix this bug

BigDecimal.limit 1000
BigDecimal(2) / 3
#=> 0.66666666666666666666666666666667e0
BigDecimal(0.111111111).sqrt(0)
#=> 0.3333333331666666666249999999791666666536...(1000 digits, too long)
BigDecimal(2) ** 1.2
#=> 0.229739670999407001359725389355585517888...(1000 digits, too long)

Limit is just a limit. It's just a maximum precision restriction when precision is omitted. It's not a default precision value.

Expected behavior is:

# When `limit < auto_determined_precision`
BigDecimal.limit 10
BigDecimal(2) / 3
#=> 0.6666666667e0
BigDecimal(0.111111111).sqrt(0)
#=> 0.3333333332e0
BigDecimal(2) ** 1.2
#=> 0.229739671e1

# When no limit or `limit > auto_determined_precision`
BigDecimal.limit 0
BigDecimal(2) / 3
#=> 0.66666666666666666666666666666667e0
BigDecimal(0.111111111).sqrt(0)
#=> 0.333333333166666666625e0
BigDecimal(2) ** 1.2
#=> 0.22973967099940700135972538935559e1

@tompng
tompng merged commit 4626b7f into ruby:master Oct 16, 2025
81 checks passed
@tompng
tompng deleted the fix_power_limit branch October 16, 2025 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant