Rendered at 23:39:57 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
Maxatar 1 hours ago [-]
I was skeptical about this article's claim about LLMs so I tried it out myself. Looks like there are several layers of pedantry involved, for example, strictly speaking in base C it's undefined behavior to divide by 0.0, even for floats, so the author's initial implementation is incorrect and a optimizer could elide the call to isfinite in certain scenarios where the denominator is 0.0.
Presumably the airtight implementation assuming Annex F is as follows:
#pragma STDC FENV_ACCESS ON
int my_div(float x, float y, float* r) {
if(!r) {
return -1;
}
fenv_t environment;
if(feholdexcept(&environment) != 0) {
return -1;
}
float result = x / y;
int failed = fetestexcept(
FE_INVALID | FE_DIVBYZERO | FE_OVERFLOW | FE_UNDERFLOW);
if(fesetenv(&environment) != 0 || failed != 0) {
return -1;
}
*r = result;
return 0;
}
When I asked about the author's implementation I got the response that the author's implementation is deficient in several areas:
- Underflow produces a finite result and passes this check.
- Overflow can produce a finite maximum value under some rounding modes.
- Quiet NaNs and valid infinite results fail this check, even without an arithmetic exception.
It also assumes floating-point traps are disabled. Annex F defines the arithmetic behavior; your function still needs a chosen definition of “failure.”
So the answer really is... safely dividing two floats really depends on what trade-offs you're willing to make, and the original reply of just checking for a 0 denominator is honestly the most sensible, portable, and safest.
weinzierl 9 hours ago [-]
The "integer arithmetic paradigm" would mean either checking after (almost) every operation or living with potentially incorrect results.
The first is terribly inefficient, the second is just wrong (even if insanely common).
This is sad because there is no reason at all why integers couldn’t follow what OP calls the "float paradigm". It doesn’t even have to be slow. Most modern processors (if we ignore x86) support some form of sticky arithmetic flags. Unfortunately, programming languages don’t support them, so they aren’t used.
There is also something to be said about
the special treatment division by zero gets.
>I sometimes wish that signed integers were symmetrical. i8 would represent the range of [-127 to 127] with 0xFF representing NaN. Any operation which can not be computed (division by zero, overflows, operation with another NaN, etc.) would result in NaN. For further symmetry we could do the same for signed integers as well.
dnautics 9 hours ago [-]
Pretty sure zig has safe integer operations: integers aren't allowed to overflow with the standard operators, there are overflowing and saturating operators if that's what you want.
weinzierl 2 hours ago [-]
Rust and D and probably others have this too. The interesting question is:
Can you chain these operations in a way that will propagate the error to the end result while simultaneously have a compiler produce code without branches after every operation. Safe integer operations alone are not enough for that.
dnautics 2 hours ago [-]
I think the operators panic. In zig, If you want safe operations I think there are functions in the stdlib
The branching does not cost as much as you think, I'm pretty sure the compiler marks the erroring path as cold, which the cpu can use to elide most of the branch cost through specex
weinzierl 1 hours ago [-]
A branch is a branch and no branch will always be better. With sticky flags there is no difference in performance to regular integer arithmetic.
kibwen 8 hours ago [-]
> Most modern processors (if we ignore x86) support some form of sticky arithmetic flags. Unfortunately, programming languages don’t support them, so they aren’t used.
I'd say the reason that programming languages don't have built-in support for checking the CPU's sticky status flags is precisely because of a lack of x86 support. Plenty of languages do have support for checking overflow on each individual operation, e.g. Rust's `overflowing_foo` methods, which return a tuple whose second member is a boolean indicating overflow: https://doc.rust-lang.org/std/primitive.i32.html#method.over...
jcranmer 7 hours ago [-]
Sticky flags tend to be really annoying for a compiler to support for various reasons, but principally it makes every operation have a hidden dependency to shared global state that is almost never read. Note that IEEE 754 standardized support for sticky flags 40 years ago, but support for these sticky bits is still poor to nonexistent in most programming languages, and sticky bit stuff for floating-point operations is less problematic than integer operations because FP math is already far more black box in practice.
a_e_k 7 hours ago [-]
For the float stuff, beware of `-ffinite-math-only`. If enabled, `isfinite()` and may compile out as assumed true (with similar assumptions around `isnan()` and `isinf()`).
And `-ffinite-math-only` is enabled by `-ffast-math` which in turn is enabled by `-Ofast`.
toolslive 6 hours ago [-]
Not only that: `-ffast-math` might change the result of flops when the arguments are regular floats too. (It abandons IEEE754 compliance)
dooglius 9 hours ago [-]
FWIW one can configure floating point exceptions at runtime at no overhead, see `man 3 fenv`
jcranmer 7 hours ago [-]
It's no overhead in the same sense that zero-cost exception-handling is zero-cost: turning them on properly (e.g., #pragma STDC FENV_ACCESS ON or equivalent command-line flags) disables a fair amount of optimizations which incurs an overhead in and of itself.
dooglius 5 hours ago [-]
Oh interesting I thought it was more or less a wrapper around MXCSR (or equivalent for other architectures)
dzaima 7 hours ago [-]
Except gcc doesn't actually support it to any sane extent (it ignores and warns on "#pragma STDC FENV_ACCESS ON", which C requires for fenv.h to actually function; and as such gcc makes a bunch invalid optimizations); clang supports it, but only as of somewhat-recently.
Presumably the airtight implementation assuming Annex F is as follows:
When I asked about the author's implementation I got the response that the author's implementation is deficient in several areas: So the answer really is... safely dividing two floats really depends on what trade-offs you're willing to make, and the original reply of just checking for a 0 denominator is honestly the most sensible, portable, and safest.The first is terribly inefficient, the second is just wrong (even if insanely common).
This is sad because there is no reason at all why integers couldn’t follow what OP calls the "float paradigm". It doesn’t even have to be slow. Most modern processors (if we ignore x86) support some form of sticky arithmetic flags. Unfortunately, programming languages don’t support them, so they aren’t used.
There is also something to be said about the special treatment division by zero gets.
>I sometimes wish that signed integers were symmetrical. i8 would represent the range of [-127 to 127] with 0xFF representing NaN. Any operation which can not be computed (division by zero, overflows, operation with another NaN, etc.) would result in NaN. For further symmetry we could do the same for signed integers as well.
Can you chain these operations in a way that will propagate the error to the end result while simultaneously have a compiler produce code without branches after every operation. Safe integer operations alone are not enough for that.
The branching does not cost as much as you think, I'm pretty sure the compiler marks the erroring path as cold, which the cpu can use to elide most of the branch cost through specex
I'd say the reason that programming languages don't have built-in support for checking the CPU's sticky status flags is precisely because of a lack of x86 support. Plenty of languages do have support for checking overflow on each individual operation, e.g. Rust's `overflowing_foo` methods, which return a tuple whose second member is a boolean indicating overflow: https://doc.rust-lang.org/std/primitive.i32.html#method.over...
And `-ffinite-math-only` is enabled by `-ffast-math` which in turn is enabled by `-Ofast`.
> These eleven functions were defined in C99, and describe the handling of floating-point rounding and exceptions (overflow, zero-divide, etc.).