py/obj: Fix REPR_C bias toward zero.
Current implementation of REPR_C works by clearing the two lower bits of the mantissa to zero. As this happens after each floating point operation, this tends to bias floating point numbers towards zero, causing decimals like .9997 instead of rounded numbers. This is visible in test cases involving repeated computations, such as `tests/misc/rge_sm.py` for instance. The suggested fix fills in the missing bits by copying the previous two bits. Although this cannot recreate missing information, it fixes the bias by inserting plausible values for the lost bits, at a relatively low cost. Some float tests involving irrational numbers have to be softened in case of REPR_C, as the 30 bits are not always enough to fulfill the expectations of the original test, and the change may randomly affect the last digits. Such cases have been made explicit by testing for REPR_C or by adding a clear comment. The perf_test fft code was also missing a call to round() before casting a log_2 operation to int, which was causing a failure due to a last-decimal change. Signed-off-by: Yoctopuce dev <dev@yoctopuce.com>
This commit is contained in:
committed by
Damien George
parent
59ee59901b
commit
d6876e2273
4
py/obj.h
4
py/obj.h
@@ -206,6 +206,10 @@ static inline mp_float_t mp_obj_float_get(mp_const_obj_t o) {
|
||||
mp_float_t f;
|
||||
mp_uint_t u;
|
||||
} num = {.u = ((mp_uint_t)o - 0x80800000u) & ~3u};
|
||||
// Rather than always truncating toward zero, which creates a strong
|
||||
// bias, copy the two previous bits to fill in the two missing bits.
|
||||
// This appears to be a pretty good heuristic.
|
||||
num.u |= (num.u >> 2) & 3u;
|
||||
return num.f;
|
||||
}
|
||||
static inline mp_obj_t mp_obj_new_float(mp_float_t f) {
|
||||
|
||||
Reference in New Issue
Block a user