Quantcast
Channel: All LabVIEW posts
Viewing all articles
Browse latest Browse all 203131

Re: Difference in data conversion when provided as a single integer or in an array

$
0
0

Hi yp,

 

an additional explanation is found in the definition of IEEE754 float format.

 

Adding 0.1 repeatedly will not give expected results as that value cannot be represented by float values precisely!


Viewing all articles
Browse latest Browse all 203131

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>