Quantcast
Channel: All LabVIEW posts
Viewing all articles
Browse latest Browse all 202843

Re: Difference in data conversion when provided as a single integer or in an array

$
0
0

 wrote:

Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?


Set your display format on the double array to 20 digits of precision and you'll see that 5.5 is not actually 5.50000000000....... but 5.4999999...644.  It is less than 5.5 so it will round down.


Viewing all articles
Browse latest Browse all 202843

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>