Quantcast
Channel: All LabVIEW posts
Viewing all articles
Browse latest Browse all 202353

Re: Difference in data conversion when provided as a single integer or in an array

$
0
0

GerdW is right, generate your list, then right click and change your formatting to 20 significant digits instead of 6. Now it makes more sense. It's not the fact that it's an array, it's the fact that the number in the array isn't exactly 5.5- it's 5.49999999999999644.

 

If you make an array and manually fill it with values of 5.5 and 9.5, they work as it does with the single number.

 

sigfigs.png

 

 

 


Viewing all articles
Browse latest Browse all 202353

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>