Since the seconds cannot exceed 59.9999, even a test whether the ss portion of mm.ss exceeds 0.6 may be sufficient. This will cause a slight error (1E–10 seconds or 3 E–15 degrees) in the final result.Ī better solution may check if the mm.ss value is very close to the next higher integer, i.e. For a quick and dirty solution, simply add a tiny amount to the ss value in the first line of code, say, 1E-12 if 15 digits are used and the angle is below 1000°. There are several ways to overcome the problem. I translated your Fortran code to Excel VBA, and indeed the same problems occured. seconds, to that the final result is 30.5111. This again is interpreted as 30 degrees, 29 minutes and 99.99999. For instance, the value "30.3" here obviously has a binary value that translates to 30.29999999999. This is caused by the inexact binary representation of decimal numbers. (03-27-2016 03:02 PM)HP67 Wrote: I'm getting some unacceptable results from rounding in old FORTRAN. I find this kind of problem is coming up a lot in various calculations I'm doing and I don't know how to code around it.Ĭan anybody please explain how I can rearrange the computations or do something differently to avoid this problem? The code winds up with 29 instead of 30 because of the truncation function. You can see in test case #3 above there is a big problem extracting the minutes.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |