Nautical Terminator – Nautical Mile

Posted on December 2, 2019 by Tony Johnson

By Tradewinds instructor Tony Johnson

As we all learn in our first navigation class, a nautical mile equals one minute of latitude. Therefore, the distance in nautical miles between the equator and the poles is, by definition, 90 degrees times 60 minutes, or 5400 nautical miles, which is very close to 10000 kilometers. This seems pretty plain, until we come across the inconvenient fact that the earth is not a perfect sphere. It’s bloated at the equator, and so is what is called an oblate spheroid. It’s a bit squashed, in other words, which means that at the poles, the radius of the earth is 6356.752 kilometers, and at the equator, the radius is 6378.137 kilometers. After various mathematical analyses in the 19th century, an Englishman named Alexander Ross Clarke came up with the spheroid, referred to by scientists as the “figure of the earth,” which became the standard way of conceiving of this shape. The investigations continue to this day and get pretty arcane. WGS 84 (World Geodetic System of 1984) is the datum for the GPS units currently in use.

Here’s a funny result of this bulge: If you are standing at the equator, your weight is reduced by centripetal force. You’re being flung into space at 1000 miles an hour, just as you would be on a playground merry-go-round. However, the centripetal force is a very small percentage of the force of gravity, so you stay on the planet, but weigh just a little less. Or you would, except for that bloatedness. Because the world is thicker at the equator, there is more mass pulling you towards the center of the earth, increasing the earth’s gravity and mostly offsetting the opposing force throwing you away from the earth. The end result is that you weigh only about .5% less at the equator than you do at the poles.

That oblateness also messes with the nautical mile. If you take this shape into account, one minute of latitude at the equator is 1842.9 meters; but at the north pole it is 1861.7 meters. In 1929, the First International Extraordinary Hydrographic Conference in Monaco decided to standardize the distance at 1852 meters or 6077 feet, which we often round to 6080, as the length of the nautical mile. This compromise between the distance at the equator and the poles works out to be the length of the nautical mile at 48 degrees of latitude.

Related image
This entry was posted in Nautical Terminator and tagged . Bookmark the permalink.

10 Responses to Nautical Terminator – Nautical Mile

  1. Chuck B says:
    December 2, 2019 at 7:51 pm

    Hi Tony, thank you for this – super interesting and timely, too. I’d just recently heard the term “WGS 84” and had been meaning to look it up. It was in the context of some older paper charts NOT being WGS 84, which can be problematic if a position is plotted on such a chart using coordinates obtained from a GPS.

    Also, am I correct in believing that celestial navigation sight reduction tables are based on a perfect sphere? So there could be some small discrepancy between a celestial fix and GPS there, but presumably less than 20 meters (?), which doesn’t make much difference when one is happy to be within 5 nm. 🙂

  2. tony johnson says:
    December 3, 2019 at 4:55 am

    A very perceptive question. I think your analysis is correct but I don’t know enough of the math of the wgs 84 to give a decent answer. My guess is that the error would not be greater than the error most celestial navigators like me would get just from an inaccurate sight. But I’ve forwarded the question to a cartographer friend of mine who deals in this kind of thing and I’ll get back to you with his answer, assuming he has one.

  3. tony johnson says:
    December 3, 2019 at 5:19 am

    In regard to remark about older paper charts not corresponding to chartplotters, this is definitely the case in areas where large shipping never visits. The surveys to create the charts were made as much as a century or more ago, long before GPS 84, and used celestial navigation to locate positions. We found this a problem in Borneo and Phuket, where the chartplotter would put us on land. I think Fiji was problematic also. I’m positive there are other areas like this, often in exactly the regions adventurous cruises want to sail. I dealt with it by creating offsets by using traditional visual fixes, since the charts are quite accurate as long as GPS isn’t involved; I made a table of these and was able to construct waypoints that reflected the adjustment. We survived without touching, although that occurred elsewhere a couple of times for other reasons which I will defend as not indicative of negligence :).

  4. tony johnson says:
    December 3, 2019 at 10:58 am

    Here’s the answer from my friend:
    I don’t have time to properly research this right now but my gut reaction is, I seriously doubt it would make much difference. The inherent error of actual celestial navigation is far greater than the effect of oblation on the sphere. And in any case you are solving for lat and long which would sort of self resolve even as you get farther north. The longitude would not be affected, just the latitude and at the north latitudes you best be watching for icebergs way more than running aground.

    • Chuck B says:
      December 3, 2019 at 8:42 pm

      Fantastic, thank you Tony for your insights and for asking your friend. Makes sense. And if I ever see an iceberg, it means I made a seriously wrong turn… about two weeks earlier LOL.

      Maybe we’ll hear more about your non-negligent touching in a future post? 😉

  6. Chuck B says:
    December 3, 2019 at 11:07 pm

    Yikes!! The reef incident story is a serious nail-biter! That’s some crazy conditions to try to (re)anchor in as well. You must’ve had a lot of chain. Also, I have about as much love for rivers as I do for icebergs haha. Thanks for sharing. I shall be reading more of your reports.

Leave a Reply