14 DAY TRIAL // Most people figure a day is a day, it's how long it takes for the Earth to spin around its axis. The same goes for the year. But things aren't so clear cut, not the least of which because the speed with which the Earth is hurtling around the Sun and around itself changes in time.
Everyone knows about leap years, most people even know what they mean and why they're needed. A lot less people know about leap seconds. Programmers it seems, are among those that don't.
In order to keep time from slipping away, quite literally, a leap second was added to June 30. At midnight, UTC time, the clock showed 23:59:60. That may look confusing for humans, but the human brain thrives with vague and approximate data and "computations." Computers, not so much.
The problem is that leap seconds can't be predicted and are added as they are required, due to the Earth's unpredictable rotation speed.
As such, designing computer systems that take them into account is hard. What's more, computers and programmers alike cherish simplicity and its beauty, for example, a minute having 60 seconds always, not 61 seconds one in a random while.
Among the sites that were affected by the leap second during the weekend are big names like Mozilla, Wikipedia, reddit and others.
The problem seems to trace back, in all cases, to Java, which couldn't handle the leap second. Software written in Java behaved erratically or stopped working at all.
In Reddit's case it was the Cassandra database, built on Java and created by Facebook, that was the problem. For Wikipedia it was the search system that uses Lucene, also written in Java. Mozilla was having problems with Hadoop, a piece of cloud software written in, you guessed it, Java.
These aren't just obscure open-source projects; Hadoop is used extensively by large websites Yahoo notably. Yet, it is used to some degree by the likes of Amazon, Adobe, AOL, Facebook and many others. Lucence is used by Twitter. Cassandra is used by Netflix, Facebook and many others.
The problem was resolved soon enough by most affected sites and they'll undoubtedly be more prepared. Interestingly enough, one company was prepared for leap seconds, Google. Its solution to dealing with the problem is ingenious and quite simple.
Rather than designing its systems around the "time" god, Google changed the way time works, at least as its servers are concerned. On leap second days, its NTP (Network Time Protocol) servers, which are used to keep its hundreds of thousands of servers in sync, add a few milliseconds once in a while so, by the time the rest of the world adds the leap second, all of its servers have accounted for it.