By Joe Levi | April 1, 2011 10:44 PM
Everyone who runs a website, ourselves included, needs to stay on top of current industry trends with the big three search engines: Google, Bing, and Yahoo! Not long ago Google announced a radical change in the way they were going to spider websites. The new spidering and indexing engine was codenamed “Caffeine”.
“Caffeine provides 50 percent fresher results for web searches than our last index, and it’s the largest collection of web content we’ve offered. Whether it’s a news story, a blog or a forum post, you can now find links to relevant content much sooner after it is published than was possible ever before.”
It turns out Google had another reason for making the radical change to their indexing methods: power saving. When you run as many servers as Google does electrical and bandwidth costs make up a very large percentage of the expenses.
“From the very beginning Google’s servers were always low-power Linux machines running our own proprietary operating system. Even with relatively small computers, 200 watts times hundreds of thousands of servers added up fast. Once we brought YouTube and Maps online it was obvious that we couldn’t continue with what got us started.”
The company needed to reduce their electrical consumption significantly, but their Linux based OS was still too bloated to run on anything but desktop CPUs. “We got the idea of swapping out our traditional computer towers with embedded systems — basically circuit boards with the entire system on just one or two chips, which use less power. A lot less.”
Google turned to a small company that was writing an operating system for embedded systems and cell phones. “Android was a no-brainer. It was light enough to be able to run even on a cell phone, powerful enough to run a server, and was based on Linux, so we were able to run our pre-existing code without many modifications — and used only a fraction of the power required to run one of the old servers.”
Part way through the development of the new servers Google forked their development on Android. Half the team stayed with the server roles and developed their new Caffeine indexing system, the other half were assigned to “toying” with the idea of realizing the originally intended use of Android as a cell phone OS.
“We already had Chrome OS on the table, but with all the foundations of Android already there, we just pushed forward with it. The rest is pretty much history.”
When asked if Google actually had smartphones in their server closets, after some initial chuckling, one engineer replied: “Well, the hardware is a little more ‘robust’ that what’s in your run-of-the-mill smartphone. We have physical ports for ethernet and the like, but the boxes are almost identical to what we did with Google TV. Some of the guys were able to build a server bank using a dozen Nexus Ones, just for fun. It’s still running back at the office.”
Indexing web pages faster than ever before, is there nothing that Android can’t do?
Source: Google Blog