Content on the web increases quicker than everyone can imagine. Perhaps Google is having trouble to index all this content using the old search index engine. That’s why they made a new seach index engine and they called it Caffeine.
Today, they officially announced it on the Google Blog. The basic idea of Caffeine is like a real time updater for any content on any web. Any small changes on your web will be known by the indexer and it will add it to the index database on Google’s server. Hence bringing the latest content to people’s search in Google’s website.
Google has also provide a minor information about how it works. It is a great system indeed.
Taken from Google Blog:
Caffeine takes up nearly 100 million gigabytes of storage in one database and adds new information at a rate of hundreds of thousands of gigabytes per day. You would need 625,000 of the largest iPods to store that much information; if these were stacked end-to-end they would go for more than 40 miles.
You can see how the old and new search index works in the illustration above.
Here’s a video about how a search process works in Google: