The existence of a web crawler designed by Apple has finally been confirmed by the company. Apple uses this crawler, known as the Applebot, for queries made on the company’s Spotlight search and Siri. The existence of such a bot or web crawler has long been rumoured but was never confirmed. In fact, until now Siri and Spotlight were thought to be using results from search service providers like Microsoft and Google only.
Apple quietly added the support page to its website, detailing the Applebot. Reports say that the Cupertino, California-based company's deal with Google for search services is going to run out soon. Apple is rumoured to be working on a full web search mechanism for the future.
Further, the Applebot reportedly works in the same manner as Google’s search algorithms, that is, it finds the robots.txt file, which tells it what results are to be excluded on a particular website. Overall the Applebot will follow similar instructions as Google’s search spiders, unless there are some Apple specific rules that are given to it. Apple’s support page clearly mentions, “If robots instructions don't mention Applebot but do mention Googlebot, the Apple robot will follow Googlebot instructions.”
Back in February, information related to a Search service from Apple had surfaced, sparking rumours of Apple working on its own search engine. It isn’t clear how long the Applebot has been operational though. Apple has written about the crawler on its support page, even providing an email address for additional queries from users.