Federated search is different from “centralized search” in that no central server downloads all the documents and indexes them. Instead, search requests are distributed to all the participating servers and results are collated by the originating server.

The benefit:

For the search engine, this makes spiders unnecessary: you no longer need to crawl the web, making all those useless requests in order to see what’s out there.

The search engine no longer requires all the space to store a copy of every document it has indexed, or some compressed variant of it.

All this data is toxic: it makes the search engine backend a valuable target since it not only contains the key to the kingdom, it contains a copy of the kingdom!

The document servers no longer need a robots.txt file to tell complying spiders what documents to index and what documents to exclude, how much delay to use between requests, and so on.

Compliance with the robot exclusion standard is optional, unfortunately: many spiders have bugs, or wilfully ignore the instructions. Thus, servers might also require scripts to watch their load, password-protect sections, or block user agents, or block IP numbers. So much fucking work! This system truly leads to a lot of wasted time.


In a world of federated search, we would still have to defend ourselves, sadly. Having federated search doesn’t mean that centralized search (and all the SEO and marketing bots) would disappear. 😟

new: 2021-05-08 18:41 UTCTimurIsmagilov: This idea is nice. Many systems provide their own search already, why not just connect them into a federation?

Also, not sure if this is a right place for this, but here are some interesting search engines:

Alex, shall we start some federated wiki search standard? I’m planning to implement some searching capabilities for myco, might as well standardise it. Let’s call it ProjectHaustoria?. Create the page, if you are interested.


Define external redirect: ProjectHaustoria

EditNearLinks: RobotsDotTxt