Thanks to the multi billion dollar advertisement industry, searching for something on the internet has devolved from a joyous Altavista guess-the-keywords activity to a tiring chore where one has to wade through endless pools of generated SEO-optimized crap, hollow company blogs with more social media link embeds than actual content, and Reddit flame wars than ever before. In short: great stuff.
Suppose you’re looking for a review of a video game. The first 100 hits will return the expected results: articles from huge journalism companies with a big enough budget to bribe any search engine to stay on top. And while these IGN et al. reviews are interesting to a certain degree, I want to read about the honest opinion of another person, another human being—whether or not journalists are human beings is still up for debate.
I’m sure you know what I mean: I’m looking for small, independent websites carefully curated by people who care. I want to discover personal blogs, not professional cookie-laden ad-riddled heavyweight junk that has my router choke on 50 MB of cruft instead of just loading one document and a few pieces of metadata. I do not want to click like and subscribe. I am not interested in Facebook embeds. I am not willing to consume words that have nothing to say besides click here. I am not bait.
How should you search the internet, while avoiding that cruft? I think by now it’s clear that a simple Google search isn’t the answer, and neither is migrating to a privacy-friendly DuckDuckGo search engine that fetches results from Bing: same shit different engine (and progressively worse results, I might add).
Instead, I have been relying on Search My Site: an open source search engine specifically geared towards personal and independent websites (like this one). Unfortunately, but perhaps not unexpectedly so, Search My Site is not very good at finding things: if your website happens to be in their index, you’re good, but if not… Is that different compared to the big ones? Not really. What do you do when you can’t find something in one search engine? You revert to another strategy. Marginalia Search is another great little gem. It clearly states its purpose:
This search engine isn’t particularly well equipped to answering queries posed like questions, instead try to imagine some text that might appear in the website you are looking for, and search for that.
Where this search engine really shines is finding small, old and obscure websites about some given topic, perhaps old video games, a mystery, theology, the occult, knitting, computer science, or art.
My experiments recently made me switch from SearchMySite to Marginalia as my go-to small engine.
Both seirdy.one and Dan Luu provided outstanding overviews on alternative search engines optimized for the small and independent web, where Search My Site and Marginalia.nu happen to be just two of the many ways to tap into personal websites and blogs. Not every engine has their own index database, but those that do can curate entries more rigorously. It’s now easier than ever to trust and rely on smaller engines, says Dan:
If you want to make a useful search engine for a small number of users, that seems easier than ever because Google returns worse results than it used to for many queries. In our test queries, we saw a number of queries where many or most top results were filled with SEO garbage, a problem that was significantly worse than it was a decade ago, even before the rise of LLMs and that continues to get worse.
Another strategy is to try and skip the first thousand results of conventional search engines and focus on what lies below and forgotten: that’s exactly what Million Short set out to do (based on the Bing index). As stated on their site, Million Short “Remove over SEO’d sites that show up over and over again with ease.”. It’s just sad strategies like that exist and are needed to scour around the web nowadays. Million Short’s about page does smell an awful lot like Silicon Valley VC-like brands though.
Then there are web directories such as Indieseek.xyz and Blogroll.org that simply list blogs as entry points, leaving the exciting spelunking up to you. There, you first search by general topic before diving deep.
Lastly, premium search engines like Kagi started popping up that claim to deliver fast and personal results, free of ads and tracking—provided that you pay $5 a month for 300 searchers or $10 for unlimited queries. Kagi aims to replace your general Google-esque search and does not focus on small sites like SearchMySite or Marginalia does. With Kagi, you can block and/or filter domains you do (not) like, which to a certain degree is also possible in Million Short. I haven’t tried it myself, but have read impressions from Dave Heinemann, Horst Gutmann, and Kev Quirk.
Google et al. are great “answer engines” when technical questions arise, but their cool new website discovery levels leave much to be desired. If you are going to rely on them, do make sure to bring protection such as a Pi-Hole and a good content blocker, as these big boys made ads and scam look like real results. Fortunately, alternative smaller engines focusing on personal sites do exist. They profoundly changed the way I use the internet—for the better—and for me made fooling around fun again, not unlike the good old StumbleUpon days.
Most browsers (and Alfred!) support custom search shortcuts that allow for quick searchers in many different engines, big and small. You owe it to yourself to check out at least a few of the ones mentioned here.