In an age of LLMs, is it time to reconsider human-edited web directories?
Back in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.
These directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/Fashion/etc.
Each of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.
Typically, these directories also had a limited web search that would crawl through the pages of websites listed in the directory.
Lycos, Excite, and of course Yahoo all offered web directories of this sort.
(EDIT: I initially also mentioned AltaVista. It did offer a web directory by the late '90s, but this was something it tacked on much later.)
By the late '90s, the standard narrative goes, the web got too big to index websites manually.
Google promised the world its algorithms would weed out the spam automatically.
And for a time, it worked.
But then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.
And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.
My question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?
Do we really want to search every single website on the web?
Or just those that aren’t filled with LLM-generated SEO spam?
Or just those that don’t feature 200 tracking scripts, and passive-aggressive privacy warnings, and paywalls, and popovers, and newsletters, and increasingly obnoxious banner ads, and dark patterns to prevent you cancelling your “free trial” subscription?
At some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of trustworthy, quality websites?
And is it time to begin considering what a modern version of those early web directories might look like?
@degoogle #tech #google #web #internet #LLM #LLMs #enshittification #technology #search #SearchEngines #SEO #SEM
@ajsadauskas @degoogle I used to be one of those human editors. I was the editor of Scotland.org from about 1994 to about 1997, back in the days when it was exactly one of those hierarchical web directories – with the intention of indexing every website based in Scotland.
@ajsadauskas @degoogle having said that, the patents on Google’s PageRank algorithm have now all expired, and a distributed, co-op operated search engine would now be possible. Yes, there would be trust issues, and you’d need to build fairly sophisticated filters to identify and exclude crap sites, but it might nevertheless be interesting and useful.
@ajsadauskas @degoogle I guess the problem though is how you make sure they are actually maintained by a human acting in good faith. The way community Facebook groups meant to be for this kinda thing get spammed by likely fake businesses doesn’t give me hope
@joannaholman @degoogle Good point.
If it were run as a private company, I think the solution might be just to pay actual humans as employees.
If it’s a community-run project, the challenge would be to come up with a robust moderation system…
@ajsadauskas @joannaholman @degoogle maybe a mix of wikipedia and search engine would be nice. WikiSearch?
I suppose any measures at all would cut out a massive number of spam pages already.
I had this exact thought earlier today. Either curated directories, or a ground-up, vetted search engine that only pulls from pre-screened sources.
@ajsadauskas @degoogle it sounds a bit like Kagi‘s Small Web initiative and search. have you seen it? https://blog.kagi.com/small-web
I don’t know if this is the intent of their small web effort, but the first impression I got just now when I clicked through and saw this was, “gross”.
Scummy looking/feeling marketing hustle is a huge turn off.
Yes to all. For a while I’ve been de facto using a miniscule subset of the web. My gateway to other, relevant websites are via human-to-human recommendations, primarily in a place like this.
And just now, as seen at the bottom of a blog post:
“Post a Comment
Unfortunately because of spam with embedded links (which then flag up warnings about the whole site on some browsers), I have to personally moderate all comments. As a result, your comment may not appear for some time. In addition, I cannot publish comments with links to websites because it takes too much time to check whether these sites are legitimate.”
@ajsadauskas @degoogle I mean we could still use all modern tools. I’m hosting a searxng manually and there is currently an ever growing block list for AI generated websites that I regularly import to keep up to date. You could also make it as allow list thing to have all websites blocked and allow websites gradually.
@ajsadauskas @degoogle I started that because it bothered me that you couldn’t just report a website to duckduckgo that obviously was a stackoverflow crawler. This problem persists since reddit and stackoverflow are a thing themselves. why are there no measurements from search engine to get a hold of it.
I never understood that.
@ajsadauskas @degoogle a bit of history of Yahoo here, started as a web directory https://www.wired.com/1996/05/indexweb/
@ajsadauskas @degoogle Since I run a small directory this is a fascinating conversation to me.
There is a place for small human edited directories along with search engines like Wiby and Searchmysite which have human review before websites are entered. Also of note: Marginalia search.
I don’t see a need for huge directories like the old Yahoo, Looksmart and ODP directories. But directories that serve a niche ignored by Google are useful.
@bradenslen @ajsadauskas @degoogle looksmart! There’s a blast from the past.
As a very early internet user (suburbia.org.au- look it up, and who ran it) and a database guy, what I learnt very early is that any search engine needed users who knew how to write highly selective queries to get highly specific results.
Google - despite everything - can still be used as a useful tool - if you are a skilled user.
I am still surprised that you are not taught how to perform critical internet searching in primary school. It is as important as the three Rs
But directories that serve a niche ignored by Google are useful.
This is a good point - as search is increasingly enshittified too (from top down, with corporate interests, and bottom up, from SEO manipulation and dodgy sites) it makes sense for topics or communities often drowned out by the noise.
I also see you are using webrings - another blast from the past that has it’s uses.
@ajsadauskas @degoogle
I’ve already seen new webrings forming.Or maybe that was old webrings updating?
Yeah, I was just looking at a webring and thinking “these still have a use”. They could definitely help with discoverablity on a broad front. I help Admin feddit.uk and had pondered reaching out to other British Fediverse services to make a Britiverse. However, how to hold it all together and navigate between them was proving tricky or clunky until I was looking at the webring and thought “FedRing”. Now that could work.
@ajsadauskas @degoogle Curation is elation.
@ajsadauskas
I agree we need better and remember the early days well. Before indexes we passed URLs, in fact just IP addresses of servers we’d visit to see what was there, and that was often a directory of documents, papers etc. It filled us with awe, but let’s not dial back that far!Another improvement will be #LocalLLMs both for privacy and personalised settings. Much of the garbage now is in service of keeping us searching rather than finding what we want.
@degoogle@ajsadauskas @degoogle I’ve been thinking back to the days of web rings and reciprocal links back when people had their own websites to add links too. I have been wanting to go back to that mode as well.
Indeed. As I mentioned below, something like a webring (a FedRing) might be the solution to something I was pondering.
It is increasingly clear to me that a lot of directions Web 1.0 was evolving in were diverted or just killed off by Big Tech’s landgrab which built walled gardens. I see the Fediverse as a return to the idea of blogs (micro and macro), forums, etc but in a more natural progression to interoperability. This still isn’t perfect and there may be other early web ideas, like webrings, that improve discoverablity.
@ajsadauskas @degoogle DMOZ was once an important part of the internet, but it too suffered from abuse and manipulation for traffic.
For many DMOZ was the entry point to the web. Whatever you were looking for, you started there.
Google changed that, first for the better, then for the worse.
It looks like there’s a couple projects to continue the directory DMOZ. I hope they’re sharing work with each other!
Got any links?
Yeah. Sorry, I was hesitant to post links at first before I vetted them.
It looks like “Curlie” is the official continuation of the DMOZ project:
The other ones I was seeing, it turns out, are static mirrors of 2017 DMOZ.
Thanks for that, a real blast from the past. I have a vague memory that I was an editor on the ODP or dmoz back in the day.
Sorry, I was hesitant to post links at first before I vetted them.
Yes, perhaps not coincidentally, I thought it best to ask for a human-curated link.
Y’know, come to think of it, Wikipedia might be a better project to point to here. All the content on there is hand curated. When I’m interested in a subject, I usually go to wikipedia first instead of a search engine. Sometimes I am directed out to other websites from there.
I set up a quick keyword search so I can type “wp blah blah blah” into my url bar and it searches wikipedia.
What’s to say we won’t have AI-curated lists and directories? That way we don’t have to deal with link rot and the like. I think the issue is the algorithms used for search. We need better ones, better AI, not more frivolous human labor.
