Web Search is Broken: The Solution [Discussion]
Hi IH!
Thanks very much for providing your insights and joining the discussion in my last post Web Search is Broken: The Problem (I changed the title from Is Web Search Broken? to make it a Series of related posts with the same title). Looking forward again for you to join the discussion by providing your feedback/insights of this new post.
Solution
The solution to solve web search is using a Tag Tree Structure (as shown below).
By this I mean that when you search for a tag you will automatically get all the results from it's children (e.g. when searching for Javascript you will automatically get articles to React.js since it's one of its children).
Tree structure is used in many applications and one noticeable one is HTML where each tag is encompassed within another tag which is considered it's parent tag.
An HTML element is a type of HTML document component, one of several types of HTML nodes (there are also text nodes, comment nodes and others). HTML document is composed of a tree of simple HTML nodes.
Wikipedia
The main search criteria is the tag and usage of tags (should be limited to 2-3 tags), and the tags used should be as specific as possible providing quality results when searched. Proper usage of tags for an article about React should only use React.js as a tag as its enough to have it discovered by members. The Tag Tree Structure isn't predefined and members can actually request changes to the tree as they see fit to create and develop it as the platform evolves. Using this kind of structure removes the abuse/misuse of tags in sites, such as, Twitter and Instagram.
Link Building
When a member finds a link worth sharing/discussing (can either like/dislike the link), the member can submit the link to the platform by filing a form of required elements depending on the type of format. The main process will be through human submissions to maximize the quality of content favoring the methods used by traditional search engines of hyping about quantity of results, which is why this platform will not scrape the web to increase the number of links.
Elements of a Link
- URL
- Title
- Owner (website owner)
- Creator (author/host/speaker which sometimes can be same as owner)
- Type (article/ video/ podcast/ blog/ guide/ tweet/ essay/ book/ product/ company/ news/ newsletter)
- Date of original creation
- Tag
- Rating (with +1 for like / -1 or dislike)
- Language
- Location (by default it will be Global but it can be specified if needed)
Search Methods
Search can be done through:
Tags (as a primary method)/ Owner/ Creator
Filtering through:
Type/ Language/ Location
Sorting through:
Rating/ Date
Evaluate Links
After the link is submitted, the member/non-member can search the platform by using one of the methods mentioned before. When a the member/non-member click on a link a new page will open so they can check the link. After they come back to the platform, they will be prompted to rate the link they have checked but only members can rate (so non-members need to create an account). Members can request amendments to any of the elements provided for a link and after review and approval it will be updated.
Engagement
Members can follow each other and discover what links are being created/rated/commented by their follows. They can have discussions of the links in threads. In the homepage, they will find the new added links. Also, they can follow specific tags to get updated of new links being added.
Differentiation
This platform isn't going to replace any of the traditional search engines because the main function they have is to use their robots to scrape ALL the web and index it favoring quantity over quality. The platform's main focus is provide insights and knowledge from links worth sharing/discussing through users and having a human ranking system to determine the best quality links.
Another way to be make the platform unique is having ads-free (as I really hate ads and I think most people do). Also, provide the privacy that many people want when browsing the internet without being tracked and having your data shared all over the place.
Similarities
I know that my solution might be sound similar to Reddit or Twitter in a way, but both actually lack the proper search function. It's not easy to go and find great links that have been submitted before. As for Twitter, even if you like a tweet most times it gets buried in the graveyard of bookmarks with difficulty in unearthing it.
Further Discussion
The platform that I'm trying to provide is trying to simplify so many of the processes that are in place and not complicate it by over-using technology. The reason for using a tag instead of searching through the link title, meta data, or the content of the page is there are many tactics to optimize search engine optimization (SEO) which gives more experienced sites with SEO an advantage over non-experienced. Search engines depend mainly on their special algorithms and SEO to rank and show results which over complicates searching for content and makes it even more complicated for content creators. So many good content is lost because of these over complications, and these algorithms and SEO rules are constantly changing which create more complications for content creators.
Many search solutions available use machine learning (ML), artificial intelligence (AI), or web scraping which isn't as accurate as when a human submits a link manually (I'm not saying I'm against them actually they are the future in many use cases, but they are still in development and these solutions can't give a rating on behalf of humans).
Observing these over complications opens my mind to many other barriers and complications that exist in the internet or in technology which I think provides MANY opportunities for us to innovate around these issues and create better and more efficient solutions. Please share your thoughts or feedback and join in the discussion!
This is indeed very interesting and actually, that's how it was done some 20 years ago.
In the beginning search engines crawled the web and categorized the page by counting keywords. Needless to say, that method was terrible as you might remember because people rapidly realized that adding hidden words increased their ranking.
That's when Yahoo did something different; use humans and human understanding to rate the search results. It was a huge difference vs others search engines like Altavista. The problem? It didn't scale. The internet outgrew Yahoo's capability to rate pages and a new competitor entered the game, yeah, Google.
What Google did back then was to automatically rate higher a page if many other pages linked to it. Furthermore, this was an automatic process that scaled nicely.
Also, they learned from its users, when they misspelled a word and got bad results they noticed who tried again and what the correction was; that was the "did you mean_____" suggestion that today is automatically added to search results. This allows learning how to spell things in any language without the need of a dictionary.
This is a broad field that has years and years of research, most of it available on the web btw.
Your model definitely will work on a very small corpus of data and with a highly motivated team of humans, but at first sight, I think you're underestimating the size of the internet.
Don't get discouraged though, I think you're on the right track.
Here's a very interesting video (13 yrs old though) when the then Google CTO explained the basics of searching. Back then the web was very different and the iPhone was just released.
Search 101 https://youtu.be/syKY8CrHkck
Oscar, thanks very much again for the video, it's eye opening and gives me more perspective about search. I'll need to watch it a few more times though using it as a reference.
Oh I just saw the video. Thanks for the video, I will check that out!
Thanks Oscar, I really appreciate your thorough reply! (and no need to edit typos I read it just fine)
I know it looks like its not scalable, but I think people's behaviors have changed from the golden Yahoo days. Now, they are tweeting daily, they are rating businesses in Yelp and writing reviews and the same is happening in Goodreads for books, and they are asking questions in Quora and StackOverflow.
This won't attract everyone, I know that, but people that are looking for quality content to share it and have it available in one place for everyone to consume.
Many problems with Google and the other search engines is that they are making website owners behave in such a as to try "new tricks" (or techniques) to win over the algorithm or the SEO rules in place. You can compare this to many books in the market (authors are also admitting this) is publishers are forcing authors to write more because they are paid per page, this means so many unnecessary pages are being filled into the books.
Again I thank you very much for joining the discussion, and for your kind words of encouragement!