SEO Dub

On May 9th we have reached our revenue goal for the whole of May. This allowed me to completely ignore sales until the next month and focus on making SEO Dub better for users. I am planning to implement "Open Startup" page next week to allow our followers to track our progress.

Today our machine learning passed a golden number of 100K tracked URLs. It checks that many every day. The URLs are accessed multiple times to ensure as much data as possible is processed - in some cases, even over 1400 metrics are collected. The data is fed to the script to do the magic and adjust the global SEO algorithm. Each URL takes around 20 seconds to process, but thanks to multi-threading and the very powerful server we are still OK.

Finally, I have managed to finish the implementation of the machine learning model to recognize the content. This is crucial for SEO's, but I think no one created anything like that (at least not this type of content recognition).

The "problem" is that if you Google "new startups," the chances are that the first link you will see (besides ads) is Techcrunch startups page, where no "new startups" keyword exists at all. So why the site ranks?

Three factors really - first is historical (how often this keyword was on that page), the second is an authority (how trustworthy site is and how much authority it built through content and backlinks from related pages), and the third is a probabilistic keyword recognition model.

My main focus is on the third (first two are already nicely covered by companies around the web and pretty much most success you will get by figuring out the third one).

I created a self-adjusting model, fed it with over 4 thousand high ranking pages (mainly from medium and small sites) across the web and created a massive database of metrics and keyword recognition schemes. (I will keep adding pages all the time, I would like to have over ten thousand soon)
The model will review all metrics daily (so it will adapt when Google adjusts its algorithm).

This allowed me to go beyond usual checks of "is your exact keyword on the page" type of thing. You feed the tool with keywords - let's say "community for makers" and the tool will try to figure it out if it is possible for the front page of Indie Hackers to be related to the keyword, even if the exact keyword does not exist on the page.

It took me two months and a crazy amount of learning about content recognition, schematics of language and math behind the building blocks of sentences.

I have also reviewed current and historical mentions of Google or Google employees involved in text categorization and recognition projects, determined most likely models that they use and implemented them. Then added few more (less critical, yet supporting the main ones).

The model is still far from where I want it to be, but I want to finish implementing it entirely by the end of the weekend. I can work on it more in the coming months. The issue is that those calculations are super heavy on the CPU, so I will need to get a VDS or Dedi from OVH to continue.

The other essential part is that the system is non-binary (as opposed to pretty much everything on the market). I am not saying SEO auditors on the market are bad - they are great. I use them all the time. But I use them to let me know the issues and errors, not areas where all is good, but I could do better.
The problem for me was that I wanted something that does not give me a 0-1 score on everything. Google does not score your page by 0-1 its elements. It would be more like 0-1 where 0.78 of a point for a metric is possible.

Anyways, I consider this part finished now, and I am wrapping everything up, so I can send out reports to my SEO customers tomorrow and after tomorrow. Few SEO customers that were testing the model already see some results (around 1-2 weeks after they changed few bits and bobs according to the tool), so I am super happy about it.

What I have managed to complete is a self-adapting, non-binary SEO audit tool with features no others provide. I am not trying to build competition to others, but an alternative. Something complementary to what we have on the market already.
I am super excited that I will be able to provide my customers with something of great value, something that afaik does not exist on the market as of today.

By far this is the largest (difficult/tiresome/demanding) and most complex project I ever worked on. And it's nearly done. Awesome.