We used AI to analyze +100,000 tweets and can now predict how well your tweet will do!

Hi here 👋

I'm Simon, founder of FeedHive.

We launched our product in public beta 8 months ago.
In this period, our users have used FeedHive to publish more than 100,000 posts to Twitter, LinkedIn, Instagram, and Facebook.

We've been using AI for idea generation for a while, but recently, we decided to up the game!

Sitting on more than 100,000 posts - including private metrics such as impressions, number of profile clicks, link clicks, details expands, etc we realized it was the perfect opportunity to start using this data to train an AI model.

The AI model that we ended up with, can predict how well your post will perform in terms of engagement rate, and - according to one of our users - "out of 4 posts, 3 scores were stunningly correct".

So - let me share some insights into how we did this.

The AI Engine

First of all, I think it's worth mentioning, that no members of our team are specialized Machine Learning experts.

We used OpenAIs popular GPT-3 to create this solution.
Their platform allows to perform a so-called fine-tuning on some of their engines.
This basically means, that you take one of their models, provide it with training data and specialize it.

We used their Curie engine to create our fine-tuned model.
This is not their most powerful model, yet, the results were surprisingly good!

Preparing the data

OpenAI made the process of fine-tuning a model incredibly easy. Huge kudos to the team, they really did a great job!

The hard part is preparing the training data and account for biased.

Accounts with a large following get more engagement, and their posts tend to reach further. Does that mean that everything they post is just better?
No - we can't make any such assumption.

So in order to account for this, we chose to use engagement rate as our metric for performance. That is, the number of total engagements (likes, replies, retweets, link clicks, profile clicks, ...etc) divided by the number of impressions/views of the post.

Yet - this introduces another bias.
Very small accounts now have an advantage. Say a profile with 10 followers gets 5 views and 1 like. That's a 20% engagement rate, which would be unheard of for big accounts.

Thankfully, OpenAI allows you to do 10 full training sessions per month - free of charge - and they will even extend this upon request.

Needless to say, it took us quite a few rounds of experimenting to find a good balance between penalizing the score of certain accounts and giving a bonus to certain others, in order to make the training data as non-biased as possible.

(We're constantly improving this still!).

Classification vs. Completion

So - this part gets a little technical.

OpenAI offers different ways to use their powerful engines.
When you look at a problem like this, your first intuitive thought might be that you should use classification. It seems like that's what we're doing, right?

Actually, we didn't get good results with classification at all.
Instead, completion seemed to work very well.

In a simplified version, we trained the Curie model to complete a prompt on a structure similar to:

Post: "This is my best tweet"
Score:

And then have the completion being a single number between 0-9 indicating the prediction.
The training data then consisted of 100,000 of these, where we had given each post the score from 0-9 based on our ranking system.

Deploying the model

After each training session, you can use your model by simply referencing it in an API request, so again - deploying the fine-tuned model and using it to serve real-time outputs has been made ridiculously easy by OpenAI.

Do you want to try? 🤩

Just head over to feedhive.io and create a user.
You'll get 7 days on the free trial to test it out.

This ended up being a small blog post, hehe.

I hope this can help and inspire someone else to get into AI and GPT-3.
It is truly a game-changer.

If you have any questions, please feel free to ask.