The AI vs Human challenge on Twitter (surprising results) 👀

Last Friday, I announced a challenge on Twitter: “Can you spot the tweets written by AI?”

I posted 6 tweets in 24 hours and asked users to identify the tweets written by ChatGPT.

The results are in and this is what I learned:

🧠 Take the challenge

If you want to play along, try the challenge for yourself and see if you can beat the algorithm👇)

The Rules I gave myself
Rule #1: I was not allowed to rewrite the AI tweets.
Rule #2: Minor editing, such as deleting long text or adding emojis, was allowed.

Here's what I asked ChatGPT to do:

Task #1: Rephrase a tweet.
Task #2: Repurpose a tweet
Task #3: Write a new tweet
Task #4: Write a joke
Task #5: Write a listicle
Task #6: Write a thread [Hard mode] 🔥

So, how did ChatGPT do?

Here are the results

Task #1 - Rephrase a tweet
I took a tweet from Justin Welsh and asked ChatGPT to rephrase it using different words and in grade 4 English. This fooled everyone. 👏

Task #2 - Repurpose a tweet
Similarly, when asked to repurpose a tweet from Justin. It did brilliantly too.

Task #3 - Write a new tweet
This time, it did less well. The tweet feels slightly mellow and awkward. Some people managed to spot this as AI.

Task #3 - Write a joke
This was the most surprising one. An accidental win that received the most engagement.

Task #5 - Write a listicle
I found that giving it a head start, for example, if I asked ChatGPT to complete the sentence: “Running a startup is about…”, it generated pretty decent results.

Task #6 - Write a thread [Hard mode] 🔥
This one was hard to implement. Took me a good 30 minutes to figure out how to do it and I had to split the prompts into chunks. But again, surprisingly, most people thought I wrote it and not ChatGPT.

Lessons:

Adding specificity (“Something a 90-year-old would say”) and context ("I am a sleep-deprived parent") helps.
ChatGPT performs best when it takes an already good content (e.g. Justin Welsh's tweet and repurposes it for a different audience/topic)
It works when people are not watching out for it. I sneakily added a 7th ChatGPT-generated tweet - the challenge's intro tweet itself - into the mix. And only 1 guy spotted it.

Conclusion

I started this challenge for a bit of fun and here we are. I’d say that ChatGPT passed the test, probably with a B grade. I knocked off some points for originality. (Ironic, I know).

But even an AI-detector tool did not capture all the AI-written tweets.

Will I continue to use ChatGPT for my tweets?

Not for now. The copy-paste job was hard work. And personally, I love writing so ChatGPT will just take the fun away from me. But I am excited to see where GPT4 leads us.