Handlr
Bookmarking service with social aspects
Had some success with a reference post on HN. While I shared Handlr a couple of times already in the past.
The reason why it had some success now, was because I showed off how I used handlr in relation to HN.
I shared the url: https://handlr.sapico.me/?domain=https%3A%2F%2Fnews.ycombinator.com
Here they can see who I'm following ( eg. comments) and which discussions I bookmarked.
I presume the performance improvements also made a lot of difference, as a pageload reduced from 5 seconds to ~1 second.
I had ~500 visits of HN and on average they visitied 2 pages per session with a minute duration. Good enough for me!
Have a good new year all!
I've been dogfooding handlr for a while and recently did some effort to improve searching performance on a ms-SQL db.
Some interesting stats from what I aggregated over the years.
- Db of 6 GB
- 1 million items ( this is mostly because it was crawled)
- 10 k. items where added by me and the rest was crawled
Here are all the url's from the domain of Indiehackers: https://handlr.sapico.me/?domain=https%3A%2F%2Fwww.indiehackers.com
Every day I had to remove 18 links. I already implemented recaptcha, but it wasn't enough.
When I cleaned up the database today, I had 500 spam accounts.
After the cleanup, 5 new spam posts appeared... So this was enough for me. Time to act.
I decided to take another approach. I saw they filled in all fields at once, which isn't usual in a Reddit like site.
So I added a hidden body field. Which, when filled in, returns an XML-bomb which could lead to a 3GB memory usage. I hope it's overloading most crawlers.
The XML-bomb itselve is called "a billion laughs" and it's returning a self expanding xml-entity. Although some headless browsers could handle it, I hope most spamming crawlers just crash.
I'm awaiting results and going to sleep now.
Update: Tweaked the honeypot method a little bit. Hope it gets results ( i saw spam appearing during the creation of this post)
-
1
I am curious about your XML-bomb technique. Can you elaborate? For example, when you say "it's returning a self expanding xml-entity", do you mean after the form is submitted? Or while it's still being filled out?
-
1
The XML bomb happens when the form is submitted in an "incorrect way". Eg. with a hidden field or when a link or a text is submitted ( which doesn't make sense for the use-case).
About the actual file:
https://www.soapui.org/docs/security-testing/security-scans/xml-bomb
-
-
1
Update: The spam crawler didn't fill in the hidden field and had a workaround for recaptcha.
The Honeypot method was tweaked with checking if all fields were filled in ;
- Title
- URL
- Tags
- Value
It's not logical to fill in all at once and it's warned against it.
Currently it's good enough, not perfect.
-
1
Update: It seems the XML-bomb is working. No more spam messages on my Reddit like site and lower traffic currently.
My current guess is that they tried to add +/- 10 posts, which would lead to a max of 30 GB. memory usage after a while. Their server had probably far lower specs.
I will monitor it the next few days and pray they won't implement a workarround.
I'm dogfooding the application.
It's now applied to where i live Bruges as an example. This way i have an update on certain interests in my neighboorhood.
Some usefull additions to the application:
- RSS imports articles
- Comments
- Tags / item
- Managing tags
After finishing the project, the project was discussed internally at the client for several months.
In the end, they went for Sharepoint. I think my "side-biz" was not sufficient for a large company with > 90 employees. Which seemed logical, my latest update ( 2019 ) is that they regretted their decision about Sharepoint.
After discussion with a friend of mine. I started an MVP for his employer ( he is a partner of the company).
I had no guarantees it would ever integrate, but the project seemed sufficiently usefull for other use-cases. A lot of the discussed functionality was a extension on HackerNews
Comment