March 20, 2019

I made Tech/Engineering Blogs Article Search Powered by Algolia

Hi everyone!

Just want to share my side project with the community :D

The Idea ... As a developer, I always tend to do a bit of research before embarking on a new feature or project. Often, I find myself visiting Tech Blogs or Engineering hubs of the famous Tech companies, just to see how they tackle certain topics.
I searched ... This made me wonder if there's a way to have a targeted search for a specific topic limited to known Tech blogs. I started googling it (emoji here) and the next best thing I could find was a curated list in a GitHub repo form A curated list of engineering blogs [ an awesome list, by the way, make sure to star ⭐!!]

I Geeked out ... Personally, this didn’t do it for me since there is no way to perform the type of search I wanted to have available.

Simply put, I wanted to type in a subject and hit Search and see how the top in the field tackle these subjects

And, in a nutshell, that’s what EngLog is!

If you have ideas or suggestions, please reach out to me via twitter @mshbeab

#landing-page-feedback

check it out and let me know
https://englog.io

  1. 2

    Hey @richRich,

    I thought about monetization possibilities in two ways:

    1- targeted job postings

    2- premium content/courses (for example "interview training for Facebook" )

    the idea for the website is quite simple, I build some crawlers and I keep running it once per day, push it to Algolia and make it available via the web app

    as for the tech stack, I am using python scrapy framework for crawling, and for the web app I am using Laravel / SQLite as DB hosted on aws ec2

    1. 1

      @H_E_Pennypacker ok, cool, thanks for the info. The courses thing you would have to build first if I understand correctly and then advertise them there (so this would also be rather an ad thing like point 1)

      How complex would it be to do targeted product advertisement with the stuff people are searching for?

      Man, I am seriously thinking if you shouldn't make it more technical, like including StackOverflow🙈.

      Because what do you want the user really to find there? You write, "I always tend to do a bit of research before embarking on a new feature or project". Shouldn't you be able to find stuff like "scraping" or "Java framework" as well in there and get a bit more into implementation details if needed?

      Anyway, I'll keep testing as I am researching quite a lot at the moment for my own project..

      I am currently also looking to crawl a forum as my first hands-on project for a Data Engineering portal I'm conceptualizing right now, so I am curious how much effort you've put into building the individual crawlers? Have you done research on the best crawling/scraping tools or you just went with the popular scrapy because it's Python and just great?

      :)

      1. 2

        Regarding the framework choice, i went with what I know and I felt comfortable with. The scrapy library is built for this kind of things. Efforts wise and giving that a lot of engineering blogs use medium as their blogging platform that made things easier for me to have my ab bunch of them scrapped simirlarly. I honestly wanted to specifically target ling well written articles on how different tech companies tackle different topics.

        Of course I still have to do a bit of cleaning to make sure the article I end up with are of an actual substance (no conference, release announcements) .

  2. 1

    that's pretty cool, !

    Have you thought about how it would be possible to monetize it in the future?

    I thought about a similar concept once, with a slightly different direction of queries, more specifically, instructional stuff, like certain commands that you google up once every year (like git reset -- something) and a function to bookmark the solutions that actually helped you avoiding the repetitive search (although I like this one as well and will use it in my future research).

    Have you crawled the list of blogs you mention on your site and now you run queries over them? How are you planning on getting more sources in there?

    Want underlying technologies are you using if I may ask?