What do you use for implementing fulltext search?
Okay, IH may not be an appropriate place to ask this, but I will ask anyway:
A few people who gave me their opinion about my software FileAgo said they are impressed with the overall features, but fulltext search is a strict requirement for such a software.
So, I am planning to implement this feature in the coming months. I have 3 options before me:
- ElasticSearch
- Solr
- Manticore (Sphinx)
I have barely scratched the surface and do not know much about them.
Has anyone used any/all of the above? ES seems to be trend these days but I am more interested to use Solr / Manticore, but I can't decide on one yet.
Any suggestions would be helpful.
I really rate Algolia, and it is sooooo fast. Have a look at their excellent documentation to see if it does what you need.
newer versions of Postgres have pretty good support for Fulltext search. If you create the appropriate index on the fields you want to search, the performance is pretty good! I've been using it in my current project for searching content tags in a pretty large database. Queries take less than a second or two at most.
At work we used Elasticsearch (note that the "s" is actually downcased). It's probably best for a big SAAS. That said on my next projects I will try to go with PostgreSQL fulltext or Sphinx first, because if they are "good enough" then you might save yourself Elasticsearch dependency.
Mine is not SaaS, but self hosted product. ES seems to be the simplest to setup. Has made up my mind to go with ES.
I must say I really had a soft corner for Sphinx while reading its docs, but it is ES that I will use for now.
If dataset is small then https://lunrjs.com/ for larger projects ElasticSearch.
We're all in on ElasticSearch at work. It's a little intimidating at first, but once you get the gist it's a breeze.
When working with Python, I highly suggest using elasticsearch-dsl. Like @ecdeveloper mentioned it's an ORM-like wrapper around the API.
Thanks, will check.
I've used both Solr and ElasticSearch. My choice is ElasticSearch. From what I remember after using Solr - it's query language is a little bit complex. With ElasticSearch it's more intuitive. You can even use ORM-like libraries to construct your queries.
It's also easy to scale ES (not sure about Solr).
Thanks for your reply!
I was just going through Manticore docs and see that I have to pass a unique id (64bit int) when creating a new record, which means I will have to manage the record count somewhere.
Is that similar in ES and Solr? I would like to have the software auto-generate the id instead of me passing to it.
Is ES resource hungry? What was your experience running in a single node?
I think this question has been answered already, but just to give you a few more bits - ES does generate a unique id itself, but you can override it with your own unique id (if you want to match your doc id's with your MySQL id's for example)
For the power that Elasticsearch provides, I don't think it's too resource hungry. It does like to use RAM and requires a Java Runtime Environment, but I think it's worth it. I run a single node on a relatively low powered VPS and it runs fine.
For a ballpark comparison, in my case, ES uses roughly three times as much memory as MySQL, but MySQL uses seven times more CPU. Your mileage may vary.
I love Elasticsearch and definitely would recommend giving it a go.
Yep. Going to proceed with ES. 👍
Same thing for ES, you want to use the same ID in your fulltext docs as in your primary storage, so you still have a reference to that. In ElasticSearch it doesn't have to be a 64bit int but can be anything unique...
exactly what i was looking for.