The new definition of tech stacks for AI startups

There are a lot of tech stack design decisions to make when you’re building a startup powered by AI. There are so many different pieces that come up together to build up a successful Saas. Now, I’m not talking about the kind of business that just connects a chat interface to the customer with OpenAI APIs 🙂. If you’re doing that, then you’re in Huge trouble and its time to think big 😎.

What technology can I possibly use other than the obvious OpenAI?

Let’s face it, if you are just making a simple app that connects your users to OpenAI, then you better sell it soon or just wait until your business gets ruined by 1000s of products just like your’s. To differ yourself, deciding on the correct tech stacks is vital and you need to decide the tech stack in the following categories:

Semantic Search🔍: this basically enables you to find relevant data that belongs to you and feed it to LLM. its like a database for your LLM. there are a lot of vector databases to pick from (Elasticsearch, Qdrant, Milvus, vespa.ai…). Here I’m going to describe the challenges we faced and why we used Elasticsearch

LLM (Brain of the product) 🧠: The AI itself that is able to answer a specific question or do a specific task as an agent. Believe it or not, OpenAI is not the only one out there 🙂. It might have been the first one with the breakthrough technology, but with the emerge of open source community, there are alternatives (Bard, Lamma, Falcon…)

Semantic Search
Semantic Search is the Long term memory of AI, I’m sure it has happened to you that most of the time OpenAI returns stupid responses, or there were times you wished that it actually answered the question based on your niche or the data of your product. Well that’s where vector databases come in. There are a tons of them to choose:

• PineCone: Paid service, widely used among indie hackers because of its competitive Pricing, downside is that you have to pay a lot once the requests increase.
• Weaviate: Paid service, Also used a lot among entrepreneurs, But again the downside is that once the requests increase, you have to pay a lot.
• Qdrant: An Open Source project written in C. there’s actually a benchmark done in their blog comparing different databases. This is my second go to
• Redis: Redis is known for its queue and high-throughput, But the team has implemented a vector search in Redis. The downside, everything is in memory with some query limitations.
• Milvus: another open source project that has been there for a while and they actually implemented alot of features based on FAISS Index, A vector search library developed by Facebook. They have also integrations to store the vectors in S3
• Elasticsearch: From ES 8.0 they have implemented KNN Features in Elasticsearch.

How We did it?

At URLSLAB we have implemented a lot of features using AI, LLMs and Semantic search. Implementing these features lead to creating a new project SemanticWrap (semantic search wrapped over Elasticsearch, I know the name is a little bit awkward). After experimenting with all the mentioned free solutions, there was one downside with all of them. You need to pay for a huge amount of memory to keep all the data in memory 😀. Memories are much much more expensive than Hard drives. On the other hand, most of the solutions out there (except ElasticSearch) don’t provide the ability for complex queries.

So why not Use Elasticsearch as a vector Database?

I wish it was that simple 😀. Here’s the deal, Elasticsearch’s speed in vector search decreases a lot with huge amounts of data. According to our Benchmarks, it took 9 seconds for 500K data (which is basically nothing).
Solution: Don’t reinvent the wheel, make it better 🛞
That’s why after one month of struggle, plan changes, not working software… Semantic Wrap was born powered by Elasticsearch. We were able to increase the performance of Elasticsearch (almost as fast as Pinecone) and maintain low consumption of Memory (the case where Other databases failed on). We’re going to open source the project to benefit the community also, but choosing the vector database depends on your needs, But I truly recommend Elasticsearch, if you have low amount of data, and for higher, well Semantic Wrap, stay tuned and I’ll let you know when we release

Is there really an alternative to OpenAI?

A thing that most of the people don’t know. Yeah 😀. Not Saying we are going to use it, but for sure this area is an untouched area up until the time of this writing. There are so many better open source alternatives than paid services like Open AI or Bard. The most famous of them are:

• LLama
• Falcon
  These are the most popular open source LLMs, So you can buy a server with GPU, spend some hours on setting it up there with an inference point and you’re good to go, the upside is that, you are paying a flat fee each month for the service whereas if you’re using OpenAI, You’ll be paying alot if you make a lot of requests. In Semanticwrap from the first days we built the system, so that it works well with different kinds of LLMs and different Embedding Models (that’s another topic too)

Did you even provide a tech stack honestly?
In AI era no one talks about which programming language to choose. Tech stacks are going to be more about which Embedding Model to choose or which vector store to choose or which LLM Model to use. And this is the field that has been untouched. So If your an indie hacker who’s looking for the next idea, then you might want to take a look at this field

Stay Tuned for more:
https://twitter.com/Yasha_br
https://www.urlslab.com