SQL? NoSQL? Just anything to get started???
Hello indie hackers!
Amateur React developer here building out a few cloud-native applications, but I keep getting stuck on the same step— choosing a database for my stack. Currently, I'm working on hedgerow.io, a platform for tracking the investments of billionaire hedge fund managers.
I've gotten user authentication up and running, and I'm serving the data out of a JSON file + Flask app hosted on AWS. I'm looking to add a little more functionality to my app, such as giving users the ability to "follow" specific investors, but every time I reach this level of complexity on a project I stall out perusing the sea of database technologies available, constantly questioning which would be the best for my use case :-(
DynamoDB has great documentation, but I've read many articles that seem to indicate a SQL option would be better— as a newbie, is there anyone who can offer me guidance on which to choose, or if it even matters at such an early stage???
Looking forward to engaging more with this community, longtime listener of the podcast :-)
It does not matter, just pick one and move on. (I would choose SQL here because one of your primary offerings will be to provide rich query capabilities of the data you collected and the overall size of the database will be small /everything nowadays small which is less than TB sizes DB/) You might create optionally an abstraction over the database access and create clear interfaces so then you can switch more quickly to the other DB later. Btw what is the source of your data?
Appreciate the advice! Currently I'm hard-coding the data into the JSON file myself, but my planned next steps include integrating the whalewisdom.com API— just want to add enough compelling features before paying for a subscription. A few of the data points also come from gurufocus.com.
Here is a list of DBs that support JSON:
https://www.quackit.com/json/tutorial/list_of_json_databases.cfm
If you have budget for the DB, I would recommend Snowflake because you get the best of both the traditional RDBMS world and also the NoSQL world. Snowflake supports JSON data types pretty well through its variant data type. You can dump the JSON data as-is and then parse it using regular SQL pretty easily (link below) and also you don't have to worry about tuning/ indexing/ vaccuming the DB later when you have lots of data because it's auto-managed. You get a trial with $400 free credits for 2 months and after that its like 3-6 bucks per compute hour and $20~/month per TB storage.
https://docs.snowflake.net/manuals/user-guide/querying-semistructured.html
If you're going to be using SQL, I've heard good things about sqlalchemy, but not used it yet myself.
https://www.sqlalchemy.org/library.html#tutorials
SQLAlchemy is super useful, highly recommend it if needed.
I second this. My general advice is to choose a relational (SQL) database unless you have a specific reason for using something else. Usually those reasons are handling extremely large sets of data or the data doesn't fit a relational model well.
Sorry to echo everyone one more time but I definitely wouldn't get hung up on this. I slightly disagree with the "just pick one" responses, because I believe it is possible to pick one that is definitely bad for what you are trying to do right now.
However I agree with SQL. Go with MySQL or PostgreSQL, they have been around for a while, pretty much every language has a solid client library for it, there's also lots of community support. It will do exactly what you need, it is relatively cheap (sometimes even free), and it will actually more than likely scale with your use-case.
If/when you get to a point where your usage model requires something else (Cassandra level sharding/availability, or a time series DB or something else) then you can cut over at that point, but I can almost guarantee you that "that point" will be very very far away if you start with SQL 👍🏾
I'd go with a SQL database like Postgres as it even has the data type "JSON" where you can just store your blobs in if you want. It's the best of both worlds and on Heroku you get a free one with 10k rows so you can play around.
In the end if doesn't matter that much but I'd stay away from esoteric choices that sound easier at the beginning but will bite you quickly if there's not much adoption / tutorials / users. There's a lot of beginner tutorials for it too.
I'd say SQL is better for those starting out because it will teach you good practices in structuring and manipulating your data. If you're using Flask then you can use Flask-SQLAlchemy library along with Flask-Migrate. Miguel Grinberg has some excellent tutorials on this topic and Flask development in general.
If it's speed and simplicity - just pick MySQL and PHPMyAdmin and you are sorted. Unless you will outgrow a proper dedicated server there is no difference, only time you need to invest in the solution. LAMP is the way if you want fast and simple. Otherwise, just toss a coin - every solution have its ups and downs, some will require much more time spent though.
NoSQL databases are generally easier to modify on the fly if you decide to start pulling in different data formats and whatnot during your initial stages, but honestly, I'd pick one DB that you think would be easiest to pick up on based on documentation and usability, and then figure out the long-term strategy if your product gains some traction.
It doesn't really matter which DB you pick, just pick one and run with it. I'd just pick a popular one like MySQL or Mongo, or go with whatever database you've used in the past and have experience with. The choice of database is going to have little bearing on anything at these early stages, and by time it might cause a problem you should hopefully have a better understand of why and the resources to deal with it.
If you are currently already storing data as json, why not go for NoSQL - like MongoDB or similar ?
This comment was deleted 6 years ago