Frameworks are discussed a lot, but what data storage are you using?
We're mostly seeing developers talk about major frameworks they are using. Be that NodeJS, Django, React, Angular, etc.
But I'm wondering what people are using in terms of data storage these days. Also what other helper libraries if that's the case.
Post your project below and tell us what you're using 😁
Postgres + Redis.
Final answer.
Use the right tool for the job.
If your data is highly relational, a relational database
If your data is likely to change and evolve constantly and you have no idea how it will look like in a couple of weeks, use a document database
If you expect high loads on your db from the early days, or you want to be "serverless", use DynamoDB
If you need search capabilities, use ElasticSearch
If you want something quick to get off the ground and are not worried about cost, use FIrebase Firestore
Unlike front-end frameworks which ultimately are opinionated ways of achieving the same end goal (render pages in the browser), data storage technologies generally exist to solve a particular set of problems.
My current project bkmark.io uses DynamoDB and ElasticSearch
Love the concept of your project.
Noob question: how You created tagging or classic pagination in DynamoDB? Do You have multiple tables or many?
I have background in relational databases. When I was trying DymaoDB it only have me headache :| I thought that only cursor pagination (similar to commits pagination in GitHub) is possible in Dynamo.
I have multiple tables, but the "core" of the app is designed around a single table. Pagination is implemented with the aws SDK, you have that basically out of the box.
It takes a bit of a mental shift coming from relational databases, and the aws docs can be daunting as well, but if you're in the serverless space, it's a fantastic product.
My go to in terms of databases is either MySQL or PostgreSQL. I'd say these are fine for 99% of use cases and are have libraries in possibly all of the popular languages.
For cache and any other simple, ephemeral storage needs I'll go with Redis.
My framework of choice is Laravel. Purely because it's easy and fast to develop with and has the libraries and documentation for employing both of the above (DB and cache).
Honestly tho, I'd just go with whatever you're comfortable with and take it from there. MySQL and some sort of cache layer is fine for most things and scales fairly well. If you feel like it's becoming a bottleneck at some point, can always carve out some time to migrate to another storage solution, but again, the chances of that happening are slim to none.
I'd only apply this to something new I'm building, where I know neither the scale nor have a good idea of what the end product will look like. If you're working with an already established product and you're already aware of the scale and the full extent of the project, then do your research and use what you think fits best.
I normally go with either Postgres or MySQL for relational data. Then DynamoDB or Redis for non-relational data.
MySQL with mysqljs on Node. Firestore for a document DB and Airtable for codeless.
In trainersjoy.com, I am using PostgreSQL for relational data, and AWS S3 for file storage. I access PostgreSQL via Java Persistence API. I access AWS S3 using Amazon's latest API.
Postgres or CockroachDB
PostgreSQL. If only for the great jsob support :). Redis and Elasticsearch if necessary (depends on the project).
MongoDB Atlas.
That said, I've only used data storage for side projects that are in a very early stage, just to test out how it would work. I can't say how well it would scale if I need to use it for thousands or even millions of entries.
Let's hope we're going to have to deal with scaling at some point soon 🙌
First time I heard about FaunaDB 😃 I'll check it out
I use CouchDB along with PouchDB.js.
The great thing about CouchDB is it provides a really easy and robust way to turn a "Cloud" app into a "Local-First" app, and also sync the two together.
Using CouchDB as a back end for a Local-First app that's also an "Offline First" app gives a web app near native app performance that still works when the user's internet connection dies.
It also gives the developer an avenue to duck out of having to provide a robust Cloud based back end.
Most importantly, it puts the users data in their possession and control, where it should be. Using the Cloud version of the app is an opt-in decision the user makes and they can turn access to that on/off.
I believe these are important features app developers need to start thinking about when it comes to data storage.
I have a new version of my ezInvoice app that's just about ready for release. There are actually 3 versions of the new app:
Web app that stores user data in the web browser's IndexedDB.
Cloud app that stores user data on a CouchDB running on my web server.
Local-First app that uses CouchDB installed on the user's desktop PC.
There's a demo of the #1 app at https://cherrypc.com/app/demo.html
This comment was deleted 6 years ago
Typically it's a db per user but a db can have more than one user, and users can be assigned "roles" that can be used to control what a user can access and what permissions they have with "design documents" that are basically a bit of javascript that describes what permissions they have.
Firestore, and Firebase storage.
Looking at Postgres next.
I'm using Postgres for the new project. I've got a combination of relational data as well as JSON data.
Do you store the JSON data in the Postgres DB as well?
Yes, I'm using JSONB data type of Postgres which offers query capabilities as well.
Postgres + Redis + DigitalOcean Spaces.
How are you finding DO's Spaces? I was thinking of experimenting with it at work. Any drawback, super cool stuff? 😃
No to both! I think at this point most object storage services seem to be clones of S3, including an S3–compatible API. I just use DigitalOcean for everything else so it was a natural fit.
I've been using it since it's release, my only issue with it is that the CDN is noticeably slow. Other than that, it's great + cheap.
Relational: MySQL, NoSQL: Redis, Objects: S3
I have also tried firebase/realtime-db around three years back, but it was nacent. Will retry a project with a realtime-db once more as things must have come a long way.
My projects (syren.app and AnimeHub) use Parse Server hosted on heroku
I currently use:
Uclusion uses DynamoDB via PynamodB and S3. We have a blog on S3 for file storage.
To expound on that a bit, we've carefully designed our key structure and data model to be safe in the face of eventual consistency. So for instance we repeat things where needed to make sure api users have enough data to piece things together without having to make another call, make sure that the mutators return full objects, etc.
This comment was deleted 5 years ago