2
0 Comments

Suggestions for querying dynamic models at high scale

Personal pet project idea I've been kicking around for a long time has this problem that limits it's ability to effectively scale to larger challenges.

The problem:
Given a shared schema across a large set of documents that was defined and versioned but query paths and schema alterations need to be done programmaticly and query patterns configurable but need to be nearly instantly available under high read and write but can be eventually consistent reads.

The factors:
🧐 High record count (thousands which is solvable to billions, which makes this challenging)

🚀 High requests per second.

⚡ Low access time (<50ms), doable but at high cost.

💰 Low cost, difficult without being very slow.

🌌 Potentially a few dozen query permutations per model.

Explorations so far:
I know I can, and did, roll my own distributed data permutation indexing solution and did try out a few existing technologies but this proves VERY expensive with low latency at high scale, but does solve the problem. Reduction of propagation speed for changes can also significantly reduce cost but makes that less viable. Disk is cheap, but allowing for frequent change and interrogating an ever growing (massive) set of data... I'm still on the hunt for an affordable solution. 💰💰🤬😅

I've tried and failed to get ER technologies to handle this without outages during index alteration as well as performance reductions on write with any sizable index volumes. 💥

I also have briefly explored a few data mining technologies such as Athena but these proved to be far too slow due to being distributed table scans effectively. For heavy write and heavy read this proves ineffective as well (unless I'm doing it wrong which is possible) ⏱️💥

Anyone have any tips or tricks? Or if this is a gap, is anyone up for a challenge and want to partner up? 👀

Trending on Indie Hackers
How I grew a side project to 100k Unique Visitors in 7 days with 0 audience 55 comments Competing with Product Hunt: a month later 33 comments Why do you hate marketing? 29 comments My Top 20 Free Tools That I Use Everyday as an Indie Hacker 18 comments $15k revenues in <4 months as a solopreneur 14 comments Use Your Product 13 comments