How to manage user file limits with AWS?
With Jamform I want my users to be able add file upload fields to their forms. But I want them to have a limit such as 1 GB of files. I’m just wondering what the best way to manage this cap is. Should I save every file under a folder with the userId, then every time a file is uploaded I fetch all the files from this folder and add up their file sizes to check if it reaches capacity? Should I just keep a running total as a field in the database? Does AWS for Node have some functionality that I’m not considering to make this easier?
I added a column for the size in bytes of every file on the database table I use to track a user's files. Then it's a query on the table. Should scale ok-ish until a user ends up with tens of thousands of files. At that point I'll probably cache the result in redis and make sure to update the cache on every file upload.
A running total in a single database row could work, but you run a risk of it getting out of sync with the actual count (e.g. file upload succeeds, but the update of the total crashes or vice versa and the operation was not made atomic). If you go that route, you'll probably need a background process that periodically checks that it is correct.
I'd put all the users files into separate folders. Then I'd setup a trigger on objectCreated on your bucket that populates your DB with a running total of the users files.
That should scale really well, and you know before uploading if a file is too large, and can query for users usage etc.
How did you end up doing it @Harrjm ?
I ended up just saving the size along with the file in the database. This allows me to run an aggregate function to grab the total size by user and by form for different use cases. The only downside is the potential for the DB and file storage to get out of sync but I built in a lot of checks to try to prevent that so hopefully not an issue.
yes, store sizes in db, you need to store location on S3 anyway