January 14, 2020

Product Inventory Sync from Distributor

Jesse Bethke @JesseBethke

My distributor has a warehouses across the country with a massive inventory. Additionally, they can ship and deliver to me with in 2 days. I already offer free 2-day shipping on my inventory via Social Inertia. So theoretically, I can do on-demand ordering and turnaround on shipping within 5-7 business days with my distributors entire available inventory.

That will only work if I have reasonable up-to-date details about their available products and stock. Porting that over and regularly updating it is essential. Not to mention that all that product data is an SEO gold mine.

One caveat. My distributor doesn't have an API (or do they...). What they do have is an zip-compressed csv of SKU's and inventory levels by warehouse.

I prepared a Lambda routine to run nightly and download that ZIP, decompress it, and then stream some 50,000 lines into an aws SQS Stream. Writing all that to Dynamo in real-time would thrash it. SQS uses concurrency limits to invoke the messages in batches back to Lambda for processing and updating in Dynamo.

After some tweaking, I have the system automatically syncing new SKU's and available inventory for 50,000 products from the distributor nightly with minimal load on the system.

Unfortunately, all this products are missing images and descriptions. That's not too useful...

Loading comments...