January 14, 2020

Product Inventory Sync from Distributor

Jesse Bethke @JesseBethke

My distributor has a warehouses across the country with a massive inventory. Additionally, they can ship and deliver to me with in 2 days. I already offer free 2-day shipping on my inventory via Social Inertia. So theoretically, I can do on-demand ordering and turnaround on shipping within 5-7 business days with my distributors entire available inventory.

That will only work if I have reasonable up-to-date details about their available products and stock. Porting that over and regularly updating it is essential. Not to mention that all that product data is an SEO gold mine.

One caveat. My distributor doesn't have an API (or do they...). What they do have is an zip-compressed csv of SKU's and inventory levels by warehouse.

I prepared a Lambda routine to run nightly and download that ZIP, decompress it, and then stream some 50,000 lines into an aws SQS Stream. Writing all that to Dynamo in real-time would thrash it. SQS uses concurrency limits to invoke the messages in batches back to Lambda for processing and updating in Dynamo.

After some tweaking, I have the system automatically syncing new SKU's and available inventory for 50,000 products from the distributor nightly with minimal load on the system.

Unfortunately, all this products are missing images and descriptions. That's not too useful...

Today's Top Milestones
  • Weekend Club launched in Americas time zones 🙏
    weekendclub.co is the weekend co-working club for bootstrappers. Until recently, we ran weekly, Saturday co-working sessions 10am-5pm on GMT (UK time)
  • You can pay us via self-service!
    Finally, we've implemented self-service payments in our user panel, so you don't need to go away from the site to buy the paid plan. 🤑 Check out our
  • Starting Hacker-AI
    966 / 5000 In my decade long marketing tech career, I've seen how artificial intelligence and automation have changed the business. However, one funda
  • Update 126: +1 customer
    1. What we worked on last week and how well did we achieve our goals? ✅ Added 1 new customer for +$49 ✅ 3 sales calls booked for this week. ✅ Onboarde
  • 100,000 monthly readers
    Focused on a narrow niche (job searching & careers, and within that, mainly job interview advice), published content very consistently, yet focused on
  • Launched Landing & Registration
    Outside of the core product, one thing I wanted to get right from day zero was the capturing of insights & information to properly product manage this
  • Launched Landing & Registration
    Outside of the core product, one thing I wanted to get right from day zero was the capturing of insights & information to properly product manage this
  • Testing a Product Hunt Ad
    Product Hunt has been our best performing referral source for the last few years so we figured it was worth trying advertising with them. The minimum
  • Our latest blog article brought 3k unique users
    Our latest blog article (https://www.nodejsdesignpatterns.com/blog/node-js-race-conditions/) ended up on the first page of Hacker News and helped to d
  • Working on new packages
    I have been beta-testing 4 new packages these last two months, with our existing customers: * interview-based blog posts (for thought leadership conte