How do sites like Karma scrape every websites product page?
I'm curious as to how extensions such as Karma and Moonsift scrape all product pages without manually scraping each individual element of the page. It seems like it would take ages as all sites are configured differently.
I'm not familiar with those extensions, but a lot of ecom sites will embed structured code to describe the product (https://schema.org/Product) to support search engines/organic discovery. Scraping this would be pretty straightforward. E.g. Shopify does this by default and I imagine a high % of ecom sites are now running Shopify.
They then likely build rulesets for other "custom" ecom sites based on traffic ranking, or have a process to auto-discover/build these rules.
Thank you for the response!