From 150 jobs to 5: 56 suppliers and 200,000+ products in one up-to-date catalogue

An electrical supplies and lighting e-shop with a very large catalogue: over 200,000 products from 56 suppliers, with prices and availability that change every day.

Every supplier sends its own file, with its own structure. To keep the catalogue up to date, about 150 import and update jobs ran on the e-shop's server.

The jobs ran in parallel and it was never clear when one finished and the next started. The server was overloaded, the site slowed down for customers, and when something went wrong, debugging was hard: wrong prices, products shown in stock after they had sold out, duplicates and stuck jobs.

On top of that, one person spent half a day monitoring and fixing the files.

The work left the e-shop's server

All file processing now happens in AfterFeeds. AfterFeeds reads the files of the 56 suppliers, cleans and merges them, and gives the e-shop ready-made feeds. The 150 jobs on the e-shop's server became 4–5.

One sequence, one place to check

Jobs run in sequence, not all at once. Every run is logged, so it is immediately clear which file was late or had a problem, without digging through server logs.

Products daily, stock hourly

Products update daily and stock every hour. Anything missing from a supplier's file is withdrawn, unless it is still in the ERP, in which case it stays visible until it sells out.

Room to grow

The setup handles up to 350,000 products, so new suppliers can be added without changes to the e-shop.

What changed

The site got faster, because the e-shop's server now serves customers instead of imports. Half a day of manual work became automatic. And errors in prices, availability and duplicates are caught before they reach the e-shop.