Product catalog normalization and entity resolution
A fragmented product catalog made search, pricing, and reporting unreliable across millions of listings.
- Industry
- Online marketplace
- Outcome
- [X million records] normalized across [N source systems]
Organization or industry
An online marketplace operating across multiple categories and regions, with a large seller base and a catalog assembled from many independent feeds.
Initial problem
The catalog had grown through acquisitions, third-party feeds, and years of manual entry. The same physical product appeared under many slightly different names, brands were spelled inconsistently, and attributes such as size, color, and material were stored in free text. Duplicate and near-duplicate listings made search results noisy, distorted pricing comparisons, and undermined trust in every downstream report. There was no reliable way to answer a question as simple as how many distinct products the marketplace actually carried.
Technical environment
- [N source systems] feeding the catalog, including seller uploads, distributor feeds, and legacy import jobs.
- A cloud data platform on Azure, with data landing in [storage layer] and processed through a set of batch pipelines.
- Roughly [X million records] in the active catalog at the start of the engagement.
- Existing search and pricing services that consumed the catalog directly, so any change had to preserve backward compatibility.
Abisam team involvement
A small Abisam team worked alongside the marketplace's data and platform engineers. Abisam led the data modeling, matching logic, and pipeline design, and paired with the internal team on review and handover so they could own the system after launch.
Approach
We started by profiling the catalog to understand where the inconsistency actually lived, rather than assuming. From there we built a normalization layer that standardized brands, units, and attribute vocabularies against a controlled reference set. On top of that we implemented an entity resolution process that grouped listings referring to the same underlying product, using a blend of deterministic rules for high-confidence matches and a probabilistic model for the ambiguous cases. Every automated match carried a confidence score, and low-confidence groups were routed to a human review queue rather than merged silently. We treated the reference vocabularies and matching thresholds as configuration the internal team could tune, not as fixed code.
Systems or processes delivered
- A normalization pipeline that maps raw attributes to a controlled vocabulary and flags values it cannot confidently place.
- An entity resolution service that assigns each listing to a canonical product identity, with confidence scores and full lineage back to the source record.
- A review workflow and queue for the ambiguous matches, so humans spend their time only where the model is unsure.
- Documentation and a handover so the internal team can maintain the vocabularies, thresholds, and pipelines.
What the organization was able to do afterward
The marketplace gained a single, stable notion of a product that search, pricing, and reporting could all rely on. Duplicate listings dropped to a level the team could manage through the review queue, category reports finally reconciled, and the internal engineers were equipped to extend the matching rules to new categories without outside help.
Have a system that has to hold up?
Tell us what is not working and we will tell you where to start.