Catalog and stock integration for Luxclusif — an ETL conveyor built for partner storefronts.

Context and intent

There is no zoo of sources in this project — the inbound stream is Luxclusif alone. The complexity sits elsewhere: every partner expects their own taxonomy, category/brand/colour/size dictionaries, rounding rules, tax and pricing policies, and CSV column layout.

We built a configurable pipeline where a single input produces many target views, each strictly aligned with a specific store's Matrixify schema. A key design choice is two-phase release: all transforms land in a draft first, pass validation and reconciliation, then commit in one atomic transaction to a final release with an immutable version.

Technology foundation

The core is native Go without heavy frameworks on top — low latency, minimal overhead, and precise control over data flow. Concurrency runs through goroutines and channels so pipeline stages parallelise without blocking critical paths.

Storage is plain SQL with explicit transactions for draft → release stages, giving atomicity and consistency. Where partners need to switch CSV versions in one tick, double buffering keeps the old version live until the new one is atomically marked active.

From Luxclusif to partner taxonomy

The flow starts with a scheduled pull of the current Luxclusif catalogue and a snapshot in a staging layer. Deltas are computed without aggressive invalidation — the draft is assembled as a full target table, then transformation rules are applied.

Transforms split into pattern-based and AI-assisted. Patterns cover predictable mappings (category families, brands, title suffixes, size templates). An AI pattern matcher trained on real examples fills gaps where manual rules inevitably leave holes.

Every card field — category, type, material, size, colour, brand, tags — normalises to the partner dictionary. Uncovered cases are reported cleanly and never break a release.

Pricing, tax and currencies

Pricing sits inside the same transactional boundary. Source Luxclusif prices pass through configurable rules: tax percentages, fees, markups/discounts, currency conversion, and rounding in a format Matrixify accepts.

Calculations are deterministic — identical input and config always produce identical output. Policy changes ship as new rule versions with safe re-release, eliminating silent drift and allowing any historical release to be rebuilt on demand.

Media and AI for visual consistency

Images go through AI background-colour and dominant-hue detection — required when a partner storefront demands strict background labels (pure white vs warm white) or when colour attributes drive filters and widgets.

We validate link availability, flag suspicious borders/padding, normalise formats and metadata. Computed attributes land in dedicated CSV columns so Shopify teams can use them without post-processing.

Drafts, cron and zero-downtime releases

Release is a two-phase transaction. Draft assembles the full partner view, runs schema validation and checksums, and produces a report of every adjusted row and why. After checks pass, release atomically switches the active version.

Cron orchestrates scheduled pulls, transforms, draft, release, and artefact publication. Two release generations stay available at all times so rollback or re-verification never blocks a partner import.

CSV as contract, Matrixify as consumer

CSV is the contract format — Matrixify handles it reliably and business teams can eyeball contents. Each table follows a fixed schema: stable column order, guaranteed types, strict headers, predictable encodings.

Rows that violate taxonomy get an exclusion reason and land in a report; the release CSV contains only valid rows — so a single bad record never breaks a nightly import.

Concurrency, performance and stability

The pipeline uses Go goroutines and channel-based workers. Heavy stages batch against database indexes — large volumes without memory churn. Deltas compute at row and field level; the final release assembles from normalised truth rather than blind full reloads.

Idempotency guarantees that rerunning unchanged inputs produces bit-identical CSV. Batch checksums instantly surface any draft vs release mismatch.

Security, versions and audit

Secrets and publish parameters live in environment/secret storage. Upload destinations get minimal scoped access. Releases carry immutable version IDs.

Logs trace every row through stages and record reasons for automatic corrections. Disputed cases rebuild from draft snapshots with the same rules — simplifying conversations with Luxclusif and partners.

Observability and Shopify Flow alerting

Pipeline metrics cover stage latency, corrected-row ratios, exclusion reasons, and publish latency. On the Shopify side, Flow triggers on purchase and order events for SKUs from our releases — surfacing how new taxonomies and pricing rules behave in production.

Signals feed Telegram/email alerting and dashboards where demand can be compared against price variance, import errors, and exclusions. An anomaly spike (e.g. cancellations in one category) quickly points to the rule that needs adjustment, and a re-release ships without stopping stores.

What the business gets

Partners receive stable CSV tables on schedule — Matrixify-ready, mapped to their taxonomy, with calculated tax/pricing and AI-enriched fields such as image background colour.

Luxclusif gets a predictable, transactional, scalable process where drafts and releases are atomically separated and reruns are bit-for-bit identical. Any change in source data reaches partner shelves quickly, safely, and deterministically — transparent, auditable, and governed by rules that can change without stopping the flow.