Skip to content

Luxury · Volume eDTL

The Brand Collector — 120,000-row dumps, a Fastify orchestrator, Go workers that actually finish.

The Brand Collector resells pre-owned luxury through a supplier network. Same class of problem as a catalogue feed — except each dump crossed 120,000 rows, and the CSV they hold sits above 169,000. Fastify stayed the orchestrator. The heavy compute moved to Go workers. About 60% faster on taxonomy, mapping and pattern-match at that volume.

The Brand Collector 2025 5 min read
The Brand Collector case study
speed
+60%
rows per dump
120k+
CSV held
169k+

The dump is the product

International pre-owned luxury, a supplier network, one feed problem that does not fit in a laptop. Each inbound dump exceeded 120,000 rows. The CSV they keep is 169,000+. Reliability under that load is the brief, not a pretty mapper.

Fastify holds the room, Go does the work

We started on Fastify because it stays up under load: API, schedules, reports. That layer stayed. The compute — taxonomy convert, mapping, pattern-match on 120k+ row sets — moved to Go workers. Hybrid on purpose. Node does not chew that volume. Go does. Shopify and Matrixify sit on the store side of the same pipe.

  • Fastify as orchestrator: API, cron, reports
  • Go workers for taxonomy, mapping, pattern-match
  • ~60% faster on those steps at 120k+ rows
  • SQL in the middle; Shopify + Matrixify on the way out

Volume is the constraint

This is not a three-second Sheets poll. It is a dump that does not fit in memory the way a partner file used to. The pipeline has to finish, write, and still be there for the next file. That is the reliability they paid for.

Technologies

  • Fastify
  • Go
  • SQL
  • Shopify
  • Matrixify

Outcomes

  • Taxonomy, mapping and pattern-match ~60% faster on 120k+ row sets
  • Pipeline holds CSV of 169,000+ rows without falling over
  • Fastify keeps API, schedules and reports; Go takes the compute
All cases