We didn't build a Discogs merch store - we built a pressings warehouse

7 min read
discogsmusicbrainzmongodbdata engineeringvinylpressings

Why we didn't build a Discogs merch store, and what we built instead: 4.7 million physical pressings joined to our catalogue, and a Pressings tab on every album page.


The brief started simply: can we sell merch sourced from Discogs? The answer turned out to be no — and the reason wasn’t technical. But chasing it produced something more interesting: a warehouse of 4.7 million physical pressings joined to our own catalogue, and a new tab on every album page that tells you every edition of a release that has ever existed.

Here is what we learned.

The licence wall

Discogs has a good API. It has artists, releases, a marketplace with prices, and a global seller inventory. It is exactly the wrong thing to build a shop on.

Three clauses in their API Terms of Use decide it:

  1. Marketplace data is “Restricted Data.” Prices, sales history and the images attached to offers for sale are not the freely licensed catalogue data. For Restricted Data the terms say you may not “use Restricted Data for any commercial purposes.” Affiliate monetisation is commercial use.
  2. A six-hour freshness rule. You may not display content more than six hours older than Discogs’ own copy, and may not cache it beyond what is necessary to serve a request. Our entire pattern is scrape → store → serve, which is the one thing explicitly forbidden.
  3. No traffic diversion. You may not use the API or its content “with the intent to drive traffic to other non-Discogs websites.”

And the kicker: Discogs has no affiliate programme. Even a fully compliant integration would have earned nothing.

There is also a functional problem that no amount of legal homework fixes. There is no marketplace-wide search and no release → listings endpoint. You can read one seller’s inventory at a time, or read aggregate counts per release. You cannot ask “who is selling this record?” across the marketplace. Building a shop from that would mean crawling every seller on the platform.

So: not a shop. But the other Discogs product — the monthly data dumps — is a different thing entirely. Those are released under CC0, No Rights Reserved. No attribution obligation, no commercial restriction, no freshness rule. They contain catalogue metadata only: no prices, no availability, no images.

That is a perfect match for a feature we actually wanted.

What a pressing actually is

Ask a music fan “how many versions of this album exist?” and you’ll get a shrug. Ask Discogs and you get a list: the 1987 UK 7″, the 1987 US 12″ maxi-single, the 1989 Japanese CD mini, the 2004 reissue, the 2021 limited blue vinyl, the 2025 zoetrope picture disc.

That data lives in the formats block of every release in the dump:

<formats>
  <format name="Vinyl" qty="1" text="">
    <descriptions>
      <description>12"</description>
      <description>45 RPM</description>
      <description>Maxi-Single</description>
    </descriptions>
  </format>
</formats>

Plus country, released, labels with catalogue numbers, identifiers (barcodes, matrix/runout etchings), and a master_id that groups every pressing of the same release together.

The releases dump is 10.5 GB compressed and represents 19.5 million releases. You do not want to load that.

The join that makes it cheap

The hard problem: our catalogue is keyed on MusicBrainz, and the dump is keyed on Discogs. Twenty-two thousand of our artists would need matching against twenty million releases, with no shared identifier.

Except there is a shared identifier — MusicBrainz records URL relationships, and most artists carry a link to their Discogs page:

GET /ws/2/artist/db92a151-...?inc=url-rels
  → discogs → https://www.discogs.com/artist/72872

That 72872 is exactly the <artist><id> value that appears inside every release in the dump. So the entire matching problem collapses into a set membership test:

  1. Read every catalogue artist’s MusicBrainz ID — 22,514 of them.
  2. Ask the local MusicBrainz mirror for each artist’s Discogs link.
  3. Build a hash set of Discogs artist IDs.
  4. Stream the 10.5 GB dump once, keeping any release whose artist ID is in the set.

No fuzzy matching. No name normalisation. No “is Queen the band or the singer?” nonsense.

Match rate: 69.3% — 15,597 of 22,514 artists. And it isn’t uniform: the well-known end of the catalogue matched at around 79%, the obscure tail at 59%. Which is the right shape, because that is where the traffic is.

The numbers, or: why you should measure

I estimated before building. I estimated we would keep about 1% of the dump, and that a typical artist would have about 40 pressings. Both were wrong, in the same direction.

EstimatedActual
Keep rate~1%24.2%
Pressings per artist~40median 152
Rows~2 million4,711,538

The keep rate is high because the artists we index own an enormous share of physical releases. The per-artist count is high because a popular record has genuinely been pressed dozens of times.

The whole extract runs in 18 minutes — 19,492,392 releases scanned, 4,711,538 kept, zero malformed rows — because the dump is decompressed straight off the socket and never written to disk.

Three bugs worth remembering

  1. Silent data loss from one line. The first version used ReadToFollowing("release") to walk the XML. Combined with XNode.ReadFrom, that skips the node the reader is already sitting on — so it silently processed every other release. Half the dataset, gone, with no error. It was caught only because I had built a four-release fixture instead of trusting a live run.
  2. Server GC on a memory-tight box. I set server garbage collection on the tool. On a workstation with 7.7 GB of RAM and under 1 GB free, .NET reserves per-core heaps and the process gets OOM-killed — twice, with no stack trace, no log line, nothing. Switching to workstation GC and adding a 512 MB heap hard limit turned a silent kill into a well-behaved stream. The real fix was running it on a host with room.
  3. The free tier wall. MongoDB Atlas M0 gives you 512 MB of storage — measured as uncompressed BSON plus indexes — and 100 operations per second. Our warehouse is 4.25 GB of BSON. Even if it fit, a full load would take about thirteen hours at that write rate. The free tier isn’t “small”, it’s a different class of product.

Where it ended up

The warehouse now runs as a MongoDB Windows service on the same server as the site, bound to loopback, so the API talks to it without crossing a network. Roughly 1.7 GB on disk for 4.7 million documents.

On every album page there is a Pressings tab, badged Discogs, showing a grid of editions newest first, ten at a time, with format, country, year, label, catalogue number and barcode. Each card links to the release on Discogs — a plain link, not API data, so no licence issue.

For one 1987 single we now surface 67 versions, from a 1988 Costa Rica 7″ to a 2025 zoetrope picture disc.

What I would take to the next one

Read the licence before the API docs. The technical blocker took twenty minutes to find. The legal one would have taken a month to discover the wrong way round.

Measure before you size infrastructure. A four-fold error in pressings-per-artist, and an eight-fold error in keep rate, together turned a comfortable two million rows into something that broke the plan.

Build the fixture. The one-line bug that halved the dataset produced no error, no warning and no suspicious log line. It was only visible against four known releases where I could count five.

Don’t estimate a dump. 10.5 GB compressed is 125 GB of XML and 19.5 million documents. Stream it. Never materialise it.

discogsmusicbrainzmongodbdata engineeringvinylpressings

Comments

No comments yet — be the first to share your thoughts!

Want to join the conversation?

Log in to comment
An unhandled error has occurred. Reload ??