Sheet Music Studio

Why I built it

My 12-year-old asked me to help him find sheet music for the songs he wanted to play. I looked in several places, and either the piece wasn't there or it was very expensive. So I built Sheet Music Studio — first to support my kid, then to make it available to everyone in a more accessible way, while validating my go-to-market skills on a product built around AI.

What it does

Upload a recording (MP3 or OGG), choose one of 22 instruments — piano and organ, guitar and bass, strings, woodwinds, brass, four voice types, drum kit — and get engraved sheet music at three difficulty levels: Original, Intermediate and Beginner. Playback follows the score with a moving cursor, and can switch to the original recording at 50–100 % speed to practice slowly. Transpose, adjust key, tempo and meter, attach lyrics, organize pieces into collections, and export PDF, MusicXML or MIDI. In English and Portuguese, with a free plan and paid plans.

Live at sheetmusic.studio.

How the AI works

A seven-stage, versioned pipeline turns audio into a score:

1. Ingest & decode — the file is checked by its content (not its name) and decoded to 44.1 kHz audio; silent or too-short recordings are rejected.
2. Source separation — for a full band mix, Demucs (a deep-learning model) isolates the part that matters for the chosen instrument: vocals, bass, drums or the rest.
3. Transcription — Spotify's basic-pitch model turns audio into notes, with detection thresholds tuned per instrument family on a test corpus (voice onset F1 0.73 → 0.82, strings 0.85 → 0.90); keyboards use a dedicated piano-transcription model.
4. Verification — for single-line instruments (flute, voice, sax…), a second, independent pitch tracker (pYIN) checks the melody note by note.
5. Analysis — beat tracking, tempo, meter, downbeat and key detection, following rubato when the music doesn't keep a straight tempo.
6. Engraving — music21 writes the score for the instrument's range and clefs, at three difficulty levels.
7. Rendering — Verovio engraves the pages to SVG and PDF.

Every transcription records the model, its checksum and parameters, so any score can be reproduced. Notes judged to be artifacts (ghost notes, harmonics, vibrato re-triggers, accompaniment) are tagged rather than deleted, so switching instrument re-transcribes for that instrument without losing anything.

Architecture

browser ─► Firebase Hosting ── /api/** ──► API (Cloud Run, public)
                                              │ Cloud Tasks (OIDC)
                                              ▼
       Cloud SQL Postgres ◄──── worker (Cloud Run, Python + ML, private)
       Cloud Storage      ◄────┘

• Web — a React + TypeScript single-page app on Firebase Hosting; scores rendered in the browser with OpenSheetMusicDisplay and played with a Web Audio sampler (88-key keyboard, metronome, mixer, loops). The API is served same-origin under /api, so there's no CORS and no separate domain.
• API — Node 22 + Express on Cloud Run. Owns sign-in (Firebase Auth: email and Google), ownership checks, uploads, quotas, Stripe billing and email (Postmark). It never imports ML code.
• Worker — Python 3.12 + FastAPI on a private Cloud Run service, reachable only through Cloud Tasks with an OIDC identity. One task per conversion, named by its id so a duplicate dispatch is a no-op; retries with backoff, a 15-minute deadline, and stuck jobs failed automatically so the user gets the song back.
• Data — Postgres (Cloud SQL) for accounts, pieces, collections, conversion history and quotas; Cloud Storage for audio and generated scores, never exposed directly to the browser. Deleting a piece or an account removes everything under its storage prefix.
• Infrastructure as code — Cloud Run, Cloud Tasks, Cloud Scheduler, buckets, secrets, IAM and domains are all Terraform; releases go through Cloud Build.

Quality, security and go-to-market

• Quality gate in CI — worker tests score transcriptions against a reference corpus, next to API integration tests, web component tests and Playwright end-to-end tests on every push.
• Security — automated scans for vulnerable packages, risky code patterns and leaked secrets on every push; per-account quotas and rate limits; immediate, complete account deletion.
• Go-to-market — public home and pricing pages pre-rendered for search engines, Stripe subscriptions with a customer portal, consent-based analytics, an admin dashboard (active users, plans, revenue, conversion failures, ratings) and an in-app feedback loop on every score.

Stack

React 18 · TypeScript · Vite · OpenSheetMusicDisplay · Web Audio · Node 22 · Express · PostgreSQL · Python 3.12 · FastAPI · PyTorch · Demucs · basic-pitch · librosa · music21 · Verovio · Firebase Auth · Stripe · Postmark · Google Cloud Run · Cloud Tasks · Cloud SQL · Cloud Storage · Cloud Build · Terraform