Nemotron 3.5 Transcribe: self-host it or pay for the API?

What it needs: one card, and not the cheapest one on the board

Two numbers on this page carry exact meanings, and both are printed wherever they are used. The planning assumption: the open-source serving stack processes audio at 100× real-time — 100 seconds of audio per second of card time. NVIDIA’s published peak for its optimized build is 2,238× on one RTX PRO 6000, and NVIDIA itself says the downloadable open-source stack is expected to run roughly two times slower than that build; we plan at 100× to leave room for long files, speaker labels and unbatched work. Sensitivities at 50× and 200× are in the cost table. An assumption is not a benchmark — nobody has published an independent throughput measurement for this model yet. Memory sizing follows our standard method — total parameters × bytes per parameter, plus about 20% headroom — and matches the sizing on this model’s catalogue page. Nemotron 3.5 Transcribe is one open-weight checkpoint (nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B) that takes a recording of up to one hour plus a plain-language instruction and returns a transcript shaped by that instruction: verbatim or cleaned-up text, speaker tags for up to eight speakers, word-level timestamps, translation between English and 31 languages, or answers to questions about the recording, across 40 languages. The checkpoint holds 34 billion parameters in total, including its audio and speaker encoders, but only about 4.3 billion are active per token — which is why one GPU can carry it. It is an offline model: finished recordings, not live streams. The accuracy headline is NVIDIA’s own and is labelled that way throughout: NVIDIA reports a 5.33% average word error rate on the Hugging Face Open ASR English leaderboard — first among open-weight models — and its card’s results table (captured Oct 12, 2026) shows a 4.71% mean across the nine English sets, 4.62% multilingual and 10.98% on long-form. No independent reproduction of those figures had been published when we checked.

The three builds, sized

By our standard method, the three builds size out as follows:

Memory alone is not the whole test, because NVIDIA publishes a hardware compatibility table on the model card — the list of GPUs its runtimes are actually built and tested for. That table, read Nov 1, 2026, is the next one.

NVIDIA’s own compatibility table — and the card that is not on it

The GPUs NVIDIA’s runtimes are built and tested for, with the cheapest verified rental for each (checked Oct 31, 2026):

The smallest sensible rented card is the RTX 5090. Its 32GB holds the ~17GB NVFP4 build with room to spare, it is on NVIDIA’s supported list (vLLM, Transformers and GGML runtimes), and at $0.69/hr it is the cheapest supported route on the board. The honest catch sits one row below it in price: the RTX 4090, at $0.34/hr on RunPod Community, is the cheapest card we list and its 24GB would hold the NVFP4 build by memory — but the 4090 is NOT in NVIDIA’s compatibility table, and its Ada-generation hardware has no native FP4 path in the official runtimes. Community builds may close that gap in time; today, planning a deployment on the 4090 means planning on unsupported ground. We do not recommend it as the plan.

What an hour of audio costs

The API side is a single published number: DeepInfra — one of the hosts NVIDIA’s launch names, serving the model through an OpenAI-compatible speech-to-text endpoint — lists $0.00200 per minute of audio on its own model page (read Nov 1, 2026): $0.12 per hour of audio, $120 per 1,000 hours, $1,200 per 10,000 hours. Requests can run to 60 minutes; transcription, word and segment timestamps, contextual prompts and translation into English are live today. The self-host columns are arithmetic on the printed assumption:

Read the self-host columns as arithmetic on an assumption, not as measurements: at 100× real-time, one hour of audio takes 36 seconds of card time, so 1,000 hours of audio is 10 card-hours. If the real stack runs at half that speed on your audio, the 50× column is your cost; at double, the 200× column. Even the slowest column beats the API rate on every card — the gap is wide enough that the assumption would have to be wrong by more than an order of magnitude before the ranking flipped.

The break-even: it’s about the card’s month, not the minute

Rent by the hour and shut the card down between batches, and self-hosting is cheaper per hour of audio from the first batch — the only real minimum order is your own setup time (downloading a 17–68GB checkpoint and standing up vLLM with the NeMo Speech plugin). The break-even that matters is the other rental shape: a card kept for a full month (730 hours) whether or not audio is flowing. Against the API’s $0.12 per hour of audio, a month of card has to process this much audio before it has paid for itself:

Below those volumes, the API wins on money alone — and wins outright on everything operational: no setup, no idle card, no stack to patch. Above them, self-hosting wins on money and on three things the API cannot sell you at any price today: speaker labels (in the open model now; DeepInfra lists them as coming soon in its API), your audio never leaving your own machine (DeepInfra states zero data retention, which is a policy promise rather than an architecture), and fine-tuning — the NeMo Speech toolkit adapts the model to your vocabulary and accents, which no hosted endpoint offers. One more honest line in the API’s favour: a 10,000-hour archive costs $1,200 there, against $69 of card time on a 5090 at the planning assumption — but the $69 route also costs the days of engineering to build and babysit the pipeline. Price your own time before you call it cheaper.

The serving reality: a published recipe, and a faster build you can’t download yet

The open serving path is real and documented on the model card. NVIDIA’s quick start serves the checkpoint on one H100 80GB with vLLM 0.32.0 plus the NeMo Speech plugin (VLLM_PLUGINS=nemo_speechlm), exposing an OpenAI-compatible endpoint — code written against OpenAI-style chat and audio APIs ports over with small changes. SGLang-Omni 0.2.0 is listed for H100 and the RTX PRO 6000, Hugging Face Transformers runs it in-process, and GGML covers the RTX 5090 and RTX PRO 6000, including native Windows. The card’s own configuration turns on native multi-token prediction for speed, with an optional DFlash2 draft model for speculative decoding. Then the caveat, in NVIDIA’s own words on the same card: its fastest published serving numbers come from the Nemotron Transcribe NIM, NVIDIA’s optimized container build — and “these serving optimizations are not yet part of the open-source release”. NVIDIA says the open-source vLLM integration that shipped on release day “is intended for model development and is expected to be roughly two times slower”. So the stack you can download today is the development-grade one, by the maker’s own description; the optimized build sits behind NVIDIA’s NIM route and its own terms. Budget on the open stack; treat anything faster as a bonus. NVIDIA’s own published figures, for the one card it measured: on the model card’s Efficiency section, NVIDIA reports that its optimized NIM build on one RTX PRO 6000 transcribes up to 2,238 seconds of audio per second (NVFP4 with DFlash2 speculative decoding), and returns a 30-second transcript in 0.13 seconds. Priced at the verified $1.69/hr, that peak works out to about $0.0008 per hour of audio — $0.76 per 1,000 hours. At the roughly-two-times-slower speed NVIDIA states for the open-source stack (~1,119×), it is about $0.0015 per hour of audio, $1.51 per 1,000 hours. Both figures are the maker’s, the first is an “up to” peak, and the NIM build behind it is not the downloadable one. We show them because they are published and checkable — not as the basis for a budget.