Subtitles that survive a change of word order.
English is subject–verb–object. Korean is subject–object–verb. A correct translation moves words across cue boundaries — so translating each caption on its own produces subtitles that are both wrong and mistimed. This is the engine I built to fix that, for eight languages.
The verb decided is spoken first but must land last in Korean; the new budget is spoken second but must land first. Both cross a caption boundary, and the cue times change to match. Nothing here is illustrative — these are the timestamps the system emitted.
Why the obvious approach fails
The intuitive way to translate subtitles is to walk the file and translate each caption. That works for language pairs that share a word order, and it breaks immediately for pairs that don't. Translate “The committee decided” on its own and you have to guess at a verb ending you cannot know yet, because the object it governs is in the next caption and hasn't been read.
Ask a language model to translate the whole file at once and the reverse problem appears: the prose is fine, but it is no longer attached to time. You have a paragraph where you needed a sequence of captions, each of which has to appear while its words are being spoken, stay up long enough to read, never overlap its neighbour, and fit on two short lines.
So the work divides cleanly in two. One half is ambiguous and needs judgement. The other half is arithmetic under hard constraints, where being approximately right is the same as being wrong.
Translation and alignment
Produce natural target-language text, and say which source cue each output word came from. Judgement calls with no single right answer — exactly what a language model is good at.
Segmentation and timing
Cut that text into captions and assign start and end times that are monotonic, non-overlapping, inside the audio window, and readable. Guarantees, not preferences.
The bridge between the halves is a single idea: every output word carries an anchor — the moment its source audio was actually spoken. Because translation reorders, those anchors do not increase as you read. That is the whole problem, made into a number the algorithm can optimise against.
"""``anchor`` is the time (seconds) the source audio for this token was spoken. These need not increase with reading order: the verb's natural Korean position is at the end but its source audio was early.""" @dataclass class Token: text: str anchor: float break_cost_after: float = 0.0
Four stages, one of which is allowed to be uncertain
Transcribe
Audio is stripped from video, downmixed to 16 kHz mono, split under the API's size cap, and transcribed. Output is an ordinary subtitle file, so everything downstream is unchanged.
WhisperParse and group
A millisecond-accurate parser reads the file, then groups captions into whole sentences — the unit a translation can actually be correct about.
DeterministicTranslate and align
One call per sentence returns the translation plus a word-to-cue alignment, which becomes each token's anchor. Glossary terms are injected and checked.
ModelRe-cut and re-time
A dynamic program partitions the tokens into captions and solves for boundary times under hard constraints. No model involvement, and no randomness.
DeterministicBecause Stage 0 emits the same format the pipeline already accepted, a raw video file and an existing subtitle file are indistinguishable from Stage 1 onward. Speech recognition cost one new module and zero changes to the rest of the system.
Segmentation as an optimisation problem
Where to cut a sentence into captions has no off-the-shelf answer, so I modelled it the way typesetters model line breaking. A Knuth–Plass-family dynamic program searches every partition of the token sequence and returns the one with the lowest total cost. Because a span's cost depends only on that span, the problem has optimal substructure and the DP solves it exactly rather than greedily.
def segment_cost(tokens, i, j, start, end, cfg) -> float: """Cost of making tokens[i:j] a single cue occupying [start, end]. Four competing terms. Local by construction, which is what gives the DP optimal substructure.""" cost = cfg.cue_penalty cost += cfg.w_reading * _reading_penalty(chars, duration, cfg) # too fast to read cost += cfg.w_line * _line_penalty(text, cfg) # too wide for the frame cost += cfg.w_split * _split_penalty(span, cfg) # cuts mid-phrase cost += cfg.w_drift * _drift_penalty(span, center) # drifts off the audio return cost
Reading speed, line length, split quality and timing drift pull against each other; the weights decide who wins. A second pass then solves for the boundary times themselves, where reading order is a hard constraint and timing fidelity is the soft one.
The payoff for keeping this half deterministic is that timing stops being a matter of opinion. Monotonic ordering, non-overlap and minimum on-screen duration are invariants the algorithm cannot violate, which makes them testable — and they are tested, on every run, against a fixed corpus.
Eight languages without eight pipelines
The system started as English → Korean. Extending it could easily have meant forking the pipeline per language; instead everything language-specific lives in a profile: prompt wording, register options, reading-speed and line-length norms, the word-segmenter for scripts that don't use spaces, and whether the script runs right to left.
The deterministic machinery never learned a language. It scales per language, not per pair — only the prompt is pair-aware, and it is assembled from the two profiles. Adding a language is one profile and a quality-gate run, not a rewrite.
How I know whether it actually works
“The translation looks good” is not a claim anyone should accept, including me. So quality is scored by a gate that runs against a golden set of clips. One layer is structural and deterministic — timing invariants, reading-speed compliance, line-length compliance, glossary adherence. The second layer asks a model to score adequacy and fluency, which catches the failures arithmetic can't see.
A language is promoted out of Beta only when it clears every threshold. One of them doesn't, and it ships labelled Beta because of it.
| Target | Reading speed | Timing | Verdict | Ships as |
|---|---|---|---|---|
| Korean | pass | 1.00 | PASS | GA |
| Japanese | pass | 1.00 | PASS | Beta |
| Chinese | pass | 1.00 | PASS | Beta |
| Spanish | pass | 1.00 | PASS | Beta |
| Russian | pass | 1.00 | PASS | Beta |
| Arabic | pass | 1.00 | PASS | Beta |
| French | 0.875 | 1.00 | FAIL | Beta |
French fails on reading speed at 0.875 against a 0.90 threshold. The cause is intrinsic: French renders the same meaning in more characters, so more captions run past a comfortable reading rate in the time the audio allows. It is documented rather than papered over, and it stays in Beta until condensation improves.
Underneath the gate sits an ordinary test suite that runs entirely offline — a static translator stands in for the model, so the whole pipeline, service and job lifecycle are exercised with no API key and no network.
The parts that make it a product
The engine is a pure library with no web framework in it. A FastAPI service wraps it and a Next.js editor consumes a frozen, typed contract, so all three evolve independently.
Editing without inference
Re-timing edited text re-runs only the algorithm, never the model — so corrections are instant and free. A single caption can be re-translated on its own for one call.
A human gate on transcription
Speech recognition stops at a review step. Nothing is translated until a person has approved the transcript, because translation quality is capped by transcript quality.
Terminology that holds across a series
A glossary is injected into the prompt and checked afterwards, flagging terms that didn't survive. A whole brief can be saved and reused across episodes.
Jobs that survive a restart
Live state serialises losslessly to SQLite or Postgres, and a startup sweep fails anything orphaned by a dead process instead of leaving clients polling forever.
Provider independence
Routing by model name drives any OpenAI-compatible endpoint, so the same code runs against a frontier model or a budget one, chosen per job.
Costs quoted before you spend them
Both file and media uploads are priced up front from token and duration estimates, with no inference call needed to produce the quote.
What I'd want you to take from this
The interesting decision in this project was not which model to call. It was working out which half of the problem a model should be nowhere near — and then building that half properly, as an algorithm with guarantees I could test, measure and report honestly, including where it falls short.