Logo

Video Translation Dubbing

Upload a video or audio file, specify the source language, target language and voice, and the server translates what is said into the target language, re-dubs it line by line, and lays each line back where it was spoken in the original: one consistent speaking rate across the whole piece, with subtitles timed to the dub. Video input returns a dubbed video plus editable project files; audio input returns dubbed audio.

This capability replaces the audio track only and never changes the picture: no lip sync, no re-cutting.

The dub sounds even because translation already targets each line's original duration, and after synthesis the whole piece uses a single global speed factor with only narrow per-line adjustment. Any remaining difference is absorbed in the order borrow from gaps → neighbour yield → rewrite → boost → freeze frame or trim, and every step is recorded in the output report and the per-line alignment record.

How it differs from related capabilities:

  • AI Subtitle: recognizes and translates subtitles, no dubbing.
  • Zero-Shot Voice TTS: speaks a given text, no translation and no alignment to a source video.

Create Task

Basics

ItemValue
MethodPOST
Path/task/video_translate_dub
Content-Typeapplication/json
AuthAuthorization header (pass the API Key directly)

Request Body

FieldTypeRequiredDefaultDescription
file_idstringYes—Source file ID (an uploaded video or audio file), up to 120 minutes long. Audio input returns dubbed audio; see "Audio input" below
source_langstringYes—Source language as a standard region language code. Currently supported: zh-CN, zh-HK, en-US, ja-JP, ko-KR, fr-FR, de-DE, es-MX, pt-BR, th-TH, id-ID, ms-MY, fil-PH, ar-SA (zh-CHS / zh-CHT are treated as zh-CN). Unsupported values return a parameter error listing the usable languages; there is no auto-detection, so pass the actual language
target_langstringYes—Target language as a standard region language code. The valid set depends on speaker: with speaker=clone it is zh-CN, zh-HK (Cantonese), en-US, ja-JP and ko-KR; with a voice ID it must be one of the languages registered for that voice (see the voice catalog). Unsupported values return a parameter error listing the usable languages
speakerstringYes—Voice. clone clones the speaker in the source (the server picks a clean single-speaker segment from the source speech as the reference; you can also supply one with ref_file_id); a voice ID from the voice catalog dubs with that voice. Some internal voices are available to matrix members only. By submitting this task you confirm that, per the User Agreement, you have obtained the separate consent of the person whose voice is being cloned and will not use the generated speech for impersonation or similar purposes
ref_file_idstringNopicked from the sourceFile ID of the clone reference media, meaningful only with speaker=clone (ignored for other voices). Audio or video, 3 seconds to 30 minutes long, containing at least 3 seconds of continuous, clear, single-speaker speech. When provided, the reference segment is picked from this media only. Reference length is not billed. Same consent notice as speaker
ref_langstringNosame as source_langLanguage of the reference segment, meaningful only with speaker=clone: zh-CN, zh-HK (Cantonese), en-US, ja-JP or ko-KR
keep_bgmbooleanNotrueWhether to keep the source background music. When the source really has background music, the output keeps it and a separate music track is returned; speech-only sources are not separated and the output audio is the dub track
fit_policystringNonaturalDuration fitting policy. natural: a small amount of freeze-frame extension is allowed and the output is at most 103% of the source duration; strict: the output has exactly the source duration (within 1 frame), and lines that do not fit are trimmed at the tail, by at most 0.3 seconds each
speed_bandnumber[]No[0.95, 1.15]Allowed range [min, max] for the global speed factor, with 0 < min ≤ max ≤ 1.3; out-of-range values return a parameter error and are never silently clamped. Lines are never slowed down individually: when a translation is short, the difference stays as a pause
subtitlestringNosoftSubtitle mode: none produces no subtitles; soft returns a subtitle file without burning it in; burn burns the translation into the video using subtitle_type; bilingual_burn burns translation and source text as two lines and also returns a bilingual subtitle file. Audio input supports only none and soft
subtitle_typestringNodefaultBurned-in subtitle style, effective only with burn / bilingual_burn; same values as AI Subtitle: default / outline / cinema_yellow / immersive_box / wide_spacing / deep_shadow / boxed (case and spelling are normalized); unrecognized values fall back to the default style
project_formatsstring[]No["gtrk"]Project file formats, same values as Smart Oral Cut: gtrk (Gitruck client project) / xml (Premiere Pro) / fcpxml (Final Cut Pro) / otio (OpenTimelineIO) / jianying (Jianying) / capcut (CapCut); multiple values allowed, one files[] entry per format. Audio input produces no project files and ignores this field
source_pathstringNo—Full local path of the source file in your editing environment (e.g. D:/clips/talk.mp4), written as-is into the project as the source media path; the server does not check that it exists or rewrite it. Placeholder file name when omitted. Ignored for audio input
local_output_dirstringNo—Local folder you download this task's outputs into; when given, the dub track, background track and extended base video are referenced as that folder + dub.wav / bgm.wav / base.mp4 (keep those file names when downloading). Placeholder file names when omitted. Ignored for audio input
struct_metaobjectNo—Same as Smart Oral Cut: nle_draft_dir is the full path of this draft's folder inside the Jianying / CapCut drafts directory (legacy name capcut_draft_path also accepted). Only then is draft_meta_info.json produced, which Jianying / CapCut need to list the draft

Fields not listed above are ignored and defaults apply.

Billing

  • Billed by the source file duration in minutes (as measured by the server), prorated and rounded up to a whole credit; every task costs at least 1 credit.
  • The voice does not change the price: the same source costs the same with clone or any voice.
  • Reference media duration is not billed.
  • See "Video Translation Dubbing" on the pricing page for the unit price (currently 50 credits per minute; the pricing page is authoritative).
  • Parameter errors at creation create no task and cost nothing; if a task fails during processing, the pre-deducted credits are fully refunded.

Request Example

curl -X POST https://api.ai-mcn.tv:10000/task/video_translate_dub \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "file_id": "537489015178246",
    "source_lang": "zh-CN",
    "target_lang": "en-US",
    "speaker": "clone",
    "fit_policy": "natural",
    "subtitle": "soft",
    "project_formats": ["gtrk", "jianying"]
  }'

Success Response Example

{
  "code": 200,
  "msg": "success",
  "data": {
    "task_id": "537489015178247",
    "task_type": "video_translate_dub",
    "status": "queued"
  }
}

Query Task Result

Basics

ItemValue
MethodGET
Path/task/video_translate_dub/{task_id}
AuthAuthorization header (pass the API Key directly)

Response Fields (output_result)

The dubbed output and the pure dub track are core outputs; the music track, subtitles, alignment record and project files are derived outputs — a failure in any one of them does not fail the task, and the reason is recorded under the matching key in errors. Per-line details live only in the alignment record file, so the size of output_result does not grow with the source duration.

FieldTypeDescription
output_file_idstringFile ID of the dubbed video (mp4); for audio input, the dubbed audio (mp3), with the original background music mixed in when kept
output_file_download_urlstringDownload path of the dubbed video or audio
dub_audio_file_idstringFile ID of the pure dub track (wav, no background music)
dub_audio_file_download_urlstringDownload path of the pure dub track
bgm_audio_file_idstringFile ID of the music track (wav). Present only when keep_bgm=true and the source really has background music
bgm_audio_file_download_urlstringDownload path of the music track
base_video_file_idstringFile ID of the extended base video (silent, no subtitles). Present only when freeze-frame extension happened under natural; project files use it as the base
base_video_file_download_urlstringDownload path of the extended base video
subtitle_file_idstringTranslated subtitle file (.srt) ID; empty string when subtitle=none or generation failed (see errors.subtitle)
subtitle_file_download_urlstringDownload path of the translated subtitles
bilingual_subtitle_file_idstringBilingual subtitle file (.srt, two lines per cue: translation above, source below) ID; produced only with subtitle=bilingual_burn, otherwise an empty string
bilingual_subtitle_file_download_urlstringDownload path of the bilingual subtitles
transcript_file_idstringFile ID of the per-line alignment record (UTF-8 JSON); fields below
transcript_file_download_urlstringDownload path of the alignment record
reportobjectBounded summary; fields below
filesarrayProject files, same shape as Smart Oral Cut: each entry is {type: "project", format, file_id, download_url, filename} (e.g. gtrk → project.gtrk). Empty for audio input
errorsobjectPartial failure reasons for derived outputs; possible keys: bgm / base_video / subtitle / bilingual_subtitle / transcript / project. An empty object when everything succeeded

Common report fields

FieldTypeDescription
input_kindstringSource type: video or audio
source_lang / target_langstringSource / target language
speakerstringVoice (clone or a voice ID)
fit_policystringDuration fitting policy
rationumberGlobal speed factor for the whole piece
over_band_linesintegerNumber of lines outside the speed band that needed fallback handling
borrowed_total_msintegerTotal milliseconds borrowed from neighbouring gaps
yielded_total_msintegerTotal milliseconds yielded by shorter neighbouring lines
delayed_linesintegerNumber of lines whose start was delayed because the previous line was extended
rewrite_linesintegerNumber of lines that were compressed by rewriting
rewrite_rounds_histobjectDistribution of rewrite rounds ("0" / "1" / "2" → line count)
freeze_total_msintegerTotal freeze-frame milliseconds (for audio input, total inserted silence)
freeze_pointsarrayEach freeze as [source time in ms, freeze ms]
freeze_cap_hitbooleanWhether total freezing reached the 3% cap (beyond it, lines are trimmed instead)
trimmed_linesintegerNumber of trimmed lines
output_duration_ms / source_duration_msintegerOutput (video or audio) duration / source duration, in ms
bgmstringBackground music result: kept; none (speech-only source or keep_bgm=false); dropped (requested but the music track could not be produced, see errors.bgm)
bgm_rationumberMeasured background-music energy ratio used for the decision
subtitle / subtitle_burnedstring / booleanSubtitle mode / whether subtitles were burned in
lines_countintegerNumber of lines
refobjectReference segment used for cloning: span_ms (start/end ms) / separated / gain_db / text_source / asr_confidence / ref_text / ref_lang / source (source_video picked from the source, ref_file picked from the uploaded reference); field meanings as in Zero-Shot Voice TTS. null for non-clone voices

Per-line alignment record (JSON behind transcript_file_id)

Top level {task_type, source_lang, target_lang, lines}; each entry of lines:

FieldTypeDescription
indexintegerLine index, starting at 0
clip_idstringID of this line's clip on the project dub track, for locating a line when you edit it; empty for audio input (no project file)
src_textstringSource text
src_st_ms / src_ed_msintegerStart / end of the source line in the source, in ms
tgt_textstringTranslation (the rewritten text for rewritten lines)
budget_charsintegerCharacter budget for this line's translation
synth_msintegerDuration of the audio actually laid on the dub track (after speed change, if any)
st_ms / dur_msintegerStart and duration of this line's dub in the output, in ms; subtitle cues use exactly these
speednumberSpeed factor of this line
borrowed_before_ms / borrowed_after_msintegerMilliseconds borrowed from the gap before / after
yielded_before_ms / yielded_after_msintegerMilliseconds taken from the spare time of the previous / next line (only when that line is itself short)
delay_msintegerMilliseconds this line starts later because the previous line was extended, at most 500
pad_before_ms / pad_after_msintegerPause kept before / after the line, in ms
freeze_ms / trim_msintegerFreeze / trim milliseconds for this line
rewrite_roundsintegerRewrite rounds for this line (0–2)
actionstringFinal handling of the line: fit fits at the global speed; borrow borrows neighbouring gaps; yield fits after gap borrowing plus spare time yielded by a shorter neighbouring line; boost raises the speed factor; freeze extends with a freeze frame; trim trims the tail. Rewritten lines are still recorded by their final action; see rewrite_rounds for how often they were rewritten

.gtrk project file

When project_formats contains gtrk, the files[] entry with format=gtrk is a gtrk v1 unified editing contract. Open it in the Gitruck client, or send it back to Project Struct to convert it to other NLE projects:

  • Video track: one full-length, muted base clip. When freeze-frame extension happened, the base is the extended base video (base_video_file_id); otherwise it is the source.
  • Audio track 0: the dub track, one clip per line; each clip's span on the track equals that line's dub start/end, and clip IDs match clip_id in the alignment record one to one.
  • Audio track 1: the music track as one clip (when there is background music).
  • With source_path and local_output_dir, media paths in the project are your local paths — download into that folder and open directly. Without them they are file-name placeholders (the source file name, dub.wav, bgm.wav, base.mp4); download the outputs into one folder and relink when prompted.
  • For a Jianying / CapCut draft, also pass struct_meta.nle_draft_dir and put draft_content.json and draft_meta_info.json into that folder so the app lists the draft.

Success Response Example

{
  "code": 200,
  "msg": "success",
  "data": {
    "task_id": "537489015178247",
    "status": "completed",
    "progress": 100,
    "output_result": {
      "output_file_id": "537489015178301",
      "output_file_download_url": "/download/c1/537489015178301.mp4",
      "dub_audio_file_id": "537489015178302",
      "dub_audio_file_download_url": "/download/c1/537489015178302.wav",
      "base_video_file_id": "537489015178303",
      "base_video_file_download_url": "/download/c1/537489015178303.mp4",
      "subtitle_file_id": "537489015178304",
      "subtitle_file_download_url": "/download/c1/537489015178304.srt",
      "bilingual_subtitle_file_id": "",
      "bilingual_subtitle_file_download_url": "",
      "transcript_file_id": "537489015178305",
      "transcript_file_download_url": "/download/c1/537489015178305.json",
      "report": {
        "input_kind": "video",
        "source_lang": "zh-CN",
        "target_lang": "en-US",
        "speaker": "clone",
        "fit_policy": "natural",
        "ratio": 0.95,
        "over_band_lines": 1,
        "borrowed_total_ms": 200,
        "yielded_total_ms": 0,
        "delayed_lines": 0,
        "rewrite_lines": 5,
        "rewrite_rounds_hist": {"0": 3, "1": 2, "2": 3},
        "freeze_total_ms": 431,
        "freeze_points": [[2100, 392], [6160, 39]],
        "freeze_cap_hit": false,
        "trimmed_lines": 0,
        "output_duration_ms": 35431,
        "source_duration_ms": 35000,
        "bgm": "none",
        "subtitle": "soft",
        "subtitle_burned": false,
        "lines_count": 8,
        "ref": {
          "span_ms": [4000, 12900],
          "separated": false,
          "gain_db": -4.26,
          "text_source": "asr",
          "asr_confidence": 0.96,
          "ref_text": "镜头背后有没有任何人？你不就面对一个镜头吗？",
          "ref_lang": "zh-CN",
          "source": "source_video"
        }
      },
      "files": [
        {"type": "project", "format": "gtrk", "file_id": "537489015178306", "download_url": "/download/c1/537489015178306.gtrk", "filename": "project.gtrk"}
      ],
      "errors": {}
    },
    "create_time": "2026-09-17T02:20:03Z",
    "update_time": "2026-09-17T02:35:53Z"
  }
}

Alignment record example (the file behind transcript_file_id), one line shown:

{
  "task_type": "video_translate_dub",
  "source_lang": "zh-CN",
  "target_lang": "en-US",
  "lines": [
    {
      "index": 1, "clip_id": "537489015178302-0002",
      "src_text": "镜头背后有没有任何人？", "src_st_ms": 3804, "src_ed_ms": 6804,
      "tgt_text": "Nobody is behind the camera, you just face a lens.", "budget_chars": 46,
      "synth_ms": 3783, "st_ms": 3413, "dur_ms": 3783, "speed": 1.0,
      "borrowed_before_ms": 83, "borrowed_after_ms": 83,
      "yielded_before_ms": 308, "yielded_after_ms": 309, "delay_ms": 0,
      "pad_before_ms": 0, "pad_after_ms": 0, "freeze_ms": 0, "trim_ms": 0,
      "rewrite_rounds": 0, "action": "yield"
    }
  ]
}

Audio input

When file_id points to audio (e.g. mp3, wav, m4a, ogg), the same translation pipeline runs, with different outputs:

  • output_file_id is the dubbed audio in the target language (mp3), with the original background music mixed in when kept; the pure dub track, music track (if any), subtitle file and alignment record are also returned.
  • Burned-in subtitles are not supported: subtitle set to burn or bilingual_burn returns a parameter error; no task is created and nothing is billed.
  • No project files: files is an empty array, project_formats is ignored, and clip_id in the alignment record is an empty string.
  • When natural needs to extend the duration, an equal-length pause is inserted at the same position in both the dub and the music, and the total still stays within 103% of the source.
  • Language rules, voices, duration limit and billing are the same as for video.

Error Codes

CodeHTTP StatusDescriptionSolution
6013400Required parameter missing (file_id / source_lang / target_lang / speaker)Add the missing parameter
6016400Invalid business parameter: unsupported language, unknown or unavailable voice, invalid fit_policy / subtitle / speed_band / keep_bgm / project_formats, reference media shorter than 3 seconds or longer than 30 minutes, burned-in subtitles requested for audio input, etc. (the response states the exact reason)Fix the parameter as described
6019400Invalid source duration or longer than the limit (over 120 minutes)Split the file and resubmit
6014400File type does not match the endpoint (video or audio required)Upload a video or audio file
6004404Source or reference file not foundCheck file_id / ref_file_id
6502401Authentication failedCheck the Authorization header
6201402Insufficient quotaTop up on the dashboard
6202402Insufficient balanceTop up on the dashboard

All of the errors above happen at task creation: no task is created and nothing is billed. Failures during processing (translation failure, dub synthesis failure, or no qualifying single-speaker segment found for cloning in the source or reference) set the task status to failed, refund the pre-deducted credits, and record the reason in the task error message.

Limits

  • Duration: the source is at most 120 minutes. Long sources take a while; polling every 10–15 seconds is recommended.
  • One voice: the whole piece is dubbed with a single voice; per-speaker dubbing of multi-person conversations is not supported yet. For cloning, the server picks single-speaker segments from the source as the reference.
  • Picture is not changed: only the audio track is replaced and there is no lip sync. Under natural, an occasional freeze frame is a brief hold of the picture, at most 3% of the source duration in total; use strict if that matters to you.
  • Speaking rate: the global speed factor stays within speed_band; short translations keep their pauses and are never slowed down line by line.
  • Line starts: when a translation runs long, it first uses spare time from a shorter neighbouring line, so that line may start slightly before the on-screen speaker, or the next line slightly later, by at most 500 ms on each side; yielded_* and delay_ms in the alignment record report it exactly.
  • Background music: music is separated only when its energy ratio reaches the decision threshold; speech-only sources are not separated. The separated music is mixed with the dub as the output audio.
  • Clone quality: zero-shot similarity depends heavily on reference quality, and cross-language cloning (e.g. a Chinese reference dubbed into English) is less similar than same-language cloning. When voice fidelity matters, provide a clear, dry, single-speaker reference with ref_file_id.
  • Subtitles: cue start/end times match the per-line dub exactly; punctuation is stripped by default (same as AI Subtitle).
  • When the generated speech or video is published or distributed, it must be clearly labelled as AI-generated content as required by law.

🤝 Join the Tonghe New Media Network to unlock internal member benefits: network members (internal) get custom-scope full-library material search (video / image / audio, including concept and non-commercial materials, with a commercial-only filter) and exclusive internal voices. Apply at the Creator Network; once approved, an administrator will enable internal access for you.