Audio-Visual Speaker Detection

Detect visible speakers in a video. The service automatically runs internal ASR, then fuses transcript sentences, visual person tracks, and speaking segments into one structured result. It is useful for interviews, meetings, podcasts, variety-show footage, and any workflow that needs to know who spoke when.

Example Video

Create Task

Basic Information

Item	Value
Request Method	`POST`
Request Path	`/task/video_speaker_detect`
Content-Type	`application/json`
Authentication	Raw API key in the `Authorization` header

Request Body

Parameter	Type	Required	Default	Description
`file_id`	`string`	Yes	-	Uploaded video file ID
`language`	`string`	No	`zh-CN`	Language code used by the internal ASR task
`max_faces_per_frame`	`integer`	No	`5`	Maximum faces kept per sampled frame, range `1-10`
`detect_body`	`boolean`	No	`false`	Whether to run additional person/body detection and return `body_bbox`
`track_sample_fps`	`number`	No	`5`	Maximum public track points returned per second, range `1-30`; the actual returned rate is automatically capped by the video and internal analysis rate

Request Example

curl -X POST https://api.ai-mcn.tv:10000/task/video_speaker_detect \
  -H "Authorization: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "file_id": "537489015178246",
    "language": "zh-CN",
    "max_faces_per_frame": 5,
    "detect_body": true,
    "track_sample_fps": 3
  }'

Success Example

{
  "code": 200,
  "msg": "success",
  "data": {
    "task_id": "537489015178247",
    "task_type": "video_speaker_detect",
    "status": "queued"
  }
}

Query Task

Basic Information

Item	Value
Request Method	`GET`
Request Path	`/task/video_speaker_detect/{task_id}`
Authentication	Raw API key in the `Authorization` header

Result Fields

Field	Type	Description
`version`	`string`	Output schema version, for example `video_speaker_detect.v1`
`video`	`object`	Video width, height, duration, original FPS, and analysis FPS
`params`	`object`	Effective runtime options for this task
`transcript`	`object`	Normalized internal ASR text, sentences, and optional word timeline
`persons`	`array`	Summary of detected visual persons
`speaking_segments`	`array`	Visual speaking ranges with scores and aggregate boxes
`sentence_segments`	`array`	Sentence-level visual speaker attribution
`tracks`	`array`	Sampled person track points. Moving people have time-varying `face_bbox` / `body_bbox` values

Bounding boxes are normalized [x1, y1, x2, y2] values in the 0-1 range.

Completed Result Example

{
  "version": "video_speaker_detect.v1",
  "video": {
    "width": 1920,
    "height": 1080,
    "duration_ms": 18000,
    "source_fps": 25,
    "analysis_fps": 12
  },
  "params": {
    "target_analysis_fps": 25,
    "analysis_fps": 12,
    "max_faces_per_frame": 5,
    "detect_body": true,
    "track_sample_fps": 3
  },
  "persons": [
    {
      "person_id": "person_1",
      "first_seen_ms": 0,
      "last_seen_ms": 17500,
      "visible_frame_count": 120,
      "speaking_duration_ms": 6400,
      "avg_speaking_score": 0.72,
      "max_speaking_score": 0.96
    }
  ],
  "sentence_segments": [
    {
      "sentence_id": "sentence_1",
      "text": "Welcome to today's episode.",
      "begin_time_ms": 500,
      "end_time_ms": 2600,
      "detected_person_id": "person_1",
      "speaker_confidence": 0.88
    }
  ],
  "tracks": [
    {
      "person_id": "person_1",
      "sample_fps": 3,
      "visible_ranges": [{"begin_time_ms": 0, "end_time_ms": 17580}],
      "points": [
        {"timestamp_ms": 0, "face_bbox": [0.41, 0.18, 0.53, 0.39], "body_bbox": [0.36, 0.16, 0.59, 0.82], "is_speaking": true, "speaking_score": 0.91},
        {"timestamp_ms": 333, "face_bbox": [0.42, 0.18, 0.54, 0.39], "body_bbox": [0.37, 0.16, 0.60, 0.82], "is_speaking": true, "speaking_score": 0.89}
      ]
    }
  ]
}

Notes

detect_body=true adds person/body detection cost. Keep it false when body boxes are not needed.
Source video FPS and internal analysis FPS are resolved by the service; callers do not pass an analysis fps.
track_sample_fps controls public result size. It means "return at most this many track points per second"; internal frame-level fusion remains private.
If a person moves while speaking, read tracks[].points over time instead of only the first box.
When visual evidence is weak or multiple visible people are ambiguous, sentence_segments[].detected_person_id may be null or have low speaker_confidence.
Internal _frame_table and frame_details are never returned publicly.