AI pipeline¶

Transcripts and summaries are optional. They run in the background jobs worker, using the workspace's own provider and key.

Choosing the provider¶

ai::resolve(org, purpose) decides which provider a job uses:

The workspace's setting: a row in ai_providers for that workspace and purpose (transcription or summary), with the key decrypted.
Otherwise, the server's environment, if the operator set one. See Configuration.
Otherwise, AI is off for that workspace.

The settings page writes both rows at once (one provider, one key, one model). The catalogue in ai/catalogue.rs lists only models that can both hear audio and write a summary:

Vendor id	Label	Wire protocol	Base URL	Model
`gemini`	Google Gemini	Gemini REST	`https://generativelanguage.googleapis.com`	`gemini-3.8-flash`
`openai`	OpenAI	OpenAI Chat Completions	`https://api.openai.com/v1`	`gpt-audio-1.5`
`qwen`	Alibaba Qwen	OpenAI-compatible (DashScope)	`https://dashscope-intl.aliyuncs.com/compatible-mode/v1`	`qwen3.8-omni-flash`

There is also a hidden custom entry (any OpenAI-compatible server, for example a self-hosted one). It is only shown to a workspace that already uses it.

Wire protocol vs vendor

kind is how Vistilla talks to a provider (openai, gemini or anthropic), not who the provider is. Qwen uses the openai kind because its API copies OpenAI's.

Transcription¶

video object --signed link (via /_b2/)--> ffmpeg keeps only the sound track
                                                 |  (sound.mka, a few MB an hour)
                                                 v
                 measure the length if the video has none
                                                 v
                                silence check --silent--> "no speech" (no AI call)
                                | not silent
                                v
                 longer than 30 s and the model guesses timings?
                   | yes                                    | no
                   v                                        v
       split into 30 s Opus windows             extract one Opus track
       (skip silent windows)                    (mono, 16 kHz, 24 kbit/s)
                   |                                        |
                   +--------------+-------------------------+
                                  v
                    send to the provider (see below)
                                  v
          clean segments -> fit timings to the real length -> store
                                  v
                           queue the summary

Step by step (`jobs.rs`)¶

Take the sound (sound_track). With S3 storage, ffmpeg reads the video straight from the bucket over a signed link (through the Worker's /_b2/ when VISTILLA_S3_DOWNLOAD_PROXY is set), and copies only its first audio track, untouched, into sound.mka: a few megabytes an hour, where the picture would be a gigabyte or two on a temporary disk the host caps at 2 GB. Every step below reads that file. A video with no audio track ends here as "no speech was found in this recording". With folder storage, or without ffmpeg, the video is downloaded instead, streamed to disk. Videos over the recording limit are refused.
Length (seekable::measure_length_ms). If the video has no duration_ms (a recording whose page lost track of it), it is measured from the sound: ffprobe reads the header, and for a file straight from MediaRecorder, which has none, ffmpeg reads it through (-c copy -f null). The length is saved on the video, because the window size below depends on it.
Silence check (audio::is_silent). ffmpeg volumedetect measures the loudest moment:
- below −50 dB, or no audio stream at all, means silent;
- the job then ends as "no speech was found in this recording", without calling the AI.
Models shown silence tend to invent words, so this check matters. 4. Windows. Models that are shown audio (Gemini, and chat models that hear audio) estimate timings, and the estimate drifts over a long recording. So recordings longer than 30 seconds are cut by ffmpeg into 30-second windows (60-second ones for recordings over 30 minutes, to halve the number of requests), with exact offsets: - each window is transcribed separately, and silent windows are skipped; - the lines are placed on the full timeline by their window's offset; - there is a 600 ms pause between windows, to stay inside free-tier rate limits.

Whisper-style engines measure timings themselves and get the whole file. 5. Audio extraction. Only the sound is sent: mono Opus at 24 kbit/s, about 11 MB per hour. 6. Send it to the provider: - Gemini: - POST {base}/v1beta/models/{model}:generateContent, with the audio inline (up to 14 MiB); - a JSON response_schema asks for {language, segments: [{start_seconds, end_seconds, text}]}; - on 429 or 5xx it retries after 4, 12, 30, 60 and 60 seconds, so a free key's per-minute quota is waited out during a long meeting. - Chat models that hear audio (OpenAI gpt-audio-1.5, Qwen Omni): - the audio is re-encoded to MP3 (32 kbit/s, up to 7 MiB); - it is sent as an input_audio part to {base}/chat/completions, with the same instructions plus "reply with JSON only"; - for Qwen (DashScope), the request is streamed (stream: true), as its Omni models require, and the streamed pieces are joined. - Whisper-style (whisper-*, *transcribe* models, set by the operator): - multipart POST {base}/audio/transcriptions with response_format=verbose_json, up to 25 MiB. 7. Clean up: blank lines are dropped. For Gemini and the chat models, start times are also made non-decreasing, and an end time can never come before its start. Whisper-style engines measure their timings, so theirs are kept as they come. 8. Fit to the real length (anchor_to_duration). If the last line ends after the video does, every timing is scaled back proportionally, then clamped to the video's length. A transcript that ends early is left alone, since that is just silence at the end. 9. Keep only what was spoken (audio::spoken). The sound's loudness is measured a tenth of a second at a time (audio::levels). A line is kept when the louder quarter of its sound is at least 10 dB above the recording's noise floor (its quietest fifth), or louder than −35 dBFS on its own; a line the model made up from room noise between sentences, often in another language, sits at the noise floor and is dropped. If nothing is left, the recording has no speech. 10. Onto the recording's timeline (audio::sound_delay_ms). A browser can start the microphone seconds after the picture; a model counts from the first sound, so every line is moved later by that delay (and clipped to the video's length), keeping captions in step with the words. 11. Store the lines in transcript_segments and mark the transcript ready, in one transaction. Then queue the summary.

The prompt asks for a verbatim transcript in short segments, with timestamps read from the audio and inside the stated length. Only words clearly heard are written: noise, music, typing and breathing are left out and nothing is guessed. It forbids summarising, translating or adding anything, and asks for an empty list when nobody speaks.

Summary¶

summarize::summarize builds one prompt from:

the title;
the length;
the transcript, one line per segment with its [M:SS] time ([H:MM:SS] past an hour). A transcript over 120,000 characters keeps its start and end, with the middle cut.

It asks for JSON only:

Field	What
`headline`	One sentence, under 90 characters.
`summary`	Two or three short paragraphs.
`highlights`	Up to 6 key moments `{text, timecode_ms}`.
`actions`	Up to 6 action items; empty if none ("do not invent actions").
`chapters`	2–8 topic chapters `{title, timecode_ms, about}`, the first at 0, `about` being two or three sentences on what was said. An empty list for a single topic.

It is written in the language of the transcript.

The answer is cleaned up:

a code fence or a preamble around the JSON is tolerated;
points are limited to 6 and their timecodes kept inside the video;
chapters are sorted, start at 0, must be at least 3 s apart, and are capped at 10 (fewer than 2 means none).

Points go to summary_points. AI chapters replace earlier AI chapters, but only if the video has no manual chapters.

Captions¶

GET /media/videos/{id}/captions.vtt and GET /media/share/{token}/captions.vtt build a WebVTT file from the segments on each request. Each cue lasts at least 200 ms.
The transcript's captions_url carries ?v={updated_at}, so a redone transcript is never served from cache.
The player attaches the file as a hidden <track> and draws the captions itself, above its own control bar.

Translation¶

The website and the app read the transcript and summary with ?lang=en or ?lang=ar, the language they are in.

The original's language comes from its letters (mostly Arabic script is ar), else from what transcription reported (ai/translate.rs, source_lang). Every answer carries source_lang and lang, the language of what it holds.
Same language: the original, as it is.
Other language: a current row of translations (same source_at as the original's updated_at) is swapped in, string for string. With none, one is queued and the original is sent with translation: "pending"; the page asks again every 4 s. A failed one says translation: "failed" and is tried again when read after 30 minutes.
The job (run_next_translation, after summaries) sends the strings (the segments' texts; or the headline, body, highlights, actions and each topic's title and about) to the workspace's summary provider, in chunks of at most 80 strings or 5,000 characters, asking for exactly as many back. A chunk that comes back miscounted is split in two and tried again, so timings and structure never depend on the model.
Captions take &lang= too; the transcript's captions_url carries it when the transcript is translated.
A redone transcript or summary has a new updated_at, so its translation is made again on the next read.

Job queue and retries¶

The transcripts and summaries tables are the queue: pending → running → ready / failed.
A failed job stores its error (up to 500 characters) and is not retried automatically. Try again on the video page queues it again.
Jobs left running by a crash go back to pending when the server restarts.
Backfill queues work for videos published before AI was turned on, one at a time, newest first:
- a missing transcript;
- then a missing summary;
- then a summary that has no chapters yet.

The settings API¶

GET /api/orgs/{org}/ai returns the vendor list and the current settings. The key is never returned, only a hint (…abcd).
PUT /api/orgs/{org}/ai/{purpose} saves. A new key is sealed with XChaCha20-Poly1305; a blank key keeps the stored one.
POST /api/orgs/{org}/ai/{purpose}/test asks the model to reply with a tiny JSON object, and records last_ok_at or last_error.
Only workspace owners and admins can use these, "because a key is spend".