AssemblyAI API : a complete guide to Speech-to-Text, Streaming, LeMUR, and production Voice AI

The AssemblyAI API is a set of endpoints and models for converting speech into text and extracting structured insights from audio. Think of it as two layers:

• Speech-to-Text (STT): turn audio into words, timestamps, and speaker segments.
• Speech Understanding: take the transcript and generate “meaningful outputs” like summaries, chapters, entities, and redaction.

As a developer, you typically use the API in one of two modes:

Base URL and API shape

The REST API uses JSON requests/responses and is hosted at a base URL of https://api.assemblyai.com.  In general, you authenticate by sending an Authorization header containing your API key.

Before writing code, it helps to align on what the API actually returns and how the pieces fit together. This prevents common mistakes like “I uploaded audio but nothing happened” or “My transcript doesn’t include speakers.”

Transcript job vs transcript result

When you call /v2/transcript, you are creating a transcription job. The API returns an id and an initial status. You then fetch the transcript later using the id until the status becomes completed (or error).

Speech models and options

The submit endpoint supports specifying speech model options and many feature flags (for example: diarization, chapters, summarization, content safety, custom spelling, etc.). The key idea is: if you want extra outputs, enable them at submission time so the job produces them.

Feature compatibility matters

Some features are mutually exclusive. For example, AssemblyAI’s docs note you can only enable one of Summarization and Auto Chapters on the same transcription.  That’s not a “bug”—it’s a product constraint you should design around (e.g., run two jobs, or choose the one you need).

The simplest way to start is to provide an audio file hosted at a public, direct URL (not a webpage). You submit that URL to the transcript endpoint and then poll until it completes. This avoids the upload step, which is useful while you’re experimenting.

Step A: Create a transcript job

The request below is representative of the official submit endpoint: POST https://api.assemblyai.com/v2/transcript.  Replace <YOUR_API_KEY> with your real key.

curl -X POST https://api.assemblyai.com/v2/transcript \
  -H "Authorization: <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "audio_url": "https://assembly.ai/wildfires.mp3"
  }'

Step B: Check status until completed

The API returns an id. Use it to fetch status:

curl -X GET https://api.assemblyai.com/v2/transcript/<TRANSCRIPT_ID> \
  -H "Authorization: <YOUR_API_KEY>"

Keep checking until the status becomes completed. AssemblyAI documents typical statuses as: queued, processing, completed, error. 

AssemblyAI uses API key authentication for REST and WebSocket streaming. The typical pattern is to send an Authorization header with your key. For REST calls, the upload endpoint explicitly shows this header as required. 

REST auth (most endpoints)

• Header: Authorization: <apiKey>
• Content-Type: JSON for most endpoints; application/octet-stream for upload.

Streaming auth (WebSocket)

Streaming uses a WebSocket URL and can authenticate either via an Authorization header (API key) or via a temporary token query parameter, per the Streaming API reference. 

You have two ways to provide audio for async transcription:

How file upload works

The upload endpoint is: POST https://api.assemblyai.com/v2/upload and expects Content-Type: application/octet-stream.  The response returns an upload_url. 

curl -X POST https://api.assemblyai.com/v2/upload \
  -H "Authorization: <YOUR_API_KEY>" \
  -H "Content-Type: application/octet-stream" \
  --data-binary @path/to/file

Then submit a transcript using the returned URL:

curl -X POST https://api.assemblyai.com/v2/transcript \
  -H "Authorization: <YOUR_API_KEY>" \
  -H "Content-Type: application/json" \
  -d '{
    "audio_url": "https://cdn.assemblyai.com/upload/<YOUR_UPLOAD_ID>"
  }'

Async STT is the most common way teams start using AssemblyAI, because most businesses already have recordings: customer calls, meetings, podcasts, interviews, training videos, lectures, and webinars. The API is job-based: create a transcript job, then fetch results.

The canonical async pipeline

1. Ingest audio: public URL or upload a local file to get an upload_url.
2. Create transcript job: POST /v2/transcript with options.
3. Wait: poll status or use webhook_url to get notified.
4. Consume results: store transcript text + structured outputs in your DB for downstream use.
5. Post-process: optionally run LeMUR or your own LLM on the transcript.

Enabling features at submit time

The submit endpoint supports many optional fields. The docs show a request containing fields like auto_chapters, auto_highlights, content_safety, custom_spelling, and more.  The core idea: “turn on what you want to receive.”

Handling partial audio (start/end offsets)

The submit endpoint also supports fields like audio_start_from and audio_end_at in examples.  This enables use cases like “transcribe only the first 10 minutes” or “skip the intro.” In production, this is a powerful cost-saving lever when your audio contains irrelevant sections.

Transcript status is the simplest (and most important) state machine in the platform. After submitting a job, it transitions through statuses such as: queued, processing, completed, and error. 

Why jobs fail

The docs mention common causes such as corrupted or unsupported audio, providing a webpage URL instead of a file, or a URL not accessible from AssemblyAI’s servers.  In production, validate audio early (format, duration, accessibility) to reduce error rates.

Polling is easy: keep requesting transcript status until it completes. But polling becomes expensive and noisy at scale. Webhooks let AssemblyAI call your server when the transcript is ready. The official docs describe webhooks as custom HTTP callbacks you define to get notified when transcripts are ready. 

When polling is okay

• Prototyping or a hackathon.
• Low volume transcription (a few jobs/day) with no strict efficiency requirements.
• Local scripts and personal automation.

When to use webhooks

• Anything production that might scale beyond a small trickle.
• Background processing pipelines (queues, workers, job systems).
• When you want reliable completion events without tight polling loops.

Webhook design best practices

1. Idempotency: treat webhook deliveries as “may repeat.” If you receive the same completion event twice, do not double-bill or double-store.
2. Verification: verify requests (shared secret / signature pattern if provided, or allowlist IPs if your infra supports it).
3. Async handling: accept quickly (200 OK), then process in background (queue) so webhook delivery doesn’t time out.
4. Safe retries: if your handler fails, you should be able to re-fetch transcript results later using the transcript ID.

Streaming transcription is for live audio. Instead of uploading a file, you open a persistent WebSocket connection and send audio frames as they are recorded. AssemblyAI returns partial and final transcripts over the same connection.

WebSocket handshake and endpoint

The Streaming API reference lists the handshake URL as: wss://streaming.assemblyai.com/v3/ws.  It also notes: to use the EU server, replace the host with streaming.eu.assemblyai.com. 

Required parameters

The handshake requires at least sample_rate. There are options like encoding (e.g., PCM 16-bit little-endian or mu-law), and settings that influence turn detection and formatting. 

Authentication choices

You can authenticate with an API key in the Authorization header, or generate a temporary token and pass it via a query parameter, per the docs. 

In real-time speech recognition, users judge “speed” by how quickly they see partial text and how quickly the system correctly decides that someone finished speaking (end-of-turn detection / endpointing). AssemblyAI’s streaming API includes parameters that help tune this behavior, such as an end-of-turn confidence threshold. 

Partial vs final transcripts

• Partial: fast, may change, used for live captions or immediate feedback.
• Final: stable, used for saving, compliance, downstream NLP, and analytics.

When to use formatted final transcripts

The streaming API includes a formatting option for returning formatted final transcripts.  Use this when the final output is user-facing (subtitles, notes, transcripts shown in UI). If you’re using the transcript as raw NLP input, you may prefer minimal formatting and do your own post-processing.

Speech-to-text gives you words. Speech understanding gives you structure and meaning. AssemblyAI provides several understanding models you can enable at transcription time, returning extra fields in the transcript response.

Common “Speech Understanding” features

The important architectural point: most of these are computed as part of the transcription job (or are tightly coupled to it). So you should decide and enable what you need when you create the transcript job, then store the resulting structured fields for fast UI rendering and analytics.

Speaker diarization answers: “who spoke when?” It detects multiple speakers and returns a list of utterances (chunks of uninterrupted speech) per speaker.  In practice, this powers meeting transcripts, call center analytics, interviews, and panel discussions.

How to enable diarization

AssemblyAI’s diarization docs show enabling it by setting speaker_labels to true in the transcription config.  The transcript result then includes “utterances” with speaker labels.

Speaker count controls

You can set speakers_expected when you’re confident about the exact number of speakers.  The docs warn to use it only when you are certain; forcing the wrong number can produce bad splits.  A more flexible alternative is to set a range of possible speakers (when supported by options).

Generic labels vs real names

Diarization typically labels speakers as “Speaker A/B/…”. AssemblyAI also describes a separate concept of Speaker Identification to name speakers (roles/names) beyond generic labels.  In product terms:

• Diarization: segmenting + labeling distinct voices.
• Identification: mapping voice turns to human-meaningful names/roles.

AssemblyAI’s Summarization model generates a summary of the resulting transcript.  The docs highlight that you control the style and format using a summary model and summary type.  They also note a constraint: you can only enable one of Summarization and Auto Chapters in the same transcription. 

Why summaries matter

Summaries convert “pages of transcript” into something users can act on: meeting highlights, lecture overviews, daily call digests, and quick review cards. They also reduce cost if you later run a separate LLM, because you can feed the summary instead of the full transcript.

When summaries fail

Summaries struggle when audio quality is poor, speakers overlap, or the content is extremely technical. If you are summarizing high-stakes content (medical, legal, financial), treat summaries as “assistive,” not authoritative.

Auto Chapters is a model that summarizes audio data over time into chapters, making it easier to navigate and find specific information.  Each chapter contains items like summary, gist, headline, and start/end timestamps. 

Why chapters are better than “one big summary”

A single summary is great for speed, but long recordings (podcasts, lectures, meetings) contain multiple topics. Chapters provide a table-of-contents style navigation: users can jump directly to the section they care about without searching raw text.

Chapters vs summarization (choose one per job)

Like summarization, chapters have a compatibility constraint: the docs state you can only enable one of Auto Chapters and Summarization.  If your product needs both, design for a second pass job or a LeMUR step (see LeMUR section).

PII redaction is about minimizing sensitive information by identifying and removing it from transcripts.  PII includes data like names, email addresses, phone numbers, and other identifiers. This feature is especially important if you store transcripts for long periods, share them externally, or process regulated content.

When PII redaction is a must

• Customer support calls that include addresses, card details, or account numbers.
• Healthcare-related audio that includes personal patient details.
• HR interviews and recruiting calls.
• Any transcript you export or share with third parties.

Redaction as a workflow, not a checkbox

In production, redaction should be treated as part of a data governance pipeline: access controls, retention policy, audit logging, and careful handling of raw audio. The API provides redaction outputs, but your system still needs policy and enforcement.

Entity detection automatically identifies and categorizes key information in audio, including names of people, organizations, addresses, phone numbers, dates, and more.  This transforms a transcript into searchable structured data.

Real-world use cases

• Sales calls: extract competitors, product names, contract dates, and pricing mentions.
• Support calls: extract ticket numbers, error codes, device models.
• Podcasts: extract guest names, brands, locations, and events.
• Compliance: detect mentions of sensitive identifiers and trigger review flows.

LeMUR is AssemblyAI’s framework for applying large language model capabilities to recognized speech. Conceptually, it turns “a transcript” into “an interactive document”: you can ask questions, generate action items, summarize with custom prompts, and run other generative tasks on top of the transcript.

How LeMUR fits into a product

Many apps need two layers: (1) stable transcript + structured fields (speakers, timestamps, chapters), and (2) generative insights (action items, Q&A, executive summaries). LeMUR is commonly used as the second layer once transcription is completed.

Using existing transcripts

LeMUR workflows typically reference an existing transcript ID. The official LeMUR docs show fetching an existing transcript via GET https://api.assemblyai.com/v2/transcript/YOUR_TRANSCRIPT_ID. 

When to prefer LeMUR vs built-in summarization/chapters

• Prefer built-in summarization/chapters when you want predictable formats and simple configuration at transcription time.
• Prefer LeMUR when you want custom prompts, multi-step insights, action items, question answering, and flexible responses.

Limits matter because they shape your product boundaries and your “edge cases.” AssemblyAI’s FAQ documents that the maximum file size for submission to /v2/transcript is 5GB and maximum duration is 10 hours.  It also notes an upload size limit for /v2/upload (local upload) of 2.2GB. 

What to do with files over the limit

• Split audio: chunk the file into segments and transcribe separately.
• Use partial offsets: if only certain sections matter (intro, Q&A segment), transcribe those sections only.
• Downsample / re-encode: reduce size by using efficient encoding (without destroying intelligibility).

Scaling without pain

Scaling isn’t only “more CPUs.” It’s also fewer failures: validate audio, use webhooks, implement retries, and make your pipeline idempotent. If you want increased rate limits, AssemblyAI’s docs suggest contacting them. 

AssemblyAI uses usage-based pricing. The official pricing page lists base rates and add-on costs. As of the pricing page details shown, streaming speech-to-text is listed at $0.15/hr, and several add-ons have their own hourly prices (for example, Speaker Diarization at $0.02/hr, Entity Detection at $0.08/hr, Keyterms Prompting at $0.04/hr, and more). 

Cost planning: think in “workload recipes”

A helpful way to estimate costs is to define a few recipe profiles your app supports, like:

Where costs get out of control

• Turning on everything by default for every transcription, even when users don’t need it.
• Re-processing the same audio multiple times instead of caching results.
• Polling at high frequency across large numbers of jobs.
• Not trimming audio (transcribing silence, intros, hold music, or irrelevant sections).

The difference between a demo and a product is reliability. Here’s a proven architecture for building on AssemblyAI.

Reference architecture (async transcription)

1. Frontend: user uploads audio/video, or connects a recording source.
2. Backend API: receives upload, stores file in your object storage, generates a job record.
3. Worker: uploads to AssemblyAI (or supplies a signed URL) and creates transcript job.
4. Webhook endpoint: receives completion event and enqueues a “fetch results” task.
5. Processor: fetches transcript JSON, stores in DB, indexes text for search, stores structured fields (speakers/entities/chapters).
6. Optional LLM layer: run LeMUR or your own LLM for action items, summaries, Q&A, or extraction.

Idempotency strategy

Use stable IDs and deduplication keys:

• Compute a file hash (or storage object key) and store it so the same file isn’t processed twice.
• Store transcript job ID and treat “fetch transcript” as safe to retry.
• On webhook receive, enqueue a job keyed by transcript ID; drop duplicates.

Retries and backoff

Not every failure is permanent. Network errors, timeouts, and occasional server errors should be retried with exponential backoff and jitter. However, “bad input” errors (corrupt audio, inaccessible URL) should not be retried until fixed. The docs mention that if failures persist after resubmitting, you should contact support. 

Storage & indexing

Store:

• Raw transcript text
• Word-level or sentence-level timestamps (for subtitles and jump links)
• Utterances and speaker labels (for meeting UX)
• Entities and redaction outputs (for search and compliance)
• Chapters/summaries (for navigation and “TL;DR” UX)

Voice data is sensitive. Your security posture should assume audio can contain PII, confidential business info, and regulated content. Good security is a combination of key handling, network controls, and data governance.

API key handling

• Keep your key in server environment variables (not in client apps).
• Rotate keys if accidentally exposed.
• Log carefully: never log full headers containing Authorization keys.

Public URLs and access control

If you pass audio_url as a public URL, ensure it is a direct file URL and accessible from AssemblyAI’s servers. Avoid exposing private recordings with globally public URLs; prefer short-lived signed URLs when possible.

Webhook security

• Use HTTPS only.
• Validate incoming webhook payloads (signature/secret if supported in your configuration).
• Make handlers idempotent and quick; enqueue heavy work.
• Rate limit webhook endpoints to defend against spam.

“My transcript is stuck”

If status is queued or processing, it may simply be waiting for capacity. Avoid polling every second. If the job stays in that state for an unusually long time, try:

• Confirm the audio URL is a direct file and reachable.
• Confirm the audio isn’t corrupted or unsupported.
• Retry submission once (don’t spam).

401 Unauthorized

Usually means missing or invalid API key. Confirm you are sending Authorization: <YOUR_API_KEY> and you did not include extra spaces or quotes.

429 Too Many Requests

Rate limiting can occur on some endpoints/services. The docs describe rate limits with X-RateLimit-* headers and 429 responses for exceeded limits.  Implement exponential backoff and reduce polling frequency. Use webhooks instead of polling at scale.

422 Unprocessable Entity / Upload failures

Upload failures are usually about request formatting or payload issues: wrong content type, sending JSON instead of binary, or a file that exceeds the upload size constraints. Ensure you use Content-Type: application/octet-stream and --data-binary for curl uploads. 

“Audio URL is a webpage”

The docs explicitly mention failures when the audio URL is a webpage rather than a file.  Fix by using a direct file URL or upload the media via /v2/upload.

Use official docs for exact request/response fields and the latest feature set. The links below are the most important pages to bookmark.

Core REST endpoints

• API Overview / Base URL: REST base URL is https://api.assemblyai.com. 
• Upload: POST /v2/upload with binary upload. 
• Submit transcript: POST /v2/transcript with audio_url. 
• Status lifecycle: queued/processing/completed/error. 

Streaming (real-time)

• WebSocket handshake: wss://streaming.assemblyai.com/v3/ws. 
• EU streaming host option: streaming.eu.assemblyai.com. 

Models & features

• Summarization docs: summary model/type + mutual exclusion with chapters. 
• Auto Chapters docs: chapter fields and navigation structure. 
• Speaker diarization docs: utterances + speaker_labels + best practices. 
• Entity detection docs: supported entity types. 
• PII redaction docs: redaction behavior and purpose. 
• Pricing page: base rates and add-ons. 
• Limits FAQ: file size/duration constraints. 