# Sonar: plug real-time answering machine detection (AMD) into your calls This document is written to be handed as-is to a coding agent (Claude Code) or a developer. It contains everything needed to plug Sonar into an outbound call, from **Twilio Media Streams** or from **FreeSWITCH**. ## In short - During the call, you send Sonar the **callee's audio only** (not your AI's voice), starting at answer. - Sonar sends verdicts back as the call unfolds: human, voicemail, carrier message, switchboard or IVR. - **You** hang up, by applying the rule in "When to hang up". - Analysis runs in parallel: it never delays your AI, which talks from the moment the call is answered. - No audio is stored. Sonar keeps one line per call: ID, duration, verdict. ## Access (development environment) | | | |---|---| | Real-time WebSocket | `wss://167-233-100-203.sslip.io/v1/stream` | | REST API | `https://167-233-100-203.sslip.io` | | Key | sent separately. Put it in `SONAR_API_KEY`, never in the code | | Quota | 20,000 analysed calls per month | | Concurrent streams | 10 at most on this environment | | Analysed duration | 120 s of audio at most per call | A call counts as soon as it has sent at least 1 s of audio. This is a development endpoint; the address will change for production. ## WebSocket protocol `/v1/stream` One WebSocket connection per call. **1. First message, as text: `start`** ```json {"type": "start", "api_key": "", "call_id": "CA1234...", "sample_rate": 8000, "format": "mulaw"} ``` | Field | Values | |---|---| | `call_id` | your call ID (Twilio CallSid, FreeSWITCH UUID), 128 characters max | | `sample_rate` | 8000 recommended; other rates are resampled | | `format` | `mulaw` (Twilio), `alaw`, `s16le` (16-bit little-endian PCM), `f32le` | | `wrap` | optional. `"transcription"` wraps every reply as `{"type":"transcription","data":{…}}`, the format FreeSWITCH's `mod_audio_fork` expects | **2. Then, audio as binary messages** Mono audio of the callee, in the announced format, in 20 to 200 ms chunks, as the call goes. Start at answer (`answer`), not while ringing nor during early media. **3. End (optional)**: `{"type": "end"}`, or simply close the connection. ### Messages you receive ```json {"type": "verdict", "call_id": "CA1234...", "verdict": "machine", "label": "operator_msg", "event": "confirmed", "action": "hangup", "t_decision": 8.4} {"type": "beep", "call_id": "CA1234...", "t": 15.04} {"type": "final", "call_id": "CA1234...", "audio_seconds": 20.4, "verdict": "machine", "label": "operator_msg", "action": "hangup", "beep": true} {"type": "error", "code": 401, "message": "clé API invalide ou révoquée"} ``` | Field | Values | |---|---| | `verdict` | `machine` or `human` (`no_decision` possible in `final`) | | `label` | `human`, `voicemail` (personal greeting), `operator_msg` (voicemail or carrier message), `robot_ivr` (switchboard, IVR, waiting queue) | | `event` | `provisional`: first suspicion, may be withdrawn. `confirmed`: machine confirmed. `final`: definitive decision. `update`: late decision after monitoring. `revoked`: suspicion withdrawn | | `action` | `hangup`: hang up. `wait`: switchboard or IVR, a human may still pick up. `continue`: human, let your AI talk | | `t_decision` | seconds of audio elapsed since answer when the verdict was issued | `beep` signals a voicemail beep. `final` always arrives last, when the stream closes. Error `message` texts are in French: rely on `code`. ### When to hang up (apply exactly as written) ``` hang up if verdict.action == "hangup" AND verdict.event in {"confirmed", "final", "update"} hang up if "beep" received AFTER a verdict with action == "wait" or event == "revoked" NEVER hang up on a "provisional" verdict alone ``` Never block the call on Sonar. If the connection fails, closes or gives no verdict, the call goes on normally: this is the "fail-open" principle. ### Errors and close codes | Code | Cause | |---|---| | 4001 | missing or invalid key (`error.code` 401), account disabled (403), quota reached (429) | | 4002 | unknown `format` | | 1013 | maximum capacity reached. The call goes on without AMD; retry on the next call | ### Latency observed in production (~400,000 calls, October 2026) - First machine suspicion: 5.5 s median after answer. - Hang-up: 9 s median, 90% under 14 s. Sonar deliberately waits for confirmation so it never cuts a human. - Definitive human verdict: around 11 s. Your AI talks from the moment the call is answered, without waiting for this verdict. ## Recipe 1: Twilio Media Streams Twilio sends 8 kHz μ-law audio, base64-encoded, in 20 ms frames. Just decode the base64 and forward the bytes as they are (`format: "mulaw"`). - If you already run a WebSocket server for `` (bidirectional): add one Sonar session per call, as below. This stream only carries the `inbound` track, which is the callee: exactly what you need. - To plug Sonar next to your current pipeline without touching it: add `` to the TwiML. Twilio then sends a copy of the callee's audio to a second WebSocket, where you run the same code. ```python """One Sonar session per Twilio call. Fail-open: any error lets the call go on.""" import asyncio, base64, json, os import websockets from twilio.rest import Client SONAR_URL = "wss://167-233-100-203.sslip.io/v1/stream" SONAR_KEY = os.environ["SONAR_API_KEY"] SHADOW = os.environ.get("SONAR_SHADOW", "1") == "1" # 1 = log only, never hang up twilio = Client(os.environ["TWILIO_ACCOUNT_SID"], os.environ["TWILIO_AUTH_TOKEN"]) class SonarSession: def __init__(self, call_sid: str): self.call_sid = call_sid self.ws = None self.waiting_beep = False self.done = False async def start(self): try: self.ws = await websockets.connect(SONAR_URL, open_timeout=3) await self.ws.send(json.dumps({ "type": "start", "api_key": SONAR_KEY, "call_id": self.call_sid, "sample_rate": 8000, "format": "mulaw"})) asyncio.create_task(self._read()) except Exception: self.ws = None async def feed(self, payload_b64: str): if self.ws and not self.done: try: await self.ws.send(base64.b64decode(payload_b64)) except Exception: self.ws = None async def stop(self): if self.ws: try: await self.ws.send(json.dumps({"type": "end"})) await self.ws.close() except Exception: pass async def _read(self): try: async for raw in self.ws: m = json.loads(raw) if m["type"] == "verdict": if m["action"] == "hangup" and m["event"] in ("confirmed", "final", "update"): await self._hangup(f'{m["event"]}:{m["label"]}') elif m["action"] == "wait" or m["event"] == "revoked": self.waiting_beep = True elif m["type"] == "beep" and self.waiting_beep: await self._hangup("beep") elif m["type"] == "final": print("sonar final", self.call_sid, m) # log it on your side except Exception: pass async def _hangup(self, reason: str): if self.done: return self.done = True print("sonar hangup", self.call_sid, reason, "(shadow)" if SHADOW else "") if not SHADOW: await asyncio.to_thread(twilio.calls(self.call_sid).update, status="completed") # In your Twilio WebSocket handler: # event "start" → sonar = SonarSession(msg["start"]["callSid"]); await sonar.start() # event "media" → if msg["media"].get("track", "inbound") == "inbound": # await sonar.feed(msg["media"]["payload"]) # event "stop" → await sonar.stop() ``` Dependencies: `pip install websockets twilio`. **The same session in TypeScript** (Node 18+, `npm i ws twilio` and `npm i -D @types/ws`). Audio received while connecting to Sonar is buffered, then sent. ```typescript /** One Sonar session per Twilio call. Fail-open: any error lets the call go on. */ import WebSocket from "ws"; import twilio from "twilio"; const SONAR_URL = "wss://167-233-100-203.sslip.io/v1/stream"; const SONAR_KEY = process.env.SONAR_API_KEY ?? ""; const SHADOW = (process.env.SONAR_SHADOW ?? "1") === "1"; // 1 = log only, never hang up const client = twilio(process.env.TWILIO_ACCOUNT_SID, process.env.TWILIO_AUTH_TOKEN); type SonarMessage = | { type: "verdict"; call_id: string; verdict: "machine" | "human"; label: string; event: "provisional" | "confirmed" | "final" | "update" | "revoked"; action: "hangup" | "wait" | "continue"; t_decision: number } | { type: "beep"; call_id: string; t: number } | { type: "final"; call_id: string; audio_seconds: number; verdict: "machine" | "human" | "no_decision"; label: string | null; action: string | null; beep: boolean } | { type: "error"; code: number; message: string }; export class SonarSession { private ws: WebSocket | null = null; private pending: Buffer[] = []; // audio received while connecting to Sonar private waitingBeep = false; private done = false; constructor(private readonly callSid: string) {} start(): void { try { const ws = new WebSocket(SONAR_URL, { handshakeTimeout: 3000 }); ws.on("open", () => { ws.send(JSON.stringify({ type: "start", api_key: SONAR_KEY, call_id: this.callSid, sample_rate: 8000, format: "mulaw" })); for (const chunk of this.pending) ws.send(chunk); this.pending = []; }); ws.on("message", (raw) => { try { void this.onMessage(JSON.parse(raw.toString()) as SonarMessage); } catch { // unreadable message: ignored } }); ws.on("error", () => { this.ws = null; }); ws.on("close", () => { this.ws = null; }); this.ws = ws; } catch { this.ws = null; } } feed(payloadB64: string): void { const ws = this.ws; if (!ws || this.done) return; const chunk = Buffer.from(payloadB64, "base64"); if (ws.readyState === WebSocket.OPEN) ws.send(chunk); else if (ws.readyState === WebSocket.CONNECTING && this.pending.length < 250) this.pending.push(chunk); } stop(): void { const ws = this.ws; if (!ws) return; try { // Sonar answers with "final", then closes the connection itself if (ws.readyState === WebSocket.OPEN) ws.send(JSON.stringify({ type: "end" })); setTimeout(() => ws.close(), 5000).unref(); } catch { // fail-open } } private async onMessage(m: SonarMessage): Promise { if (m.type === "verdict") { if (m.action === "hangup" && (m.event === "confirmed" || m.event === "final" || m.event === "update")) { await this.hangup(`${m.event}:${m.label}`); } else if (m.action === "wait" || m.event === "revoked") { this.waitingBeep = true; } } else if (m.type === "beep" && this.waitingBeep) { await this.hangup("beep"); } else if (m.type === "final") { console.log("sonar final", this.callSid, m); // log it on your side } } private async hangup(reason: string): Promise { if (this.done) return; this.done = true; console.log("sonar hangup", this.callSid, reason, SHADOW ? "(shadow)" : ""); if (!SHADOW) { try { await client.calls(this.callSid).update({ status: "completed" }); } catch { // the call may already be over } } } } // In your Twilio WebSocket handler: // event "start" → const sonar = new SonarSession(msg.start.callSid); sonar.start(); // event "media" → if ((msg.media.track ?? "inbound") === "inbound") sonar.feed(msg.media.payload); // event "stop" → sonar.stop(); ``` - The Twilio stream starts at answer, when the TwiML runs: nothing to filter. - If Twilio's own answering machine detection (`machineDetection`) is on, do not let it hang up at the same time as Sonar. Keeping it in shadow mode, however, is the best way to compare both: log `AnsweredBy` next to Sonar's verdict. - To hang up, you can also stop your AI agent instead of ending the Twilio call, depending on your architecture. ## Recipe 2: FreeSWITCH (`mod_audio_fork`) This is the integration running in production since July 2026, more than 1.5 million calls analysed. **When the callee's leg is answered** (`CHANNEL_ANSWER` event, never during early media), start the fork over ESL: ``` bgapi uuid_audio_fork start wss://167-233-100-203.sslip.io/v1/stream mono 8000 {"type":"start","api_key":"","call_id":"","sample_rate":8000,"format":"s16le","wrap":"transcription"} ``` - `mono`: only the audio received on this channel is sent, which is the callee. Your AI's voice is never sent. - The last argument is the `start` message. It must be compact JSON with no spaces, because it is a single ESL argument. - Use `bgapi`, not `api`. Otherwise the WebSocket handshake blocks the ESL connection; under load, events pile up and FreeSWITCH ends up closing the ESL. - `"wrap":"transcription"` is required: `mod_audio_fork` only relays messages of type `transcription`. **Receive the verdicts**: subscribe the ESL to `events plain CHANNEL_ANSWER CHANNEL_HANGUP CUSTOM mod_audio_fork::transcription`. - Each verdict arrives as a `CUSTOM` event, subclass `mod_audio_fork::transcription`. - The JSON is in the event body, and `Unique-ID` gives the channel UUID. - Depending on the module version, the body holds the full envelope `{"type":"transcription","data":{…}}` or directly `{…}`: unwrap `data` when present. **Hang up**: apply the "When to hang up" rule, then `bgapi uuid_kill NORMAL_CLEARING`. If your AI sits on another bridged leg, kill the leg that carries it. **With `mod_audio_stream` instead**: same principle (`uuid_audio_stream start mono 8k `). Sonar's messages arrive as `mod_audio_stream::json` events: do not use `wrap`. Check the exact syntax of your module version, and set `sample_rate` to 16000 if you stream at 16k. ## Test without a real call **Live demo in the browser**: https://sonar-docs.pages.dev/en/#demo (audio file or microphone, with your key). **Replay a recording as a live call.** You need Python 3.10+, `pip install websockets` and `ffmpeg`. ```python """python sonar_replay.py call.wav [--format mulaw] [--channel 0] [--realtime]""" import argparse, asyncio, json, os, subprocess import websockets ap = argparse.ArgumentParser() ap.add_argument("wav") ap.add_argument("--format", default="mulaw", choices=["mulaw", "alaw", "s16le"]) ap.add_argument("--channel", type=int, default=0, help="callee channel if stereo") ap.add_argument("--realtime", action="store_true") a = ap.parse_args() width = 2 if a.format == "s16le" else 1 audio = subprocess.run(["ffmpeg", "-v", "quiet", "-i", a.wav, "-af", f"pan=mono|c0=c{a.channel}", "-ar", "8000", "-f", a.format, "-"], capture_output=True).stdout async def main(): async with websockets.connect("wss://167-233-100-203.sslip.io/v1/stream") as ws: await ws.send(json.dumps({"type": "start", "api_key": os.environ["SONAR_API_KEY"], "call_id": a.wav, "sample_rate": 8000, "format": a.format})) async def reader(): async for raw in ws: m = json.loads(raw) print(m) if m["type"] in ("final", "error"): return r = asyncio.create_task(reader()) step = 1600 * width # 200 ms for i in range(0, len(audio), step): await ws.send(audio[i:i + step]) if a.realtime: await asyncio.sleep(0.2) await ws.send(json.dumps({"type": "end"})) await asyncio.wait_for(r, 15) asyncio.run(main()) ``` **Analyse a whole file at once** (WAV, MP3; `channel` = the callee's channel): ```bash curl -H "x-api-key: $SONAR_API_KEY" -F file=@call.wav \ "https://167-233-100-203.sslip.io/v1/verify?channel=0" ``` **Usage and latest calls**: ```bash curl -H "x-api-key: $SONAR_API_KEY" https://167-233-100-203.sslip.io/v1/usage curl -H "x-api-key: $SONAR_API_KEY" "https://167-233-100-203.sslip.io/v1/calls?limit=50" ``` ## Measure the gain (shadow mode) To put a number on the return on investment before letting Sonar hang up: 1. Plug Sonar in with `SONAR_SHADOW=1`: it gives its verdict but cuts nothing. 2. Log, for each call, Sonar's decision and when it would have hung up, next to what actually happened: duration, Twilio's `AnsweredBy`, conversation outcome. 3. Compare the AI and telephony seconds saved on machines, and the humans the other detection wrongly cut. 4. Switch to `SONAR_SHADOW=0` once the numbers are validated.