You’ve built a voice agent on Twilio ConversationRelay. It greets callers, understands what they say, and answers in a natural voice. Then a caller asks where their order is, and the agent has nothing to give them. It can explain how shipping generally works. It can apologize. What it can’t do is open your database and read back a tracking number, because nothing connects what the model says to your actual systems.
The model isn’t the problem here. It’s perfectly capable of working out that it needs to look something up. The gap is architectural: in a plain ConversationRelay loop, the model can say it wants to call get_order_status, but nothing runs the function and feeds the result back. Build that missing piece and your talker becomes a doer.
This post keeps the scope tight. We build just the tool-calling loop in Python, start to finish, small enough to read in one sitting and repoint at your own backend. By the end you’ll have a runnable FastAPI agent that looks up an order and books an appointment over the phone, and you’ll have written every line of it yourself.
The complete project is on GitHub. Clone it to follow along, or just run it straight away:
https://github.com/mostafaibrahim17/twilio-conversationrelay-tools-python
Prerequisites
To build this and place a real test call, you’ll need:
A Twilio account. Sign up for a free trial if you don’t have one. The trial comes with a small balance and one phone number.
A voice-capable Twilio phone number (your trial number works fine).
An OpenAI API key with a little credit on it. The model runs on OpenAI’s side, and your server is what calls it.
Python 3.11+ installed locally.
ngrok (or any tunnel) so Twilio can reach your machine while you develop.
Most of the work is the code itself. The rest is Twilio and ngrok wiring, where the thing that usually trips people up is unverified caller IDs on trial accounts.
What is tool calling?
Tool calling (also called function calling) lets the model call a function you’ve described, based on what the caller asked. It then uses what the function returns to write its reply. You give the model a list of tools with typed parameters. It never runs anything itself. It just says, in effect, “call get_order_status with order_id = A1001,” and hands control back to you. You run the function and give it the result.
If you’re already running ConversationRelay, the reassuring part is that none of this touches the voice layer. Speech-to-text, text-to-speech, and barge-in all keep working exactly as they did. Tool calling lives entirely inside your WebSocket server. This is the bring-your-own-LLM pattern: Twilio owns the voice, your server owns the logic, and the model, along with its bill, stays under your control.
This is the loop we spend the rest of the article building:
Architecture: Twilio owns the voice, your FastAPI server owns the logic, and it calls OpenAI and your backend in a loop.
Twilio and OpenAI never talk to each other directly. Every hop runs through your server, and that’s the whole reason you get to slip a tool call into the middle.
Provision Twilio and wire up the FastAPI server
The easiest path is a Twilio trial account and one trial number. There’s one required account setting: the Predictive and Generative AI/ML Features Addendum. In the Console you accept it under Voice > Settings > Privacy & Security. Until you do, ConversationRelay won’t run, and calls fail with Twilio error 95250, “AI/ML Features Addendum has not been accepted.” If your account has already accepted it, you won’t be prompted again. Either way, check it first, because the failure it causes is easy to mistake for a bug in your own code.
The server is just two endpoints. When a call comes in, Twilio fetches a webhook URL to ask what to do, and you answer with TwiML that hands the call off to ConversationRelay and on to your WebSocket:
from xml.sax.saxutils import escape
from fastapi import FastAPI, Request
from fastapi.responses import Response
from app import config
app = FastAPI()
WELCOME = "Hi! I can check an order status or book an appointment. How can I help?"
@app.get("/twiml")
@app.post("/twiml")
async def twiml(request: Request) -> Response:
ws_url = f"wss://{config.PUBLIC_HOST}/ws"
xml = (
'<?xml version="1.0" encoding="UTF-8"?>'
"<Response><Connect>"
f'<ConversationRelay url="{escape(ws_url)}" welcomeGreeting="{escape(WELCOME)}" />'
"</Connect></Response>"
)
return Response(content=xml, media_type="text/xml")
The second endpoint is the WebSocket that TwiML points at. Once the call connects, ConversationRelay opens a persistent connection to /ws and streams JSON frames over it for the rest of the call. The first frame is a setup message carrying call metadata like the callSid. After that, you get a prompt frame every time the caller finishes talking, an interrupt frame if they cut in over the agent, and an errorframe when something breaks. ConversationRelay defines more message types than these, including DTMF input and outbound commands to play audio or switch language, but these are the ones this agent needs. You reply with text frames, which ConversationRelay turns into speech. The connection stays open for the whole call, so it’s the natural place to keep per-call state, most importantly the running conversation history the model needs to stay coherent across turns.
Twilio dials into your server, so the wss:// endpoint has to be reachable from the public internet. While you’re developing, ngrok http 8000 gives you a public host. Drop that host into PUBLIC_HOST and point your number’s voice webhook at https:///twiml. Watch out for one thing: by default ngrok gives you a new host each time you start it, so if you re-tunnel you have to update PUBLIC_HOSTand restart uvicorn, or the wss:// URL in your TwiML ends up pointing at a dead address. The most common problem is unverified caller IDs on trial accounts, where you can only dial numbers you’ve already verified until you upgrade.
Define the tools and a mock backend
Real systems are messy, so we mock them with two in-memory stores. Think of them as the seam where your database or internal API eventually plugs in. What matters is the shape of the functions, not what’s inside them.
ORDERS = {
"A1001": {"item": "Wireless headphones", "status": "shipped", "eta": "Tuesday June 30"},
"A1002": {"item": "Standing desk", "status": "processing", "eta": "next week"},
"A1003": {"item": "Mechanical keyboard", "status": "delivered", "eta": "delivered Monday"},
}
SLOTS = ["Monday at 10 AM", "Monday at 2 PM", "Wednesday at 11 AM", "Friday at 4 PM"]
_booking_counter = 5000
def get_order_status(order_id: str) -> dict:
order_id = (order_id or "").strip().upper()
order = ORDERS.get(order_id)
if order is None:
return {"found": False, "order_id": order_id}
return {"found": True, "order_id": order_id, **order}
def book_appointment(slot: str, name: str) -> dict:
global _booking_counter
# Tolerant match: caller speech rarely matches a slot string exactly.
matched = next((s for s in SLOTS if slot.lower() in s.lower() or s.lower() in slot.lower()), None)
if matched is None:
return {"booked": False, "reason": "slot_unavailable", "available_slots": list(SLOTS)}
SLOTS.remove(matched)
_booking_counter += 1
return {"booked": True, "confirmation_id": f"C{_booking_counter}", "slot": matched, "name": name}
We hand the model two tools instead of one on purpose, so it actually has to pick the right function based on what the caller wants rather than reaching for the only option available. The schemas describe each tool to the model, and a dispatch table maps the names back to the real callables:
from app import backend
TOOLS = [
{"type": "function", "function": {
"name": "get_order_status",
"description": "Look up the status and ETA of a customer's order by its order ID.",
"parameters": {"type": "object",
"properties": {"order_id": {"type": "string", "description": "e.g. 'A1001'"}},
"required": ["order_id"]}}},
{"type": "function", "function": {
"name": "book_appointment",
"description": "Book an appointment. Only call once you have a slot AND the caller's name.",
"parameters": {"type": "object",
"properties": {"slot": {"type": "string"}, "name": {"type": "string"}},
"required": ["slot", "name"]}}},
]
DISPATCH = {"get_order_status": backend.get_order_status,
"book_appointment": backend.book_appointment}
Adding a tool later takes two edits: one schema, one dispatch entry.
The model also needs a voice-aware system prompt. Everything it says gets read out loud, so we ban markdown and bullet points, keep answers to a sentence or two, tell it to spell out order IDs one character at a time, and require it to have both a slot and a name in hand before it books anything.
The tool-calling loop
This is the core of the article. The loop takes the conversation so far plus the caller’s latest words, asks OpenAI what to do, runs whatever tool the model asks for, feeds the result back, and keeps going until the model returns plain text to speak:
async def respond(history, user_text, on_tool_start=None):
history.append({"role": "user", "content": user_text})
interim_sent = False
for _ in range(MAX_TOOL_ROUNDS):
completion = await client.chat.completions.create(
model=config.OPENAI_MODEL, messages=history,
tools=TOOLS, tool_choice="auto",
)
message = completion.choices[0].message
if not message.tool_calls: # plain text -> speak it
answer = (message.content or "").strip()
history.append({"role": "assistant", "content": answer})
return answer
if on_tool_start and not interim_sent: # kill the dead air, once
interim_sent = True
await on_tool_start()
history.append(message.model_dump(exclude_none=True))
for call in message.tool_calls:
args = json.loads(call.function.arguments or "{}")
result = DISPATCH[call.function.name](**args) # run the real work
history.append({"role": "tool", "tool_call_id": call.id,
"content": json.dumps(result)})
# loop: the model now sees the result and writes the reply
A few lines there are quietly doing a lot. The history list gets mutated in place and lives on the WebSocket connection, so on every turn the model sees the whole call, including orders the caller mentioned earlier and the name they already gave. We append the assistant’s tool-call turn verbatim with model_dump() before we append the results, because the OpenAI API expects each tool response to answer a preceding tool_calls message, paired and in order. Skip the assistant message and the API rejects the follow-up request. The for loop is capped by MAX_TOOL_ROUNDS so a confused model asking for tool after tool can’t spin forever on a live call. Once it hits the cap, we return a fallback the agent can actually say instead of just hanging.
That on_tool_start callback is there for a reason. A tool call means another round-trip: model, then function, then model again. On a phone call, even a second of dead air feels like the line dropped. So the first time a tool is about to run, we play the caller a quick “Let me check that for you.” while the real work happens behind it. It fires once per turn rather than once per tool, so the caller never hears it pile up. The WebSocket handler wires it in:
@app.websocket("/ws")
async def conversation_relay(ws):
await ws.accept()
history = agent.new_history()
while True:
msg = await ws.receive_json()
if msg.get("type") == "prompt" and msg.get("last", True):
async def interim():
await ws.send_json({"type": "text", "token": "Let me check that for you.",
"last": True})
answer = await agent.respond(history, msg["voicePrompt"], on_tool_start=interim)
await ws.send_json({"type": "text", "token": answer, "last": True})
Now place a real call and watch the loop run. Here’s one full turn, from the caller’s question to the spoken answer. The question arrives as a prompt frame, the model asks for a tool, your server runs it, and the answer comes back as speech:
Sequence of one turn: caller speaks, ConversationRelay transcribes, the server calls OpenAI, OpenAI requests a tool, the server runs it against the backend and feeds the result back, then the spoken answer returns
You ask, “Where’s my order A1001?” ConversationRelay transcribes it into a prompt frame. The model reads it and, rather than answering, returns a get_order_status call with order_id set to A1001. The server runs the function, gets back {"status": "shipped", "eta": "Tuesday June 30"}, appends that to the history, and calls the model again. This time the model has the fact it was missing, and it speaks: “Your order, A 1 0 0 1, has shipped and arrives Tuesday June 30th.” You hear the “Let me check that for you” line during the gap, so the lookup never turns into silence.
Then you say, “Book me Monday at 10 AM, my name is Sam.” This time the model reaches for the othertool, book_appointment, off the same menu, because the intent has changed. It pulls both the slot and the name out of that one sentence, the server takes the slot and hands back a confirmation id, and the model reads it out. Try it without the name, just “book me Monday at 10,” and the model asks for your name first, because the prompt told it to and the schema marks both fields required. That’s what we were after. The agent didn’t walk through a process or recite a script. It picked the right function from your words and got the work done, over the phone.
Voice Record: The call on your phone.
Here’s an exchange with the example backend, two turns using two different tools, with spelled-out IDs and the interim line covering each lookup. The model phrases each reply on the fly, so the exact wording varies from call to call:
Agent : Hi! I can check an order status or book an appointment. How can I help?
Caller: Where's my order A1001?
Agent : Let me check that for you.
Agent : Your order, A 1 0 0 1, which is for wireless headphones, has been shipped
and is expected to arrive on Tuesday, June thirtieth.
Caller: Great, book me Monday at 10 AM, my name is Sam.
Agent : Let me check that for you.
Agent : You're booked for Monday at 10 AM, Sam. Your confirmation ID is C 5 0 0 1.
The model never once says “tool,” “function,” or “database.” All the plumbing stays invisible to the caller. The mechanics live on your server, and the person on the phone just hears a helpful assistant that happens to know real things about their account.
Production notes and next steps
Two rough edges you’ll hit in production. The first is interruptions. When a caller talks over the agent, ConversationRelay sends an interrupt frame with utteranceUntilInterrupt, and you’ll want to trim the last assistant turn in your history down to what was actually said out loud. Otherwise the model “remembers” words the caller never heard. The second is compliance.
ConversationRelay can support PCI-compliant and HIPAA-eligible workflows, but only when you configure it with compliant text-to-speech and transcription providers, and for HIPAA, sign a Business Associate Agreement with Twilio. The default providers aren’t guaranteed to be compliant, so don’t route card numbers or health data through them without checking.
A few natural places to go from here. You can add more tools behind the same dispatch table, and the loop itself never changes. You can stream the reply in smaller text frames as the model produces them, so the caller starts hearing the answer sooner instead of waiting for one final block. And once the agent is doing work worth watching, you can layer in Twilio Conversational Intelligence for transcripts and analytics.
The end result is the same ConversationRelay voice agent you already had, now backed by a WebSocket server that can reach into your systems mid-call. Not a chatbot with a phone number, but an agent with hands. The full project is on GitHub. Clone it, point the tools at your own backend, and call your number.