Skip to content
Docs

Build a self-hosted AI phone agent with Twilio

Voxos answers calls to your Twilio number and places calls with a goal, using OpenAI, Gemini or xAI realtime voice over LiveKit on your own machine.

This guide connects a local Voxos install to a Twilio number, tests a call in each direction, and keeps callers you don’t trust on a short leash.

What you need

Section titled “What you need”
  • A local Voxos stack from the Quick Start, with sign-in to the console working.
  • A Twilio account with a voice-capable number, an API key (its SID and secret) and the account’s Auth Token.
  • An API key for one realtime voice provider: OpenAI, Google or xAI.
  • The ngrok CLI with an auth token, so Twilio can reach your machine.

1. Give the API a public address

Section titled “1. Give the API a public address”

Twilio delivers calls and texts to webhooks, so the API needs a public HTTPS address. In a second terminal, start a tunnel to the local API:

Terminal window
make ngrok

Put the tunnel’s https:// address in .env as API_PUBLIC_BASE_URL, then apply it:

Terminal window
make up

Twilio’s audio streams use the same host over wss://. Connect the number only after this step, because Voxos gives Twilio its webhook addresses at that moment. Public webhooks has more on the tunnel.

2. Choose a voice

Section titled “2. Choose a voice”

Open Providers and save the key for your voice provider. Then open Agent and pick the voice provider, model and voice for the agent that will take calls. For OpenAI, gpt-realtime-2.1-mini is the recommended default for routine calls; Voice and LiveKit compares the models.

3. Connect LiveKit

Section titled “3. Connect LiveKit”

Every call runs in a LiveKit room. With make up, the API and the voice worker run in containers, so the LiveKit address you save must resolve inside them, and in your browser too if you use in-app voice. ws://localhost:7880 points each container back at itself, and ws://livekit:7880 only resolves inside Compose.

Give the bundled LiveKit dev server one name that works everywhere by adding this line to your machine’s /etc/hosts:

127.0.0.1 livekit

Then open Connection and save ws://livekit:7880, with the dev server’s API key devkey and secret secret. A public LiveKit endpoint, such as a LiveKit Cloud project, works too; see the Docker Compose notes. If you run the API and the worker directly on your machine instead, as in local Python development, ws://localhost:7880 works as it is.

4. Connect the Twilio number

Section titled “4. Connect the Twilio number”

Open Phone, choose Twilio, and enter a label, the Account SID, the API Key SID and secret, and the Auth Token. Voxos then lists the numbers on the account. Select yours, with its voice and SMS capabilities.

Voxos creates webhook routes for voice, SMS, status updates and the audio stream, and points the number’s Twilio webhooks at them. If you disconnect the number later, Voxos puts back the webhook settings it replaced. See Twilio setup for the details.

5. Save your contacts

Section titled “5. Save your contacts”

Open Contacts and add the people the agent should know, each with a phone number in E.164 form, such as +447700900123. Voxos matches callers by their exact number.

Turn on the admin flag only for people you trust with the agent’s full set of approved tools. Everyone else, including saved contacts without the flag and unknown or withheld numbers, gets a limited set of tools and a caller-safe prompt that leaves out your instructions and memories. Caller ID can be spoofed, so keep high-impact tools away from phone callers. See Contacts and caller access.

6. Call the agent

Section titled “6. Call the agent”

Call your Twilio number. Voxos checks Twilio’s signature, starts a voice worker for the call, and the agent answers. The call shows up in the recent activity on the Phone page.

7. Have the agent call someone

Section titled “7. Have the agent call someone”

Agents place calls through a phone tool, and that tool needs a grant. Open Tools and allow the call tool for your agent. Connecting the number already lets the default agent search your contacts, unless you have set that yourself.

Then ask in Chat, for example: “Call Sam and ask whether Tuesday at 10:30 still works.” The agent looks Sam up in your contacts and hands the saved contact and the goal to the call tool. Voxos gets the voice model ready before it dials, and the agent says nothing until Sam answers, then opens with the goal. Goal-directed outbound calls explains each stage.

If something goes wrong

Section titled “If something goes wrong”

Before relying on the setup, run the four-direction smoke test with numbers you are allowed to call and text.

Next steps

Section titled “Next steps”

Privacy