If you have not stood the deployment up yet, start with
On-Prem / Self-Hosted — it describes
what runs where and the shape of the install. This page assumes it is already
deployed and something is wrong.
Triage first
Match the symptom, then jump to the section that explains the cause and the fix.| Symptom | Likely cause | Section |
|---|---|---|
ss shows no UDP :443 listener, but the box is “up” | Expected — the NIC is owned by the kernel-bypass fast path | NIC ownership |
| Deployed a new build / config, nothing changed | Config and builds are reboot cutovers, not hot reloads | Reboot cutover |
Carrier INVITEs never arrive, or land on the wrong tenant | Missing SRV/CNAME records, or wrong subdomain label | DNS & provisioning |
| QUIC/WebTransport handshake fails from the browser | Missing/expired TLS cert for the brand domain (SNI) | DNS & provisioning |
| Calls reach the agent but trunk auth or provider keys fail | Engine cannot unseal sealed credentials (KEK mismatch) | Sealed credentials |
| Agent answers but turns fail / fall back to a cloud model | On-prem model server not attached, or egress blocked | Inference not attached |
| Reports and recordings stay empty | Control-plane services down, or auth to the event plane misconfigured | Control plane quiet |
NIC ownership: there is no normal socket
The engine takes full ownership of its network interface with a kernel-bypass zero-copy fast path — SIP, RTP, and QUIC all ride one interface without going through the kernel’s UDP stack. The single most common false alarm is an operator runningss -ulpn (or netstat), seeing no UDP :443 and no UDP :5060
listener, and concluding the engine is down.
Check liveness the right way instead:
Confirm the process and its interface
Verify the engine process is running and the fast-path filter is attached to
the interface (your deploy runbook names both). If the process is up and the
filter is loaded, the data plane is live even with no visible socket.
Hit the health endpoint
The control-plane API exposes
/health and /healthz. A green health check
plus a live process is the correct “engine is up” signal — not a socket table.Place a real probe call
Register a softphone against the tenant SIP domain and dial in, or send a test
INVITE from a trunk. A call that lands and answers is the ground truth. See
Call not connecting
if it doesn’t.The fast-path filter is loaded before the engine starts (as a pre-start
step) so that a crash-loop cannot black-hole the interface. If you ever need to
hand the NIC back to the kernel (to debug with ordinary tooling), stop the
engine and unload the filter — you will then see normal sockets again, but no
calls will flow until the engine reclaims the interface.
Reboot cutover: changes need a reboot
Because the process owns the NIC, applying a new build or a config change is a reboot cutover, not a hot reload. If you shipped a change and behavior did not move, the overwhelmingly likely cause is that the engine host was never rebooted onto it.| Symptom | Cause | Fix |
|---|---|---|
| New build shipped, old behavior persists | Running process still holds the NIC on the previous image | Reboot the engine host to bring the data plane up on the new build |
| Trunk / dialplan / agent config edit ignored | NIC-owning modules read owned state at start; not all knobs hot-reload | Reboot; treat engine-side config as a windowed change |
| Brief call outage during a deploy | Expected — the interface quiesces while the engine is down | Deploy inside a maintenance window; drain traffic first |
DNS and tenant provisioning
The engine multiplexes every tenant onto one interface and tells them apart by the leftmost DNS label of the inbound SIPINVITE (and by SNI on the QUIC
plane). If DNS is wrong, calls either never arrive or land on the wrong workspace.
| Symptom | Cause | Fix |
|---|---|---|
Carrier INVITEs never reach the engine | Missing SIP SRV record, or it points at the wrong host | Create the SRV for the tenant SIP domain → engine host; verify with dig SRV |
| Calls arrive but resolve to the wrong tenant (or none) | Wrong leftmost label — <workspace-id>.sip.telequick.dev mismatch | Fix the per-tenant CNAME so the label matches the provisioned workspace id |
| Browser QUIC/WebTransport handshake fails | No valid TLS cert for the brand domain, or expired | Install a valid cert; the transport does SNI across brand domains |
| Console / relay unreachable | portal.telequick.dev, agent.telequick.dev, relay.telequick.dev not pointed | Point each hostname at the correct box (engine vs control plane) |
Verify the per-tenant SIP subdomain
Confirm
<workspace-id>.sip.telequick.dev resolves to your engine host and
that a SIP SRV record routes INVITEs to it. The engine reads the leftmost
label (<workspace-id>) as the tenant, so the label must match the
workspace id exactly.Verify the WebRTC / browser subdomain
Confirm
<workspace-id>.webrtc.telequick.dev resolves to the engine for
browser-legged calls.Verify the console, portal, and relay hostnames
portal.telequick.dev and agent.telequick.dev point at the control-plane
box; relay.telequick.dev points at the relay (same host single-site, or the
relay mesh multi-site).A wildcard
*.telequick.dev record proves nothing about a specific label —
confirm the explicit sip/webrtc/SRV records exist, not just that the
wildcard answers. If trunk INVITEs are the problem, cross-check
SIP trunk issues.Sealed credentials won’t unseal
Tenant secrets — trunk digest passwords, agent provider API keys — are sealed at rest. The engine and the control plane share one key-encryption key (KEK) so the engine can unseal a secret at call time; there are no plaintext secrets on disk. This is what makes the deployment HIPAA / SOC 2 defensible — but if the two sides don’t share the same KEK, the engine holds ciphertext it cannot open.| Symptom | Cause | Fix |
|---|---|---|
| Trunk registration / digest auth fails for every tenant | Engine cannot unseal the sealed trunk secret (KEK mismatch) | Align the engine’s KEK with the control plane’s; reboot the engine |
| Every agent’s provider call returns 401 / auth error | Sealed provider API key cannot be unsealed | Same KEK mismatch — realign, then reboot |
| Works for old config, breaks for newly saved secrets | KEK was rotated on one side only | Re-seal secrets under the current KEK on both sides |
Confirm both hosts share one KEK
The engine host and the control-plane host must be configured with the same
key-encryption key. A mismatch is the number-one cause of “auth fails for
everything at once.”
Re-save the affected secrets
If the KEK was rotated, secrets sealed under the old key are unreadable.
Re-enter and save the trunk secrets and provider keys in the console so they
are re-sealed under the current KEK.
Inference not attached
On a self-hosted engine you can still use the managed-cloud speech providers (cloud realtime, ASR/LLM/TTS) as long as the engine is allowed egress to them. Running your own models on-prem is a separate, explicit step: the model server must be registered with the inference control plane and the agent’s provider must point at your endpoint over QUIC. If either is missing, the agent silently uses whatever provider its config still names.| Symptom | Cause | Fix |
|---|---|---|
| Agent answers, but turns go to a cloud model you meant to replace | Agent provider still set to the cloud provider | Point the agent at your self-hosted endpoint |
| On-prem model attached, but turns error or time out | Model server not registered with the inference control plane, or unreachable over QUIC | Register the server; confirm the engine can reach it |
| Cloud providers fail on-prem | Egress to the provider is blocked by your firewall | Allow egress, or move fully to self-hosted inference |
Register the model server
Register your GPU-backed model server with the inference control plane so the
engine can discover and route to it over QUIC. Full steps are in
Self-Hosted Inference.
Point the agent at your endpoint
Set the agent’s provider/endpoint to your on-prem model. Until you do, the
engine keeps using whatever provider the agent config names — often a cloud
one from an earlier test.
Confirm reachability, then place a call
Verify the engine can reach the model host, then dial in and watch a turn
complete. Use Latency breakdown
to confirm turns are actually hitting your GPUs.
Honest status. The QUIC-based inference control plane that discovers and
routes to your GPU pods is shipped, but the fully on-prem speech-to-speech model
path has not yet been live-verified end-to-end in a customer deployment. If
you need on-prem GPU inference from day one, plan it as a joint bring-up rather
than a drop-in. The managed-cloud model providers work against a self-hosted
engine today if egress is permitted.
Control plane or observability plane is quiet
If calls connect and sound fine but reports, call traces, and recordings never appear, the fault is on the control-plane host, not the engine. The engine emits every call-lifecycle event, QoS/MOS record, and recording to the control-plane services; if those services are down or unauthenticated, the data is produced but never landed.| Symptom | Cause | Fix |
|---|---|---|
| Reports / call traces stay empty | Event pipeline service on the control-plane host is down | Start the control-plane systemd units; confirm they’re healthy |
| Recordings never surface in the console | Recording service or object store down / misconfigured | Check the recording service and object-store units |
| Events silently dropped | The event plane is always TLS + SASL authenticated — no plaintext mode | Fix the SASL credentials on both producer and broker sides |
| A service won’t start at all | There is no Docker — units run against a pinned Node runtime under /opt | Confirm the pinned runtime exists and the unit’s paths are correct |
Check the control-plane units
The admin API, config database, event pipeline, and object store all run as
systemd units (no containers). Confirm each is active and healthy — a stopped
unit is the usual cause of empty dashboards.Verify the event plane authentication
The internal event transport is always TLS + SASL authenticated; there is no
plaintext fallback. A credential mismatch drops events silently — realign the
SASL credentials.
Some in-console dashboards are still being wired end-to-end on self-hosted
installs — if a specific panel is blank while call audio is clearly working,
verify the underlying pipeline is up before assuming a bug. See
Observability overview for what each
panel is fed by.
There is no sandbox
Related
On-Prem / Self-Hosted
The install itself: what runs where, and the reboot-cutover deploy flow.
Deployment Models
Managed, self-hosted, and hybrid — splitting the media and control planes.
Self-Hosted Inference
Register your GPU model server and route turns to it over QUIC.
Call not connecting
When the probe call itself won’t land or answer.