Troubleshooting Playbook
Start every investigation with the Diagnostics panel and relevant logs. This section catalogs common failure patterns, how to isolate them, and recommended tools.
1. Registration / signaling
| Symptom | Investigation | Resolution |
|---|---|---|
| Extension refuses to register | Diagnostics → SIP → Locator registry to confirm bindings and expiry | Verify password, SIP port, firewall rules; reset the password or clear stale bindings when required |
| Trunk status degraded | Run Diagnostics → Trunks probes or OPTIONS probe | Confirm peer IP and auth mode; enable backup trunks in config/trunks |
| INVITE has no response | Use sngrep or Diagnostics → Routing Evaluate to confirm rule hits | Double-check routing matches and ACL permissions |
| ACL blocks unexpected traffic | ACL rules are only evaluated on inbound trunk paths — outbound calls from extensions are not ACL-filtered | Add ACL rules targeting your trunk’s inbound_hosts; internal extension-to-extension calls skip ACL |
| Callee dialog hangs | Stale callee dialogs after transfer or re-INVITE may persist in older versions | Upgrade to v0.4.10+ which streamlined callee dialog management in SipSession |
2. Media & quality
- One-way / no audio:
- Inspect NAT/port mappings between server and peers.
- Ensure
rtp_start_port/rtp_end_portranges are open inconfig.tomland firewalls. - Reproduce via Diagnostics → Web Dialer or a handset, then capture RTP with
tcpdump/sngrepto verify return packets.
- Noise or jitter:
- Switch to lower bitrate codecs.
- Enable the denoise models from
fixtures/or turn on echo cancellation at the endpoint. - Check QoS policies and link bandwidth.
3. Routing & billing
- Routing ineffective: confirm Reload ran and validate
config/routessyntax viatomlcheckor CI. - Wrong route selected: Diagnostics → Routing Evaluate shows the hit rule/trunk; adjust
priorityormatchfilters accordingly. - Billing mismatch: export CDRs from Call Records, compare billing templates, and look for
no_ratealerts caused by missing prefixes.
4. Console / API
- Cannot log in: inspect the
[console]config and DB connection; make sure browser time is accurate to avoid expired tokens. - API returns 500: read
logs/console(or stdout) stack traces; most errors stem from missing config or unfinished DB migrations. - Diagnostics blank page: typically SIP server is down or the user lacks permission; validate
/healthreportsokand grantdiagnosticsaccess.
5. Performance & stability
- High CPU: use
top/btto locate hot threads, lower concurrency or scale out, and check for excessive transcoding. RTP proxying with SRTP encryption is the primary CPU consumer — estimate 5–8 concurrent calls per vCPU. - Growing memory: verify recording buffer cleanup in
callrecord/storage.rs. Long-running calls with large recording buffers (particularly in SipFlow capture mode) can accumulate memory. Restart nodes during maintenance windows if leak patterns are detected. - Call lifecycle leaks: ensure
call-lifecycleresources are released properly. In versions prior to 0.4.10, certain call teardown paths could retain session state. Upgrade to the latest release if you observe memory growth proportional to call volume. - Crashes / restarts: consult
journalctlor container logs—configuration syntax errors or unreachable dependencies (DB/Redis) are common causes. The binary auto-retries with exponential backoff (up to 10 attempts). - Concurrent call limits: per-trunk
max_callsand per-tenantmax_concurrencyare enforced independently. Exceeded limits return SIP 503. Check Diagnostics → Trunks for capacity utilization.
6. Incident workflow
- Gather evidence: screenshots from Diagnostics, log exports, precise timestamps.
- Roll back quickly: if caused by configuration, revert
config/in Git and reload. - Validate fix: place test calls and confirm CDRs/alerts return to normal.
- Document: record root cause, impact, and remediation steps in the internal wiki for future reference.