Troubleshooting Playbook

Start every investigation with the Diagnostics panel and relevant logs. This section catalogs common failure patterns, how to isolate them, and recommended tools.

1. Registration / signaling

SymptomInvestigationResolution
Extension refuses to registerDiagnostics → SIP → Locator registry to confirm bindings and expiryVerify password, SIP port, firewall rules; reset the password or clear stale bindings when required
Trunk status degradedRun Diagnostics → Trunks probes or OPTIONS probeConfirm peer IP and auth mode; enable backup trunks in config/trunks
INVITE has no responseUse sngrep or Diagnostics → Routing Evaluate to confirm rule hitsDouble-check routing matches and ACL permissions
ACL blocks unexpected trafficACL rules are only evaluated on inbound trunk paths — outbound calls from extensions are not ACL-filteredAdd ACL rules targeting your trunk’s inbound_hosts; internal extension-to-extension calls skip ACL
Callee dialog hangsStale callee dialogs after transfer or re-INVITE may persist in older versionsUpgrade to v0.4.10+ which streamlined callee dialog management in SipSession

2. Media & quality

  1. One-way / no audio:
    • Inspect NAT/port mappings between server and peers.
    • Ensure rtp_start_port / rtp_end_port ranges are open in config.toml and firewalls.
    • Reproduce via Diagnostics → Web Dialer or a handset, then capture RTP with tcpdump/sngrep to verify return packets.
  2. Noise or jitter:
    • Switch to lower bitrate codecs.
    • Enable the denoise models from fixtures/ or turn on echo cancellation at the endpoint.
    • Check QoS policies and link bandwidth.

3. Routing & billing

  • Routing ineffective: confirm Reload ran and validate config/routes syntax via tomlcheck or CI.
  • Wrong route selected: Diagnostics → Routing Evaluate shows the hit rule/trunk; adjust priority or match filters accordingly.
  • Billing mismatch: export CDRs from Call Records, compare billing templates, and look for no_rate alerts caused by missing prefixes.

4. Console / API

  • Cannot log in: inspect the [console] config and DB connection; make sure browser time is accurate to avoid expired tokens.
  • API returns 500: read logs/console (or stdout) stack traces; most errors stem from missing config or unfinished DB migrations.
  • Diagnostics blank page: typically SIP server is down or the user lacks permission; validate /health reports ok and grant diagnostics access.

5. Performance & stability

  • High CPU: use top/bt to locate hot threads, lower concurrency or scale out, and check for excessive transcoding. RTP proxying with SRTP encryption is the primary CPU consumer — estimate 5–8 concurrent calls per vCPU.
  • Growing memory: verify recording buffer cleanup in callrecord/storage.rs. Long-running calls with large recording buffers (particularly in SipFlow capture mode) can accumulate memory. Restart nodes during maintenance windows if leak patterns are detected.
  • Call lifecycle leaks: ensure call-lifecycle resources are released properly. In versions prior to 0.4.10, certain call teardown paths could retain session state. Upgrade to the latest release if you observe memory growth proportional to call volume.
  • Crashes / restarts: consult journalctl or container logs—configuration syntax errors or unreachable dependencies (DB/Redis) are common causes. The binary auto-retries with exponential backoff (up to 10 attempts).
  • Concurrent call limits: per-trunk max_calls and per-tenant max_concurrency are enforced independently. Exceeded limits return SIP 503. Check Diagnostics → Trunks for capacity utilization.

6. Incident workflow

  1. Gather evidence: screenshots from Diagnostics, log exports, precise timestamps.
  2. Roll back quickly: if caused by configuration, revert config/ in Git and reload.
  3. Validate fix: place test calls and confirm CDRs/alerts return to normal.
  4. Document: record root cause, impact, and remediation steps in the internal wiki for future reference.
TroubleshootingFlow