Diagnostics Toolkit

The Diagnostics page and companion APIs concentrate reload workflows, routing simulations, registration introspection, and web-based test calls—this is the primary entrypoint for day-to-day troubleshooting.

1. Reload & preflight checks

  1. How to trigger:
    • UI: Settings → Reload, with switches to reload Routing, ACL, or the full application.
    • API: management endpoints such as /reload/acl, /reload/app, etc., suitable for scripted deployments.
  2. Preflight logic: preflight::validate_reload blocks unsafe changes (removing the [console] section, deleting in-use trunks, ACL syntax errors).
  3. Common hints: Reload would disable the console means [console] is missing in config.toml; trunk-related errors require restoring the trunk record in the database or TOML files.

Reload Panel

2. Reading the Diagnostics tabs

Implemented in templates/console/diagnostics.html, the page exposes multiple tabs:

  • Connection: shows realm, enabled transports (UDP/TCP/TLS/WebSocket), sample credentials, plus a WebRTC ICE probe that validates [ice_servers] entries.
  • Trunks: renders a runtime trunk snapshot. /diagnostics/trunks/test verifies whether an IP or domain matches inbound_hosts/dest and sends carrier OPTIONS probes.
  • Routing: /diagnostics/routes/evaluate simulates rule matching for runtime or database datasets, returning the hit route, selected trunk, rewrite result, queue/IVR action, and optional abort code.
  • SIP:
    • Active SIP dialogs: lists active INVITE dialogs and lets you inspect SDP offer/answer pairs.
    • Locator registry: /diagnostics/locator/* queries or removes bindings to debug registration issues.
  • Web Dialer: embeds JsSIP so you can dial directly in the browser using the credentials from Connection; it captures the latest ~80 WebSocket frames for signaling analysis.

Diagnostics Tabs

3. Logs, health checks, and external tooling

  1. Logging: configure log_level, log_file, and log_rotation in config.toml. File logs use tracing’s structured format with optional log rotation (hourly/daily).
  2. Call records: the callrecord module writes CDRs and recording URLs; the console’s Call Records view reads from models::call_record for reconciliation.
  3. Health probe: handler::ami exposes /health, reporting database, SIP core, and task runner status; load balancers or perfcli --ping can consume it.
  4. Traffic capture: no built-in packet capture button—run sngrep, tcpdump, or reproduce scenarios via Web Dialer then capture RTP/SIP manually on the host.
  5. Prometheus metrics: GET /metrics exports Prometheus-format metrics including SIP registrations, dialogs, trunk call rates, RTP statistics, queue wait times, and system resources.
  6. SipFlow query: query captured SIP/RTP flows via GET /sipflow/flow/{call_id} (signaling) and GET /sipflow/media/{call_id} (media). The console’s SipFlow page provides a UI for browsing captured calls.
  7. Frequency limits: GET /frequency_limits lists active rate-limit blocks; DELETE /frequency_limits clears them. Monitor for abnormal blocking patterns during incident response.
  8. Trunk registrations: GET /trunk_registrations returns the registration status of all outbound trunks, including expiry timers and last error codes.

4. AMI API reference

The diagnostic REST API is served under {ami_path} (default /ami/v1):

MethodPathDescription
GET/healthNode health (version, uptime, active calls, task status, SipFlow stats)
GET/dialogsList all active SIP dialogs with SDP details
GET/hangup/{id}Force-hangup a specific dialog
GET/transactionsList running SIP transactions
POST/shutdownGraceful shutdown
POST/reload/trunksReload SIP trunks from config
POST/reload/routesReload routing rules
POST/reload/aclReload ACL rules
POST/reload/appFull application reload (with preflight validation)
GET/trunk_registrationsGet trunk registration statuses
GET/DELETE/frequency_limitsList or clear frequency limits
GET/sipflow/flow/{call_id}Query SIP signaling flow for a call
GET/sipflow/media/{call_id}Query RTP media flow for a call

5. Typical troubleshooting paths

ScenarioPlaybook
Routing changes not effectiveSettings → Reload → Diagnostics → Routing (Evaluate database & runtime) → inspect config/routes/*.toml
Registration failuresDiagnostics → SIP → Locator registry → confirm bindings → review proxy.acl_rules for blocked IPs
Inbound trunk rejectedDiagnostics → Trunks → test peer IP/domain → adjust inbound_hosts in config/trunks/*.toml if mismatched
One-way audio or media issuesVerify rtp_start_port/rtp_end_port exposure → reproduce via Web Dialer or a handset → capture RTP and inspect NAT/firewall
Billing or queue anomaliesCompare Call Records with billing templates → Diagnostics → Routing to confirm expected queue/route

6. Automation & scripting

  • Reload pipeline: CI jobs that modify config/ or config.toml can call /reload/acl and /reload/app, parsing HTTP responses to ensure preflight success.
  • Health monitoring: curl http://<host>:8080/health; alert or remove the instance if status != ok.
  • Performance baselines: examples/perfcli.rs performs bulk registrations/calls; compare results with Diagnostics → Routing output to validate optimizations.

Mastering this trio—change, reload, validate—keeps incident loops within minutes and dramatically reduces carrier operations overhead.