Wholesale Cluster Deployment Guide

This guide demonstrates deploying a 3-node Wholesale high-availability cluster, including MySQL database, Nginx load balancing, and SipFlow signaling collection.

1. Architecture Overview

                     ┌─────────────────────┐
                     │   SIP Load Balancer  │
                     │  (Nginx/OpenSIPS)    │
                     └──────────┬──────────┘
                ┌───────────────┼───────────────┐
                ▼               ▼               ▼
         ┌────────────┐ ┌────────────┐ ┌────────────┐
         │  Node 1    │ │  Node 2    │ │  Node 3    │
         │ 10.0.0.11  │ │ 10.0.0.12  │ │ 10.0.0.13  │
         │ RustPBX    │◄──► RustPBX  │◄──► RustPBX  │
         │ +Wholesale │    +Wholesale │    +Wholesale│
         └─────┬──────┘ └─────┬──────┘ └─────┬──────┘
               │              │              │
               └──────────────┼──────────────┘
                              ▼
                     ┌─────────────────┐
                     │  MySQL 8.4 LTS  │
                     │  10.0.0.20:3306 │
                     └─────────────────┘
                              │
                     ┌────────┴────────┐
                     ▼                 ▼
              ┌────────────┐   ┌────────────┐
              │  SipFlow A │   │  SipFlow B │
              │ 10.0.0.21  │   │ 10.0.0.22  │
              └────────────┘   └────────────┘

2. Node Planning

Image version: docker.cnb.cool/miuda.ai/rustpbx:0.4.5

3. MySQL Deployment

3.1 Install (Docker)

docker run -d --name mysql \
  --restart always \
  -e MYSQL_ROOT_PASSWORD=RootP@ss2026 \
  -e MYSQL_CHARACTER_SET_SERVER=utf8mb4 \
  -e MYSQL_COLLATION_SERVER=utf8mb4_unicode_ci \
  -v /data/mysql:/var/lib/mysql \
  -p 3306:3306 \
  mysql:8.4-oracle

3.2 Create Database and User

docker exec -it mysql mysql -uroot -pRootP@ss2026 <<'EOF'
CREATE DATABASE IF NOT EXISTS rustpbx CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER IF NOT EXISTS 'rustpbx'@'%' IDENTIFIED BY 'WsDbP@ss2026';
GRANT ALL PRIVILEGES ON rustpbx.* TO 'rustpbx'@'%';
FLUSH PRIVILEGES;

-- Optimization (connection pool)
SET GLOBAL max_connections = 500;
SET GLOBAL innodb_buffer_pool_size = 4294967296;
EOF

3.3 Verify

mysql -h 10.0.0.20 -u rustpbx -pWsDbP@ss2026 -e "SELECT 1"

3.4 MySQL Parameter Recommendations

4. SipFlow Cluster Deployment

4.1 SipFlow A (10.0.0.21)

docker run -d --name sipflow-a \
  --restart always \
  --network host \
  -v /data/sipflow:/data/sipflow \
  docker.cnb.cool/miuda.ai/rustpbx:0.4.5 \
  /app/sipflow --addr 0.0.0.0 --port 3000 --http-port 3001 --root /data/sipflow

4.2 SipFlow B (10.0.0.22)

docker run -d --name sipflow-b \
  --restart always \
  --network host \
  -v /data/sipflow:/data/sipflow \
  docker.cnb.cool/miuda.ai/rustpbx:0.4.5 \
  /app/sipflow --addr 0.0.0.0 --port 3000 --http-port 3001 --root /data/sipflow

4.3 Verify

curl http://10.0.0.21:3001/health
curl http://10.0.0.22:3001/health

5. RustPBX Wholesale Node Deployment

5.1 Prepare Directories

Run on each node (using Node 1 as an example):

mkdir -p /opt/rustpbx/{config/trunks,config/routes,config/acl}

5.2 config.toml (Node 1 — 10.0.0.11)

http_addr = "0.0.0.0:8080"
database_url = "mysql://rustpbx:WsDbP@ss2026@10.0.0.20:3306/rustpbx"
external_ip = "10.0.0.11"
log_level = "info"

[proxy]
addr = "0.0.0.0"
udp_port = 5060
modules = ["acl", "auth", "registrar", "call"]
media_proxy = "auto"
addons = ["wholesale"]
max_concurrency = 1000

[proxy.dos]
enabled = true
max_cps_per_ip = 50

[console]
session_secret = "wholesale-cluster-secret-2026"
allow_registration = false

# Core cluster: registration/presence sync
[cluster]
peers = [
  { addr = "10.0.0.12", sip_port = 5060, ami_port = 8080 },
  { addr = "10.0.0.13", sip_port = 5060, ami_port = 8080 },
]

# SipFlow remote cluster
[sipflow]
[sipflow.remote]
flush_interval_secs = 5

[[sipflow.remote.nodes]]
udp = "10.0.0.21:3000"
http = "http://10.0.0.21:3001"

[[sipflow.remote.nodes]]
udp = "10.0.0.22:3000"
http = "http://10.0.0.22:3001"

# Recording upload (optional)
[recording]
enabled = false

# CDR configuration
[callrecord]
enabled = true

5.3 config/wholesale.toml (Node 1)

# Wholesale cluster configuration
[cluster]

[[cluster.peers]]
name = "node-2"
url = "http://10.0.0.12:8080"
api_key = "wholesale-cluster-api-key-2026"

[[cluster.peers]]
name = "node-3"
url = "http://10.0.0.13:8080"
api_key = "wholesale-cluster-api-key-2026"

# Route cache
[route_cache]
capacity = 10000
ttl_secs = 30

# Circuit breaker defaults
[circuit_breaker]
failure_threshold = 5
open_duration_secs = 30
half_open_probes = 1
failure_codes = [503, 408, 504]

# Sliding window ASR/ACD monitoring
[sliding_window]
enabled = true
window_secs = 300
max_events_per_trunk = 10000
asr_alert_threshold = 30.0
min_calls_for_stats = 10

5.4 Other Node Configuration

Node 2 (10.0.0.12): Copy Node 1’s configuration and modify these fields:

# config.toml
external_ip = "10.0.0.12"

[cluster]
peers = [
  { addr = "10.0.0.11", sip_port = 5060, ami_port = 8080 },
  { addr = "10.0.0.13", sip_port = 5060, ami_port = 8080 },
]

# config/wholesale.toml
[[cluster.peers]]
name = "node-1"
url = "http://10.0.0.11:8080"
api_key = "wholesale-cluster-api-key-2026"

[[cluster.peers]]
name = "node-3"
url = "http://10.0.0.13:8080"
api_key = "wholesale-cluster-api-key-2026"

Node 3 (10.0.0.13): Similarly, set external_ip = "10.0.0.13" and point peers to Node 1 and Node 2.

5.5 Start Nodes

On each node:

docker run -d --name rustpbx \
  --restart always \
  --network host \
  -v /opt/rustpbx/config.toml:/app/config.toml \
  -v /opt/rustpbx/config:/app/config \
  docker.cnb.cool/miuda.ai/rustpbx:0.4.5

5.6 Verify Startup

# Check HTTP service
curl http://10.0.0.11:8080/api/sbc/data
curl http://10.0.0.12:8080/api/sbc/data
curl http://10.0.0.13:8080/api/sbc/data

# View logs
docker logs -f rustpbx 2>&1 | head -50

On first startup, Node 1 will automatically run database migrations. Subsequent nodes will skip them.

6. Load Balancer Configuration

6.1 SIP Load Balancing (Nginx Stream)

On the SIP Load Balancer node:

# /etc/nginx/nginx.conf
stream {
    upstream sip_backend {
        # Source IP hash: route REGISTER/INVITE from the same client to the same node
        hash $remote_addr consistent;

        server 10.0.0.11:5060;
        server 10.0.0.12:5060;
        server 10.0.0.13:5060;
    }

    server {
        listen 5060 udp;
        proxy_pass sip_backend;
        proxy_timeout 30s;
        proxy_responses 1;
    }
}

6.2 HTTP Load Balancing (Nginx HTTP)

http {
    upstream rustpbx_http {
        ip_hash;  # Session stickiness (required for console login)
        server 10.0.0.11:8080;
        server 10.0.0.12:8080;
        server 10.0.0.13:8080;
    }

    server {
        listen 80;
        client_max_body_size 50m;

        location / {
            proxy_pass http://rustpbx_http;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

6.3 RTP Ports

RTP traffic does not pass through the load balancer; it goes directly to the node handling the call. Therefore:

• Each node’s external_ip must be set to the node’s own reachable IP
• The firewall must allow RTP port range on each node (default 12000-42000/udp)

7. Wholesale Business Configuration

7.1 Create Superuser

Create on any node (database is shared, only needs to be done once):

docker exec -it rustpbx /app/rustpbx --create-superuser admin Admin@2026

7.2 Log In to Console

Access via the load balancer address: http://<lb-ip>/console/login

7.3 Configure Carrier Trunks

config/trunks/carrier_a.toml (synced to all nodes):

# Create the same trunk configuration on each node
cat > /opt/rustpbx/config/trunks/carrier_a.toml << 'EOF'
[[trunk]]
name = "carrier-a"
dest = "sip:10.0.1.100:5060"
direction = "outbound"
codec = ["pcmu", "pcma", "g729"]
max_calls = 300
max_cps = 50
weight = 100
EOF

Recommended: use rsync or Git for syncing:

# Sync from Node 1 to other nodes
rsync -avz /opt/rustpbx/config/ 10.0.0.12:/opt/rustpbx/config/
rsync -avz /opt/rustpbx/config/ 10.0.0.13:/opt/rustpbx/config/

7.4 Rate Decks / Routing Profiles / Tenants

These are managed via the web console in the database. All nodes share the same database, so no per-node configuration is needed.

After configuration, click the “Reload” button in the console. Each node reloads from the database into its in-memory Trie.

8. Cluster Verification

8.1 Cluster Status

Wholesale → Cluster Management: Should show node-2 and node-3 online.

8.2 Call Test

Initiate a test call through the SIP LB and check:

1. Call establishes normally
2. Wholesale → CDR Management: CDR has been generated
3. Wholesale → Cluster → Active Calls: Cross-node call list is visible

8.3 Failover Test

1. Stop Node 2: docker stop rustpbx (on 10.0.0.12)
2. Initiate a call through the SIP LB → should auto-route to Node 1 or Node 3
3. Restore Node 2: docker start rustpbx
4. Check the cluster page for Node 2 coming back online

8.4 Data Consistency Verification

1. Create a rate entry on Node 1’s console
2. Click “Reload” on Node 2’s console
3. Use the price calculator on Node 2 to verify the new rate is visible

9. Limit Allocation Strategy

Since the SIP LB uses source IP hashing, actual traffic may not be perfectly even. Recommend leaving 20% headroom on limits.

Actual configuration is adjusted per node under Wholesale → Tenant Details → Settings.

10. Monitoring System

10.1 Prometheus Scrape

# prometheus.yml
scrape_configs:
  - job_name: 'rustpbx-wholesale'
    scrape_interval: 15s
    static_configs:
      - targets:
          - '10.0.0.11:8080'
          - '10.0.0.12:8080'
          - '10.0.0.13:8080'

10.2 Key Metrics

10.3 Health Checks

# Each node
curl http://10.0.0.11:8080/healthz
curl http://10.0.0.12:8080/healthz
curl http://10.0.0.13:8080/healthz

11. Daily Operations

11.1 Configuration Sync

11.2 Rolling Restart

# Restart one node at a time, ensure at least 2 nodes remain online
ssh 10.0.0.11 "docker restart rustpbx"
sleep 30  # Wait for startup
ssh 10.0.0.12 "docker restart rustpbx"
sleep 30
ssh 10.0.0.13 "docker restart rustpbx"

11.3 Database Backup

# Daily full backup
mysqldump -h 10.0.0.20 -u root -pRootP@ss2026 \
  --single-transaction --quick \
  rustpbx | gzip > /backup/rustpbx_$(date +%Y%m%d).sql.gz

# Retain 30 days
find /backup -name "rustpbx_*.sql.gz" -mtime +30 -delete

11.4 CDR Data Maintenance

-- Check CDR data volume
SELECT COUNT(*), DATE(call_start) FROM wholesale_cdrs
GROUP BY DATE(call_start) ORDER BY DATE(call_start) DESC LIMIT 7;

-- Recommended monthly partitioning (MySQL 8.4)
ALTER TABLE wholesale_cdrs PARTITION BY RANGE (TO_DAYS(call_start)) (
    PARTITION p202605 VALUES LESS THAN (TO_DAYS('2026-06-01')),
    PARTITION p202606 VALUES LESS THAN (TO_DAYS('2026-07-01')),
    PARTITION p_future VALUES LESS THAN MAXVALUE
);

12. Version Upgrade

# 1. Pull new version
docker pull docker.cnb.cool/miuda.ai/rustpbx:0.4.5

# 2. Upgrade node by node
ssh 10.0.0.11 "docker stop rustpbx && docker rm rustpbx"
ssh 10.0.0.11 "docker run -d --name rustpbx --restart always --network host \
  -v /opt/rustpbx/config.toml:/app/config.toml \
  -v /opt/rustpbx/config:/app/config \
  docker.cnb.cool/miuda.ai/rustpbx:0.4.5"

# Wait for startup, verify health
curl http://10.0.0.11:8080/healthz

# 3. Repeat for Node 2, Node 3