Kazoo v3 Single or Multiple Server VoIP Telephony Platform Install Guide

Kazoo is an open source, scalable, distributed, cloud-based VoIP telephony platform.  It is designed to handle anything from small offices to small countries.  Kazoo relies heavily on other mature, stable, open source applications including Kamailio, Freeswitch, and Bigcouch.   

This guide shows how to install an entire Kazoo system on one or more CentOS v6 x64 servers from RPM.  It can be used for testing and development or production scenarios.

Assumptions: 

• *64bit RHEL/CentOS v6 (there are no 32bit Kazoo RPMs)   
• *Console text mode (init 3)
• *Installation done as root user (#)
• *A single ethx interface with a single static IP address
•  

Quickstart

If you want to get right to it the install is as follows.

Single All-in-One Server
All Servers > Bigcouch > Freeswitch  > Kazoo  > Post Install > Basic Administration

Three Servers
Server1 >  All Servers  >  Bigcouch  > Server2 > All Servers  >  Freeswitch   > Server3 >  All Servers  >>  Kazoo > 
Post Install > Basic Administration

Seven Servers
Server1 >  All Servers  >>  Bigcouch  >> Server2 >  All Servers  >>  Bigcouch  >> Server3 >  All Servers  >>  Bigcouch  >> Cluster together >
Server 4 >  All Servers  >>  Freeswitch  >> Server 5 >  All Servers  >>  Freeswitch  >>
Server 6 >  All Servers  >>  Kazoo  >> Server 7> >  All Servers  >  Kazoo >
Post Install > Basic Administration
 

Overview

System Layout

In the following diagram OpenSIPS proxy has been replaced by Kamailio SIP proxy as of v3.  The AMQP messaging layer uses RabbitMQ messaging broker.  The HAProxy connection between Freeswitch and Bigcouch is not shown.  That connection is used to serve media files (ie, music on hold, voicemail recordings, prompts) which are all stored on the Bigcouch server cluster.

Kazoo System Architecture Layout Diagram

Server Separation

If not using a single all-in-one server, the most basic multiple server system for testing consists of 3 separate servers: 

1 Bigcouch Database Server
1 Freeswitch Server
1 Kazoo Server

Any one server failure would cause the whole system to fail.  The most basic multiple server system for production consists of 7 servers:

3 Bigcouch Database Servers, configured as a cluster
2 Freeswitch Servers
2 Kazoo Servers

3 is the minimum recommended database cluster size due to quorom.  In laymans terms, you need at least 3 members to resolve a majority rules dispute with any one member. For testing and development it is possible to cluster 3 separate Bigcouch database instances on one server. 

Software Separation

This guide assumes the following separation of software between servers:

Bigcouch Database Servers: Bigcouch
Freeswitch Servers: HAProxy, Freeswitch
Kazoo Servers: HAProxy, RabbitMQ, Kamailio SIP Proxy, Kazoo eCall Manager, Kazoo Applications, Kazoo GUI

Erlang Cookies and Hostnames

Software applications that use Erlang (RabbitMQ, Bigcouch, Freeswitch, Kazoo) use TCP/IP sockets, hostnames, and cookie authentication to communicate with each other.  Erlang servers must therefore have consistent and correct TCP naming configurations.  

Each Erlang application must also share the same cookie.  The cookie is just a string of alphanumeric characters.  It can be as long or short as you like.  All cookies are set the same by default in the Kazoo RPMs.  This guide shows how to change that cookie so as not to be using the publicly known one.

Erlang Reference link.

Public vs Local Interface   

This guide assumes that always using the public interface and always using the FQDN is the most correct way of configuring everything in all cases.  

For some scenarios, using 127.0.0.1 and localhostcan achieve the same result.   This is often the case with the single all-in-one server.  This is also useful for locking down servers in a production environment.  How to do that is beyond the scope of this document.

Install

All Servers

yum -y update && yum -y groupinstall core && yum -y groupinstall base

wget -P /etc/yum.repos.d/ -N http://repo.2600hz.com/2600hz.repo

​hostname -f
hostname -i

# Add the following to the /etc/hosts file if it does not already exist.
# "xx.xx.xx.xx" is the static ethx IP address 
# "somedomain.tld" is a valid FQDN in the /etc/sysconfig/network file.
# A valid local FQDN is in the form somelocaldomain.local.
# A valid local and global FQDN is in the form somepublicdomain.com (for example).
# "somedomain" is the first part of the FQDN before the first (".").

xx.xx.xx.xx   somedomain.tld somedomain

To change the assigned hostname edit /etc/sysconfig/network and /etc/hosts and reboot.  Multiple server installs usually require the FQDN to be resolvable by all other servers for most of the following configurations.  

Using the IP address instead of the FQDN, for some of the following configurations, can sometimes get around the requirement to use resolvable FQDN for some settings, but not all.  The only way to guarantee a proper configuration, in all situations, is to always use the FQDN and always make sure it is resolvable by all other servers.   

Disable firewall

service iptables save && service iptables stop && chkconfig iptables off

sestatus

If not edit /etc/selinux/config and reboot

SELINUX=disabled

Set timezone

Kazoo assumes UTC system time.  From there it is converted to the individual account timezones.  

ln -sf /usr/share/zoneinfo/UTC /etc/localtime

service ntpd start
chkconfig ntpd on
​date

Bigcouch Database Server

yum install -y kazoo-bigcouch-R15B

For clustered Bigcouch servers, each server must share the same Erlang magic cookie.  Changing the magic cookie from the default is optional for development but recommended for production.

MAGIC_COOKIE="some_cookie"

sed -i "s/^-setcookie .*$/-setcookie $MAGIC_COOKIE/g" /etc/kazoo/bigcouch/vm.args 

​service bigcouch restart

Clustering

To cluster Bigcouch servers, run the following commands on one of them.  In this example we are running the command on the server with the  FQDN  db01.somedomain.com to cluster it with db02 and db03. 

curl -X PUT db01.somedomain.com:5986/nodes/[email protected] -d {}
curl -X PUT db01.somedomain.com:5986/nodes/[email protected] -d {}

The servers must be able to resolve the other servers FQDN's so they can communicate with each other.  The name resolution is done by a daemon called epmd. 

Verify the servers are clustered by running this on each server.

curl 127.0.0.1:5984/_membership   

Note: Bigcouch only adds newly created databases to a cluster. So it is best to create the cluster first before installing Kazoo.   The Kazoo installation is when the databases get created.  If you add a server to a cluster that has existing databases you have to manually replicate the databases to the new server and add the new server details in 127.0.0.1:5986/dbs/databasename  by_node and by_range fields. 

-----------------------------------------------
For all-in-one or single Bigcouch server

Do not do this for clustered Bigcouch servers.

sed -i "s/^r=.*$/r=1/g" /etc/kazoo/bigcouch/local.ini
sed -i "s/^q=.*$/q=1/g" /etc/kazoo/bigcouch/local.ini
sed -i "s/^w=.*$/w=1/g" /etc/kazoo/bigcouch/local.ini
sed -i "s/^n=.*$/n=1/g" /etc/kazoo/bigcouch/local.ini

service bigcouch restart

Freeswitch Server

yum install -y haproxy kazoo-freeswitch-R15B

Configure HAproxy

Create a symbolic link to the Kazoo HAproxy config file.

rm -rf /etc/haproxy/haproxy.cfg
ln -s /etc/kazoo/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg

Assuming the default haproxy.cfg file now has 127.0.0.1:15984 listening to 127.0.0.1:5984it will work for an all-in-one server as is.

For multiple servers, edit the file to add the Bigcouch server hostnames and IP addresses. Here is a link to an example file.  

For consistency, hostname should be the FQDN of the Bigcouch server(s).  It is used for internal purposes such as logging so it does not need to be resolvable.

Erlang Cookie

The Freeswitch Erlang cookie must match the cookie on the Kazoo server.

MAGIC_COOKIE="some_cookie"
FS_CONFIG="/etc/kazoo/freeswitch/autoload_configs/kazoo.conf.xml"

sed -i "s/<param name=\"cookie\" .*$/<param name=\"cookie\" value=\"$MAGIC_COOKIE\" \/>/g" $FS_CONFIG

Start/restart and set to start on boot. 

service haproxy restart
service freeswitch restart
chkconfig haproxy on
chkconfig freeswitch on

Check that Freeswitch and command line mode are working.  

fs_cli

To exit fs_cli type Ctl-D or /exit.

Kazoo Server

yum install -y haproxy httpd kazoo-kamailio kazoo-R15B kazoo-ui

Update Kamailio configurations

For this example, the RabbitMQ server represented by RABBIT_IP is on the Kazoo server.  It is listed as a separate variable to show that it could potentially be a separate server. 

KAZOO_IP=$(hostname -i)
RABBIT_IP=$(hostname -i)
FQDN=$(hostname -f)
KAMAILIO_CONFIG="/etc/kazoo/kamailio/local.cfg"

sed -ri 's/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b'/$KAZOO_IP'/g' $KAMAILIO_CONFIG
sed -ri 's/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b':5672/$RABBIT_IP:5672'/g' $KAMAILIO_CONFIG

# The following command will only work the first time it is used.
# Manually verify the default file uses "kamailio.2600hz.com"
# Recent RPM's have used different domains.
# The following command would need to be altered if that is the case.

sed -i 's/kamailio.2600hz.com/'$FQDN'/g' $KAMAILIO_CONFIG

#The following variable assignment is only for all-in-one server install
FREESWITCH_IP=$(hostname -i)

sed -ri 's/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b'/$FREESWITCH_IP'/g' /etc/kazoo/kamailio/dbtext/dispatcher

Configure RabbitMQ

If using an external RabbitMQ server, replace $RABBIT_IP with the IP address of the server.  For a single all-in-one server this can probably remain at the default 127.0.0.1 if you prefer.

RABBIT_IP=$(hostname -i)

sed -i "s|uri = \"amqp.*$|uri = \"amqp://guest:guest@$RABBIT_IP:5672\"|g" /etc/kazoo/config.ini

Configure HAproxy

Create an HAproxy symlink   

For single all-in-one server this will have already been done in the Freeswitch setup.

rm -rf /etc/haproxy/haproxy.cfg
ln -s /etc/kazoo/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg

Add your Bigcouch server hostnames and IP addresses to the above file.  Here is a link to an example file.  For single all-in-one server this will have already been done in the Freeswitch setup. 

Configure the Kazoo UI

KAZOO_IP=$(hostname -i)

# The following command will only work the first time it is used.
/bin/sed -i s/'api.2600hz.com'/$KAZOO_IP/g /var/www/html/kazoo-ui/config/config.js

Recent RPM's use an IP address instead of api.2600hz.com for some reason.  If that is the case then you would want to do something like the following instead.  Also check that the urls are http not https.  Also the ports have been changed to 8443 for some reason.   If that is the case the ports should be set to 8000.

KAZOO_IP=$(hostname -i)

/bin/sed -ri 's/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b'/$KAZOO_IP'/g' /var/www/html/kazoo-ui/config/config.js

This optional configuration sets the apache web server root folder to the Kazoo-UI folder.

/bin/sed -i 's#/var/www/html#/var/www/html/kazoo-ui#g' /etc/httpd/conf/httpd.conf

Cookies

Edit /etc/kazoo/config.ini and set the cookies to what they were set on each application earlier in this procedure.  When done, all cookies here and in each application will all be exactly the same cookie.  If not then applications will not be able to communicate with each other via Erlang.

MAGIC_COOKIE="some_cookie"

sed -i "s/^cookie.*$/cookie = $MAGIC_COOKIE/g" /etc/kazoo/config.ini

Start Kazoo Services 

service rabbitmq-server restart 
service rsyslog restart
service haproxy restart
service kz-whistle_apps restart
service kz-ecallmgr restart
service kamailio restart
service httpd restart

If Kamailio fails to start, ensure that /etc/kazoo/kamailio/local.cfg is properly configured with the server IP address.

Set services to start on boot.

chkconfig --add rabbitmq-server
chkconfig rabbitmq-server on

chkconfig --add kz-ecallmgr
chkconfig kz-ecallmgr on

chkconfig --add kz-whistle_apps
chkconfig kz-whistle_apps on

chkconfig haproxy on
chkconfig httpd on

Post Install

On all Kazoo servers and Freeswitch servers, test that the server can reach the database through HAproxy.

curl localhost:15984

​{"couchdb":"Welcome","version":"1.1.1","bigcouch":"0.4.2"}

On each Kazoo/Kamailio server, check that Kamailio dispatcher is distributing calls to the Freeswitch server(s).

kamctl fifo ds_list

On one of the Kazoo servers, import the media files into the database.

/opt/kazoo/utils/media_importer/media_importer /opt/kazoo/system_media/*.wav

Ecall Manager

Add Kamailio IP's as "authoritative" and incoming servers as "trusted".   To do that every Kamilio FQDN and IP must be added to the system_config > ecallmgr > acls field in the database.  The following command run on each Kazoo/Kamailio server accomplishes that.  Alternatively, the information for all Kazoo/Kamailio servers can be added manually from any one Kazoo server command line.

# $FQDN variable should be assigned the Kazoo hostname assuming Kamailio resides on the Kazoo server.
# $KAMAILIO_IP should be assigned the Kazoo IP assuming Kamailio resides on the Kazoo server. 
FQDN=$(hostname -f)
KAMAILIO_IP=$(hostname -i)

/opt/kazoo/utils/sup/sup -n ecallmgr ecallmgr_maintenance allow_sbc $FQDN $KAMAILIO_IP

Attach Ecallmgr to the Freeswitch server(s) by adding an entry in the system_config > ecallmgr > fs_nodes field in the database.  For multiple Freeswitch servers rerun the command from any Kazoo server and manually change $FQDN for each Freeswitch server

# The following variable is for single all-in-one server only.
# For separate servers replace $FQDN with the FS server resolvable FQDN.
FQDN=$(hostname -f)

/opt/kazoo/utils/sup/sup -n ecallmgr ecallmgr_maintenance add_fs_node freeswitch@$FQDN

To verify/change the above information directly on the database, browse to the IP of any one Bigcouch server. 

http://xx.xx.xx.xx:5984/_utils 

At the bottom right change Rows per page to 100 in order to see everything.

Flush cache for changes to take effect.

/opt/kazoo/utils/sup/sup whapps_config flush
/opt/kazoo/utils/sup/sup -n ecallmgr ecallmgr_config flush

For a complete list of sup commands refer to this link.

On the Freeswitch server(s), check if Ecall Manager is connected.

# fs_cli
> erlang nodes list
or
># fs_cli -x 'erlang nodes list'

You should see at least one node.

Check SIP status

On Freeswitch, check if the SIP interface is working.

# fs_cli
> sofia status
or
# fs_cli -x 'sofia status'

You should see at least one profile running.  If not try disable TLS in /etc/kazoo/freeswitch/sip_profiles/sipinterface_1.xml and restart Freeswitch.

Create Master Account

On one of the Kazoo servers, create the master account for logging into Kazoo UI.   Use all lowercase names.

service kz-whistle_apps restart

ACCOUNTNAME="someaccountname"
SOMEREALM="somerealm.com"
LOGINNAME="someusername"
PASSWORD="somepassword"

/opt/kazoo/utils/sup/sup crossbar_maintenance create_account $ACCOUNTNAME $SOMEREALM $LOGINNAME $PASSWORD

Browse to http://kazoo_server_IP/kazoo-ui/ and verify you can log in with the master account.

Login/Email = $LOGINNAME
Password = $PASSWORD
Account Name = $ACCOUNTNAME

$SOMEREALM is used internally to determine which subaccount a registering device belongs to.   It needs to be internally unique and in FQDN format but does not need to be resolvable. 

To understand the account hierarchy refer to this link.    In case that link does not work for some reason, this basic hierarchy diagram might be of some help.

To understand how account credentials are used refer to this link.  In case that link does not work, this account login diagram and this device lookup diagram might help.

Kazoo UI Basic Administration 

To better understand how to configure users, devices, call flows etc. refer to this guide.  

First thing to do is log into master account, click on the down arrow on the upper right, click on App Store, and enable Accounts & Hosted PBX.

For setting up your first device, it is important to note that you do not connect a device directly to the Master Account or the next account level down (Reseller), even though the GUI will not prevent that or complain.  You need to first set up a Reseller account under the Master account, set up a user under the Reseller account, and then set up a device and connect it to that user.  

For configuring a device where your 'realm' is not a resolvable FQDN, set domain and/or proxy and/or registar server  to the 'realm'.  Outbound proxy should be set to the IP address of your Kazoo/Kamailio server.  The device has no way of finding the server if you do not do that.  Configuration examples for different devices are here.  Once you have a device connected and registered try an echo test by dialing 555-555-5552.  

To do anything useful you need to create call flows.  A good start would be to create an extension number for your first user and then set up a voicemail box.  The step by step procedure with screen shots is here.  You will also probably want to enable the *97 feature code and set up a voicemail greeting.