Vinteo Servers scaling and fault tolerance

To organize a video conferencing service (hereinafter referred to as VCS) based on Vinteo, one physical server is sufficient but to obtain a fault-tolerant system and a system that can withstand high loads it is necessary to use a cluster of several servers.

There are two server roles in the Vinteo system:

• communication server and management server (hereinafter it referred to as the management server);
• media server.

There can only be one management server in the system during operation. The number of media servers is defined as N+1; minimum one media server. The minimum number of physical servers is one; it is combining both roles. When using redundancy we get another management server to hot-swap the primary management server if it fails. There can be only 1 backup management server.

Management server

Management server is the web server and the server that provides interaction via communication protocols (SIP, H.323, WebRTC).

To reserve the management server the following scheme is used: primary - secondary. The primary server is active and serving the system; the secondary server is passive. It monitors the state of the primary server and with an interval of 5 minutes replicates data from the database and from the file system to itself. For redundancy to work correctly it is necessary that the main and backup servers be comparable in power.

If the primary server fails the secondary server takes over the primary role and raises interfaces with system IP addresses registered in the database and turns on management and telephony services.

When the first server returns to service (when turned on) it (the first server) will take on the role of the secondary determining that the primary server is already on the network on the eth1 interface. Thus the primary server becomes the server that will be turned on first and the second one will become the secondary server.

If the primary server loses connection with the secondary server the license becomes temporary and is valid for 72 hours from the moment the connection is lost. During this time it is necessary to solve the technical problem with the failed server or if this is not possible contact Vinteo technical support to resolve the issue with the validity period of temporary licenses.

If the connection between the secondary and primary servers on the eth1 interfaces is lost the situation may occur where both servers become primary. This can lead to unpredictable consequences including complete unavailability of the service.

Media server

Media server is the server that encodes/decodes media streams and builds layouts. To increase system performance it is necessary to use cascading. Cascading is the horizontal scaling of media servers. The management server will strive to load the media servers evenly balancing the load between them, i.e. media servers mutually back up each other.

The media server is based on the Multipoint Control Unit (hereinafter referred to as MCU). The MCU is responsible for processes such as mixing, transcoding and transrating.

Mixing implies the ability to create an image layout in the form in which it would be convenient for the user. Transcoding allows you to change the format of the video stream and transrating makes it possible to optimize the data transfer rate.

To ensure redundancy of media servers in the system it is necessary to have at least N+1 media servers where N is the number of media servers required to serve the maximum number of participants for given video conferencing scenarios. The MCU can also be a management server.

Load distribution occurs as follows: the layout for each conference is built by one of the media servers on which this conference is hosted. At the same time this server must serve all participants whose video is included in any layout of this conference. Therefore when planning the infrastructure it is necessary to take into account the maximum number of participants on the layout and include this parameter in the capacity of the media servers.

When the first participant connects to the conference the management server selects the least loaded media server to host this conference. When the load on the media server processing the conference is over 40% new participants not included in the conference layout are sent for processing (balanced) to other available media servers. Media servers do not receive or send traffic to clients; they only distribute the load of decoding and encoding conference layouts among themselves exchanging traffic with the main server (control server).

If connection with one of the media servers is lost the layout switches to another media server that is the conference does not turn off.

Setting up server reservation and cascading

Server reservation and cascading scheme

When using all the capabilities of the Vinteo server for redundancy and cascading the following videoconferencing network diagram is obtained.

For ease of setup it is recommended to assign the IP address of the eth0 interface on the backup server from the same subnet as the main server.

If addressing is required that differs from that shown in the diagram then you will not be able to configure redundancy and cascading yourself; for this you need to contact Vinteo technical support. If technical support staff cannot connect to the servers via SSH you will be given patches with the necessary addressing.

The eth0 interfaces of the main and backup servers must be connected to the network from which accounts are expected to connect.

It is expected that the connection between the main and backup servers on the eth1 interfaces will be organized directly by cable and the switch between them will create an extra point of failure.

The network listed as 192.0.2.128/25 may not have any external connection other than to the administrator's PC for initial setup. It is understood that it will only be used to connect media servers and the control server.

The 192.0.2.128/25 network is subject to increased requirements for stability and connection speed when cascading. This network does not use error correction mechanisms so even minor losses in this network can lead to artifacts and image distortion in conferences. After successfully setting up the redundancy the main server will take on the primary role and the reserve server will take on the secondary role. To independently configure redundancy and cascading of Vinteo servers follow the instructions described below.

Preparing for setup

1. To configure it yourself IP addressing must correspond to that shown in the diagram from the section “Server reservation and cascading scheme”; 2. For media servers, for ease of configuration, it is recommended to assign IP addresses on the eth0 interface accessible from the administrator’s PC; 3. Get the necessary configuration files from Vinteo technical support:

• config_patch_generate_replicator_keys.bin
• config_patch_to_configure_media_servers.bin
• config_patch_prepare_slave.bin

4. Install the main, reserve and media servers according to the instructions; 5. Update all server software to the latest version. The software of all servers must be the same version; 6. Install licenses on the primary and reserve servers. On the primary server the license must contain a port license and a reservation license.

On the reservation server the license must contain redundancy and ports if servers with Static/Dynamic licensing are used. If you are using a server with Enterprise licensing then only reservation is anough.

Configuring the network of the main and reservation servers

Assign IP addresses (add the necessary interfaces using the "Add" button in the "Settings" -> "Network interface settings section"):

Main server

1. Eth0 is the address to which accounts will be connect (from the internal network or from the DMZ) 2. Eth1 (reservation) - 192.0.2.1 / 255.255.255.252 3. Eth2 (cascading) – 192.0.2.200 / 255.255.255.128 4. Reboot the server to apply the settings

Reserving server 1. Eth0 – is the address from the Eth0 subnet of the main server. After activation of the server reservation will not be used 2. Eth1 (reservation) - 192.0.2.2 / 255.255.255.252 3. Eth2 (cascading) – 192.0.2.199 / 255.255.255.128 (or any free address from the cascading servers subnet. Once activated the reservation will not be used) 4. Reboot the server to apply the settings

Setting up the server reservation

1. Check the availability of the backup server on the main server at 192.0.2.2 on the "Settings" page -> Diagnostics -> Ping

2. On the main server (future primary) in the “Upgrade” section install the patch config_patch_generate_replicator_keys.bin

3. Immediately after installation go to the “Status” section and wait for the operating time to begin counting.

4. As soon as the countdown of the server's operating time begins you need to go to the address https://<ip of the primary server>/config_patch_for_slave_and_media.bin and download the file. IMPORTANT: this file is available for downloading within 5 minutes from the moment the server operating time begins.

5. Install the downloaded patch on the reserve server (future secondary) config_patch_for_slave_and_media.bin in the “Upgrade” section. Wait for the server operating time to begin counting in the “Status” section.

6. Install the config_patch_prepare_slave.bin patch on the backup server (future secondary).

7. Turn off the main server which should become the primary server (192.0.2.1).

8. On the reserve server which should become the secondary server (192.0.2.2); enter the address 192.0.2.1 in the reservation settings. Save settings.

9. Shut down the reserve server.

10. Turn on the main (future primary server (192.0.2.1) and enter the address in the reservation settings: 192.0.2.2. Save the settings.

10. Reboot the main server.

11. После того, как загрузится основной сервер (192.0.2.1), включите резервный сервер (192.0.2.2). В результате 192.0.2.1 будет primary-сервером, а 192.0.2.2 secondary-сервером.

12. After the main server (192.0.2.1) boots up turn on the backup server (192.0.2.2). As a result 192.0.2.1 will be the primary server and 192.0.2.2 the secondary server.

13. View the synchronization status on the "Status" page.

Setting up the server cascading

1. Install the server that will be a media server; update it to the latest version.

2. On the “Network Interface settings” page on the media server for the eth0 interface the IP addressing should be exactly like this on media servers: 192.0.2.201 (202, 203, ...) / 255.255.255.128

You also need to make sure that on the management server the address is set on the appropriate interface - 192.0.2.200 / 255.255.255.128

3. Check that future media servers are available to the management server at the addresses: 192.0.2.201 (202, 203...).

4. Important! If you configured a reservation then go to step 8 and use the file downloaded in the section Setting up a reservation, point 4. Otherwise if you reinstall the config_patch_generate_replicator_keys.bin patch the management server will regenerate the keys and the reservation will be disabled.

5. On the control server in the “Upgrade” section install the patch config_patch_generate_replicator_keys.bin

6. Immediately after installation go to the "Status" section and wait for the operating time to begin counting in the section of the server operating time.

7. As soon as the countdown of the server's operating time begins, go to https://<ip of the main server>/config_patch_for_slave_and_media.bin and download the file. Important: this file is available for downloading within 5 minutes from the moment the server operating time begins.

8. Install the config_patch_for_slave_and_media.bin patch obtained in the previous step or during the redundancy setup process on future media servers. You must wait until the server operation time starts counting in the “Status” section.

9. On the control server in the "Cascading" section add the required number of media servers. It is recommended to indicate mcu1, mcu2, ... mcuN in the server names.

10. If the management server will also use a local media server then in the settings for localmcu you must specify the address 192.0.2.200 in the "IP" and "Local IP" fields. You do not need to fill in the password for local mcu. For other media servers enter the address of the media server in the "IP" field (for example, 192.0.2.201) and in the "Local IP" field: 192.0.2.200. In the "Password" field enter the password for the media server admin user. In the "Port" field enter: 5000.

11.Save changes.

12. You need to wait until all media servers have the green checkmark in their status.

13. On the management server in the "Update" section install the config_patch_to_configure_media_servers.bin patch. You must wait until the countdown of the server operating time begins in the "Status" section.

14. Additional tabs with the states of media servers should appear on the main page of the management server.

Checking cluster statuses

To check whether the reservation is active (whether the secondary server is available) look at its status on the "System" - "Status" page of the web interface on the "Reservation" section:

Decoding the string

Current role; Next is the date (Role set at) when the role was installed;

Last time db change checked at is the time of the last database synchronization between the secondary and primary servers;

Remote(IP) last seen at is the time of the last check by the primary server of the secondary server. The check occurs once every 30 seconds.

When the reservation is running all three parameters will be displayed. If the primary server does not see the secondary server for some reason then in Remote(IP) last seen at the time of the last successful check or never will be written. The web interface will also display the following message: "Attention! The Secondary server is not synchronizing".

To check whether cascading is active, which media servers are currently active and which are not you can go to the "System" - "Settings" page go to the "Cascading" tab. If any of the servers is unavailable its status will have a red cross.

If all servers are available there will be green checkmarks. The status of each server can be viewed separately on the main page.

If the media server is unavailable its tab will display the message: "Server not found".