Configuring performance features

The following three configuration settings impact scale and latency for live media delivery:

• Configure the size of stream chunks

• Send aggregate messages

• Combine audio samples

To configure the server to deliver live media, choose your use case:

Large Scale Broadcast (scale is more important than latency)

In this use case, you want to reach as many people as possible and are willing to allow some latency. You may also want to keep costs down while still reaching as many people as possible. Send large aggregate messages and configure large stream chunk sizes.

Large Scale Broadcast (low latency is more important than scale)

In this use case, you want to keep latency as low as possible while reaching as many people as possible. You are willing to trade some scale for decreased latency. Send smaller aggregate messages. Or, combine audio samples and do not send aggregate messages.

Interactive (lowest latency)

Interactive applications require very low latency. Do not send aggregate messages. For the lowest latency, do not combine audio samples.

If data latency is your concern, combine audio samples. Combining audio samples introduces audio latency but doesn't affect the rest of the stream. Aggregating messages introduces latency to the whole stream.

Interactive applications are the most difficult to scale and require more hardware than the large scale broadcast use cases.

To configure the server to deliver on-demand (vod) media, complete the following tasks:

64-bit servers can address more physical memory than 32-bit servers. When delivering on-demand video on 64-bit servers, increase the size of the recorded media cache size. Only the amount of RAM available on the computer limits the recorded file cache size.

 Configure the recorded media cache

To deliver media that contains only audio,

1. Combine audio samples

2. Send aggregate messages

When a stream is requested from the server, segments of the stream are stored in a cache on the server. As long as the cache has not reached capacity, the server places segments in the cache. Each time a stream attempts to access a segment, the server checks the cache. If the segment is available, the server gives the stream a reference to the segment for playback. If the segment is not available, the server retrieves the segment from its source, inserts it into the cache, and returns a reference to that segment to the stream.

When the cache is full, the server removes unused segments, starting with the least recently used. After removing all unused segments, if there still isn’t enough room for a new segment, the server notifies the client that the stream is not available and makes an entry in the core log file.

If you have cache-full events in the core log file, increase the size of the cache or limit the number of streams playing. For more information about the core log file, see Monitoring and Managing Log Files.

1. Open the rootinstall/conf/fms.ini file.

2. Edit the SERVER.FLVCACHE_MAXSIZE parameter.

   This is the maximum size of the cache, in megabytes. The default value is 500. The file cache shares process address space with the core process. Each core process has a separate file cache. For 32-bit processes, it has a limit of 2 GB in Windows and 3 GB in Linux. For 64-bit installations, the limit is greater. For more information, see Configure a 64-bit server.

   The size of the cache limits the number of unique streams the server can publish. To increase the probability that a requested stream will be located in the recorded media cache, increase the value of SERVER.FLVCACHE_MAXSIZE. To decrease the amount of memory the server process uses, decrease the value of SERVER.FLVCACHE_MAXSIZE. While a large cache size is useful, Adobe recommends that you ensure that your total system memory usage does not exceed the process limit of your OS. Consider memory limits and desired memory and stream performance when utilizing the memory cache.

   Note: Cache settings have no effect on live streams, as live streams do not need or utilize the cache.

3. Restart the server.

There is no exact way to calculate the value of the cache size because it varies depending on the amount of RAM available and the number of other processes that are running. However, you can follow some general guidelines to approximate the recommended size of the cache.

Amount of RAM available for FMS (R) = RAM size - RAM used by the OS and non-FMS processes

Amount of RAM per core (Rc) = Amount of RAM available for FMS (R) / Number of Cores

Recommended File Cache Size = Amount of RAM per core (Rc) - RAM used by each core process

RAM used by each Flash Media Server process varies based on load, and there is no exact figure. You can use 512 MB as a ballpark figure.

RAM size = 8 GB 
RAM used by OS and non-FMS processes = 1.2 GB (approximately) 
Number of FMS core processes allowed = 3 
Amount of RAM available for FMS (R) =  8 - 1.2 = 6.8 
Amount of RAM per core (Rc) =  6.8 / 3 = 2.27 GB 
Recommended file cache size = 2.27 - 0.5 = 1.76 GB

Streams break into chunks as they are written to the network. You can specify the size of a chunk. Large values reduce CPU usage because there are fewer writes. However, large values can delay other content on lower bandwidth connections. The larger the content size and the higher the bandwidth of the receiving connection, the more benefit is gained from larger chunk sizes.

1. Open the Application.xml file.

   Note: You can set these values in an Application.xml file at the VHost level or at the application level. To set the value at the application level, copy an Application.xml file to the application’s folder.

2. In the Client element, set the OutChunkSize element to a value between 128 and 65536 bytes. The default value is 4096 bytes.

   For more information, see Application.xml file.

3. Restart the server.

An aggregate message is a single message that contains a list of submessages. Sending aggregate messages reduces CPU usage and increases server capacity. You can configure applications to deliver aggregate messages to clients running on Flash Player 9.0.60.0 and above. When this setting is disabled, the server breaks up aggregate messages into individual messages before delivering them to clients.

Aggregate messages can be used with live and recorded streams.

An origin server can deliver aggregate messages to an edge server, and an edge server can deliver aggregate messages from the FLV data that is has cached on the disk. An edge server can deliver aggregate messages even if the origin server did not.

If any of these conditions are present, Flash Media Server breaks the aggregated message into its submessages, and the messages are delivered individually.

After you confirm that aggregate messaging is enabled, you can use the settings in the Server.xml file to configure the size of the messages. You can also enable or disable the messages based on the type of stream. For more information, see Streams.

You can also configure message queuing in the Application.xml file. For more information, see Queue.

To handle more connections while broadcasting a live stream, combine audio samples.

1. Open the rootinstall/conf/_defaultRoot_/_defaultVHost_/Application.xml file.

   Note: You can set these values in an Application.xml file at the VHost level or at the application level. To set the value at the application level, copy an Application.xml file to the application’s folder.
2. Locate the StreamManager/Audio/CombineSamples section of the file and set values the following elements:

   Element	Description
   Subscribers	If there are more than this number of subscribers to a stream, audio samples are combined. The default value is 8. To increase live streaming capacity, set this value to 1.
   LoCPU	If the CPU is lower than this value, audio samples are not combined. The default value is 60. To increase live streaming capacity, set this value to 1.
   HiCPU	If the CPU is higher than this value, audio samples are not combined. The default value is 80. To increase live streaming capacity, set this value to 1.
   MaxSamples	Combine this many samples into one message. The default value is 4. To increase live streaming capacity, set this value to 8.

3. Restart the server.

A high connection rate to the server can negatively impact the experience of users already connected to the server.

1. Locate the following code in the Server.xml configuration file:

   <Root> 
       <Server> 
           ... 
           <Protocol> 
               <RTMP> 
                   <Edge> 
                       <MaxConnectionRate>100</MaxConnectionRate>

   Element	Description	Impact
   MaxConnectionRate	The maximum number of incoming connections per second the server accepts, per listener. Listeners are defined in the HostPort element in the Adaptor.xml file. Each port the server is configured to listen on represents a listener. You can set a fractional maximum connection rate, such as 12.5. A value of 0 or -1 disables this feature. The value of this element is a global setting for all listeners. If the element is set to 10 connections per second, each listener has a limit of 10 connections per second. If there are three listeners and the MaxConnectionRate is set to 10, the server imposes a maximum total combined rate of 30 connections per second.	Connections requested at a rate above this value remain in the TCP/IP socket queue and are silently discarded by the operating system whenever the queue becomes too long.

2. Validate the XML and save the XML file.

3. Restart the server.

Sometimes clients have left an application without the server-end of the socket knowing. This behavior can happen when a router is unplugged or crashes without sending TCP close messages for the sockets it is managing, when a laptop is undocked from a docking station, and so on. To reclaim these resources for new and active clients, the server can close the idle clients.

Flash Media Server sends a small ping message over sockets that have been quiet for awhile. If the client at the far end has gone away, the server's TCP eventually stops retransmitting that data. When that happens, the socket is shut down and the server can clean up any associated client object, and so on.

Elements in the Server.xml, Vhost.xml, and Application.xml configuration files specify how often the server should check for idle clients. When a client has been idle longer than the maximum idle time (10 minutes, by default), the server sends a status message to the NetConnection object (the client) with the code property set to NetConnection.Connect.Idle followed by NetConnection.Connect.Closed. The server closes the client connection to the server and writes an x-status code of 432 in the access log. The server also writes a message such as “Client x has been idle for y seconds” in the core and event logs.

To close idle connections, enable the feature in the Server.xml file. Once you enable the feature in the Server.xml file, you can disable the feature for individual virtual hosts or individual applications in the Vhost.xml files and Application.xml files. The values defined in the Server.xml configuration file apply to all clients connected to the server, unless the values are defined in the Vhost.xml file (the Vhost.xml values override the Server.xml values). The values defined in the Application.xml file override the values defined in the Vhost.xml file.

In some scenarios, you might want to change how applications are assigned to server processes. When you start the server, you are starting a process called FMSMaster.exe (Windows) or fmsmaster (Linux). Application instances run in processes called FMSCore.exe (Windows) and fmscore (Linux). The master process is a monitor that starts core processes when necessary. There can be only one master process running at a time, but there can be many core processes running at a time.

You can configure how applications are assigned to server processes in the Process section of the Application.xml configuration file. Settings in an Application.xml file in a virtual host folder (for example, rootinstall/conf/_defaultRoot_/_defaultVHost_/Application.xml) apply to all the applications running in that virtual host. Settings in an Application.xml file in an application’s folder (for example, rootinstall/applications/myApp/Application.xml) apply only to that application. The following is the XML structure:

<Application> 
    <Process> 
        <Scope>vhost</Scope> 
        <Distribute numprocs="3"></Distribute> 
        <LifeTime> 
            <RollOver></RollOver> 
            <MaxCores></MaxCores> 
        </LifeTime> 
        <MaxFailures>2</MaxFailures> 
        <RecoveryTime>300</RecoveryTime> 
    </Process> 
    ... 
</Application>

Configure a process scope

 The Scope tag specifies at which level application instances are assigned to core processes. An application instance can run by itself in a process or it can run in a process with other instances. Enter one of the following values for the Scope element.

Value	Description
adaptor	All application instances in an adaptor run together in a process.
vhost	All application instances in a virtual host run together in a process. This is the default value.
app	All instances of a single application run together in a process.
inst	Each application instance runs in its own process. This provides the best application isolation and uses the most system resources. Running every instance in its own process can create many processes. You can set the Distribute numprocs attribute to a value greater than 1 to distribute instances across that number of processes.

Configure the number of core processes and how long each process runs

 Specify the number of core processes in the MaxCores tag (the maximum number of core processes that can exist concurrently) and the number of seconds that a core process can run in the RollOver tag. When a core process reaches the limit, any new connections roll over to a new core process.

The following diagram depicts the way in which the server rolls over processes. In the XML, the rollover time is set to 3600 seconds (1 hour), indicating that every hour a new process should start, and the maximum core processes value is set to 3, indicating that the maximum number of processes at any given time is 3:

<Process> 
    <Scope>app</Scope> 
    <LifeTime> 
        <RollOver>3600</RollOver> 
        <MaxCores>3</MaxCores> 
    </LifeTime> 
...

View full size graphic

A. Client connections B. Process 1 starts C. Process 2 starts D. Process 3 starts E. Process 4 starts; Process 1 ends, because the maximum core processes limit was reached

When each process starts, it accepts new connections until the next process starts: that is, when process 1 starts, it accepts new client connections to the application until process 2 starts; process 2 then accepts new client connections until process 3 starts; and so on.

Note that the duration of process 1 might or might not be the full duration specified by the rollover value, because rollover values are calibrated to the real clock time. The duration of process 1 is partially determined by the current time when process 1 starts. For example, as shown in the diagram, when process 1 starts, the current time is 12:20, so the duration of process 1 is only 40 minutes (because it is 40 minutes until the beginning of the hour in real time). The duration of the first process is determined by the clock time; subsequent processes have a duration equal to the specified rollover time.

To disable this feature, set RollOver to 0. This feature is disabled by default.

Note: If you have multiple VHosts with Process/Scope set to adaptor, you must set an identical RollOver value for each VHost.

In stateless applications, such as vod applications, old core processes continue to serve earlier connections. In this case, you can specify a value in the MaxCores tag to limit the maximum number of core processes that can run simultaneously. If the application is not stateless, the server ignores any value you assign to MaxCores and sets it to 1. This ensures that an application instance is not split across multiple processes, but clients are disconnected periodically. To disable this feature, set MaxCores to 0. This feature is disabled by default.

Note: An application is considered stateless if you configure it to distribute clients over multiple processes. To do this, set the Distribute numprocs attribute to a value greater than 1, then set the Distribute tag to clients or set the Scope tag to inst.

The RAW (Record and Watch) file format records live media into configurable chunks that stream to any version of Flash Player. Use the RAW file format to serve long-length, multi-bitrate DVR streams without running into performance issues. The RAW file format records and plays back all streams that Flash Media Server supports, including H.264 video, data-only, audio-only, and so on.

In some scenarios, however, you might want to disable RTMPE (encrypted Real-Time Messaging Protocol). Because RTMPE uses encrypted channels, there is a minor impact on performance; RTMPE requires about 15% more processing power than RTMP. If you don’t control the applications that connect to Flash Media Server and you don’t want them to use RTMPE, you can disable RTMPE at the adaptor level.

To request an encrypted or encrypted tunnelling channel, applications specify rtmpe or rtmpte, respectively, in the NetConnection.connect() URL, for example, nc.connect("rtmpe://www.example.com/myMediaApplication"). If an application specifies RTMPE without explicitly specifying a port, Flash Player scans ports just like it does with RTMP, in the following order: 1935 (RTMPE), 443 (RTMPE), 80 (RTMPE), 80 (RTMPTE).

1. Open the Adaptor.xml file for the adaptor you want to disable (located in rootinstall/conf).

2. Locate the following XML:

   <RTMP> 
   <!-- RTMPE is the encrypted RTMP protocol covering both RTMPE and RTMPTE --> 
   <!-- This is enabled by default, setting enabled to "false will not     --> 
   <!-- allow these protocols on this adaptor                 --> 
       <RTMPE enabled="true"></RTMPE> 
   </RTMP

3. Set the RTMPE enabled attribute to "false".

4. Restart the server.