Configure HTTP Streaming failover

In a real-world clustered HTTP Streaming scenario, you want your site to offer robust playback, even in the face of server-side streaming problems. HTTP Streaming failover, new in FMS 4.5.2, lets you control how FMS handles certain streaming problems.

Overview of HTTP Streaming failover

HTTP Streaming failover features primarily address the issues of liveness and dropout.

The following figure illustrates a simple two-server cluster of packagers with a Reverse Proxy controlling access. The explanations that follow use this figure as a reference.

Packagers in a two-server cluster
  • Liveness is a server-side situation in which a packager advertises a stale bootstrap (that is, a stale view of a live stream). Although there are other potential scenarios, the following describes one liveness situation:

    1. The stream has been running normally for some time on the server side.

    2. Packager 2's connection to the encoder fails. The connection between Packager 2 and the proxy continues running. Suppose that at the time of this failure, the last successfully written fragment on Packager 2 was still #700 with current media time as X.

    3. A short amount of time elapses, in this case, 10 fragment intervals.

    4. The server-side now suffers from liveness. Packager 2 is serving stale bootstrap information. Packager 2 indicates that there has been no update in the bootstrap after time X, even though more recent fragments are available on Packager 1 (that is, the current media time is greater than X.)

    5. Playback stalls at the player end.

  • Dropout - is a server-side situation in which a packager has gaps in its bootstrap (that is, gaps in its fragment list). The following scenario describes a dropout situation:

    1. The stream has been running normally for some time on the server side.

    2. Packager 2's connection to the encoder fails. The connection between Packager 2 and the proxy continues running. Suppose that at the time of this failure, the last successfully written fragment on Packager 2 was still #700.

    3. A short amount of time elapses, in this case, 10 fragment intervals.

    4. Packager 2's encoder connection is restored and remains restored for a long time.

    5. The server-side now suffers from dropout. Packager 2's bootstrap is missing entries for #701-710, even though those fragments are still available on Packager 1.

    6. Playback stalls at the player end.

HTTP Streaming failover provides a two-part solution:

  • Best-effort fetch

  • Control plane management

Enable best-effort fetch

Best-effort fetch enables the OSMF and iOS video players to continue playback as normally as possible in the presence of short-term liveness and dropout problems on the server-side.

You enable best-effort fetch by adding and configuring a <bestEffortFetchInfo> element in the manifest.xml file on the server side, as as illustrated in the following:

<?xml version="1.0" encoding="UTF-8"?> 
<manifest xmlns="http://ns.adobe.com/f4m/2.0"> 
  <dvrInfo windowDuration="7200"/> 
  <bestEffortFetchInfo 
    segmentDuration="40.000" 
    fragmentDuration="4.000" 
    maxForwardFetches="2" 
    maxBackwardFetches="2" 
  /> 
  <baseURL>http://www.example.com/media</baseURL> 
  <media href="low.f4m" bitrate="512" /> 
  <media href="medium.f4m" bitrate="1296" /> 
  <media href="high.f4m" bitrate="3500" /> 
</manifest>

No client-side configuration is required. For more information, see Manifest.xml.

OSMF

The OSMF 2.0 player adds client-side robustness by supporting best-effort fetch. Specifically, when best-effort fetch is enabled on the server, OSMF 2.0 attempts fetches for fragments that have not been advertised in the bootstrap, but are expected to be present.

For more information on OSMF 2.0 client side functionality, see the OSMF documentation on SourceForge.

HLS

For iOS Video players, FMS 4.5.2 enables best-effort fetch for HLS through settings in the <bestEffortFetchInfo> element in the manifest.xml file.

Configure HTTP Streaming failover server settings

You configure server settings for HTTP Streaming failover in the httpd.conf and manifest.xml files.

Configure HDS and HLS settings in httpd.conf. For more information on server-side configuration in the httpd.conf file, see Configure live and on-demand HTTP Streaming at the server level (httpd.conf). Configure the bestEffortFetchInfo element in the manifest.xml file, as described in Manifest.xml.

Note: Best-effort fetch relies on configuring the back-end reverse-proxy with HTTP 503 failover, meaning that when the reverse proxy receives a 503 response from a packager, it retries the request with the next packager in the pool. If all packagers generate a 503, the proxy generates an error that it caches for less than a fragment interval. You can configure the HTTP error code, as described in Configure live and on-demand HTTP Streaming at the server level (httpd.conf).

Code the control plane application

To implement HTTP Streaming failover, you write a client application that manages the state of events and streams by using a set of REST-based control plane APIs. Control plane is a router term and in effect, that is what your client application does through these APIs.

The basic workflow is as follows:

  • Begin repeatedly checking stream status at a set interval. The exact interval depends on your configuration.

  • Disable any stream with a bad status code.

  • You could also compare the age of the bootstrap file and disable the stream for which the bootstrap is older than some predetermined interval.

FMS control plane REST interface request path

The FMS control plane REST interface request path differs, depending on whether you are using HDS, HLS with adaptive bit-rate (ABR), or HLS with a single bit-rate stream:

  • HDS: http://server/ctrlplane/application-name/instance-name/event-name.f4m/

  • HLS with ABR: http://server/ctrlplane/application-name/instance-name/event-name.m3u8/

  • HLS with single bit-rate: http://server/ctrlplane/application-name/instance-name/event-name/stream-name.m3u8/

For example:

  • HDS: http://www.example-server.com/ctrlplane/livepkgr/_definst_/liveevent.f4m/

  • HLS with ABR: http://www.example-server.com/ctrlplane/livepkgr/_definst_/liveevent.m3u8/

  • HLS with single bit-rate: http://www.example-server.com/ctrlplane/livepkgr/_definst_/liveevent/livestream.m3u8/

You use HTTP GET or POST, depending on the endpoint and parameters passed.

Query stream status

Description

Returns stream status.

Endpoint

status

Result

If the .disabled file exists in the streams directory for the respective associated stream, the control plane module returns "disabled" in the status element for the particular stream. This endpoint returns stream age for each stream (multiple streams in case of MBR) calculated as current time in minutes - last modified time for the bootstrap file.The up element indicates whether the stream is up. It is set to false if the bootstrap has not been updated in specified number of seconds (configurable as MaxBootstrapAge). The global up element is set to true if all bootstrap up elements are true, otherwise it is set to false. The endpoint sets the upper level status to disabled if all streams are disabled, otherwise it is set to enabled.

The done element indicates whether the stream is finished.

Sample request

[GET] http://www.example-server.com/ctrlplane/livepkgr/_definst_/livestream.f4m/status

Sample response

This is an example response for an MBR stream in which all streams are disabled:

200 OK 
 
<?xml version="1.0" encoding ="UTF-8"?> 
<controlplane-event-status xmlns="http://ns.adobe.com/hds/controlplane/status/1.0"> 
    <manifest-file> 
        /livepkgr/_definst_/livestream.f4m 
    </manifest-file> 
    <status> 
        disabled 
    </status> 
    <up> 
        false 
    </up> 
    <done> 
        false 
    </done> 
    <bootstrap name="livestream2 "> 
        <up> 
            false 
        </up > 
        <age> 
            75 
        </age> 
        <status> 
            disabled 
        </status> 
        <done> 
            false 
        </done> 
    </ bootstrap> 
    <bootstrap name="livestream 3"> 
        <up> 
            false 
        </up > 
        <age> 
            75 
        </age> 
        <status> 
            disabled 
        </status> 
        <done> 
            false 
        </done> 
    </ bootstrap> 
    <bootstrap name="livestream 1"> 
        <up > 
            false 
        </up> 
        < age> 
            75 
        </age> 
        <status> 
            disabled 
        </status> 
        <done> 
            false 
        </done> 
    </ bootstrap> 
</controlplane-event-status>

Disable an event

Description

Disables all streams for an event (works for both HDS and HLS streams). You send this as an HTTP POST request with no request body.

Endpoint

disable

Result

Upon receiving this request, the control plane module creates a .disabled file in all streams directories (multiple streams directories for an MBR stream) and returns an XML formatted response indicating success or failure for each stream.

If a client subsequently requests streams for the event, the Apache module (mod_f4fhttp or mod_hlshttp) checks for the existence of a .disabled file before processing the request. If a .disabled file exists, the module returns a 503 error (configurable in httpd.conf).

Sample request

[POST] http://www.example-server.com/ctrlplane/livepkgr/_definst_/livestream.f4m/disable

Sample response

<?xml version="1.0" encoding="UTF-8"?> 
<controlplane-response xmlns="http://ns.adobe.com/hds/controlplane/status/1.0"> 
    <manifest-file> 
         /livepkgr/_definst_/livestream.f4m 
     </manifest-file> 
    <stream name="livestream2"> 
        <status-code> 
             1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
    <stream name="livestream3"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
    <stream name="livestream1"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
</controlplane-response>

Disable a stream

Description

Disables a stream (works for both HDS and HLS streams). You send this as an HTTP POST request with the stream name specified in the body.

Disable a stream if you detect a liveness problems or to take a faulty packager out of production for some other reason. As mentioned below, the Apache module returns a 503 error, making the faulty packager a non-entity.

Endpoint

disable

Result

Upon receiving this request, the control plane module creates a .disabled file in the streams directory and returns an XML formatted response indicating success or failure.

If a client subsequently requests the stream, the Apache module (mod_f4fhttp or mod_hlshttp) checks for the existence of a .disabled file before processing the request. If a .disabled file exists, the module returns a 503 error (configurable in httpd.conf).

Sample request

[POST] http://www.example-server.com/ctrlplane/livepkgr/_definst_/livestream.f4m/disable

<controlplane-request> 
    <stream> livestream2</stream> 
</controlplane-request>

Sample response

<controlplane-response xmlns="http://ns.adobe.com/hds/controlplane/status/1.0"> 
    <manifest-file> 
        /livepkgr/_definst_/livestream.f4m 
    </manifest-file> 
    <stream name="livestream2"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
</controlplane-response>

Enable an event

Description

Enables all streams for an event (works for both HDS and HLS streams). You send this as an HTTP POST request with no body.

Endpoint

enable

Result

Upon receiving this request, the control plane module deletes any existing .disabled files in all streams directories (multiple streams directories for an MBR stream) and returns an XML formatted response indicating success or failure for each stream.

If a client subsequently requests streams for the event, the Apache module (mod_f4fhttp or mod_hlshttp) processes the request.

Sample request

[POST] http://www.example-server.com/ctrlplane/livepkgr/_definst_/livestream.f4m/enable

Sample response

<?xml version="1.0" encoding="UTF-8"?> 
<controlplane-response xmlns="http://ns.adobe.com/hds/controlplane/status/1.0"> 
    <manifest-file> 
        /livepkgr/_definst_/livestream.f4m 
    </manifest-file> 
    <stream name="livestream2"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
    <stream name="livestream3"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
    <stream name="livestream1"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
</controlplane-response>

Enable a stream

Description

Enables a stream (works for both HDS and HLS streams). You send this as an HTTP POST request with the stream name specified in the body. This request enables a stream and removes the .disabled file.

Note: Streams are enabled by default.

Endpoint

enable

Result

Upon receiving this request, the control plane module deletes any existing .disabled file in the streams directories and returns an XML formatted response indicating success or failure.

If a client subsequently requests the stream, the Apache module (mod_f4fhttp or mod_hlshttp) processes the request.

Sample request

[POST] http://www.example-server.com/ctrlplane/livepkgr/_definst_/livestream.f4m/enable

<controlplane-request> 
    <stream>livestream2</stream> 
</controlplane-request>

Sample response

<?xml version="1.0" encoding="UTF-8"?> 
<controlplane-response xmlns="http://ns.adobe.com/hds/controlplane/status/1.0"> 
    <manifest-file> 
        /livepkgr/_definst_/livestream.f4m 
    </manifest-file> 
    <stream name="livestream2"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
</controlplane-response>

Mark an event as done

Description

Signals that an event is complete. You send this as an HTTP POST request with no body.

You signal an event as done so that the client is notified when content ends. This achieves two things: the control plane does not activate failover, and client applications can take appropriate action to signal the end of content to the user.

Endpoint

done

Result

Upon receiving this request, the control plane module creates a .done file in all streams directories (multiple streams directories for an MBR stream) and returns an XML formatted response indicating success or failure for each stream.

Sample request

[POST] http://www.example-server.com/ctrlplane/livepkgr/_definst_/livestream.f4m/done

Sample response

<?xml version="1.0" encoding="UTF-8"?> 
<controlplane-response xmlns="http://ns.adobe.com/hds/controlplane/status/1.0"> 
    <manifest-file> 
        /livepkgr/_definst_/livestream.f4m 
    </manifest-file> 
    <stream name="livestream1"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
    <stream name="livestream2"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
    <stream name="livestream3"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
</controlplane-response>

Mark a stream as done

Description

Signals that a stream complete. You send this as an HTTP POST request with the stream name specified in the body.

You signal a stream as complete so that the client is notified when content ends. This achieves two things: the control plane does not activate failover, and client applications can take appropriate action to signal the end of content to the user.

Endpoint

done

Result

Upon receiving this request, the control plane module creates a .done file in the associated streams directory and returns an XML formatted response indicating success or failure.

Sample request

[POST] http://www.example-server.com/ctrlplane/livepkgr/_definst_/livestream.f4m/done

<controlplane-request> 
    <stream name="livestream1"></stream> 
</controlplane-request>

Sample response

<?xml version="1.0" encoding="UTF-8"?> 
<controlplane-response xmlns="http://ns.adobe.com/hds/controlplane/status/1.0"> 
    <manifest-file> 
        /livepkgr/_definst_/livestream.f4m 
    </manifest-file> 
    <stream name="livestream1"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
</controlplane-response>

Mark an event as in-progress

Description

Signals that an event is still in progress. You send this as an HTTP POST request with no body. This request undoes a /done request.

Endpoint

inProgress

Result

Upon receiving this request, the control plane module deletes any existing .done files in all streams directories (multiple streams directories for an MBR stream) and returns an XML formatted response indicating success or failure for the stream.

Sample request

[POST] http://www.example-server.com/ctrlplane/livepkgr/_definst_/livestream.f4m/inProgress

Sample response

<?xml version="1.0" encoding="UTF-8"?> 
<controlplane-response xmlns="http://ns.adobe.com/hds/controlplane/status/1.0"> 
    <manifest-file> 
        /livepkgr/_definst_/livestream.f4m 
        </manifest-file> 
        <stream name="livestream1"> 
            <status-code> 
                1 
            </status-code> 
            <status-desc> 
                Command executed successfully. 
            </status-desc> 
        </stream> 
        <stream name="livestream2"> 
            <status-code> 
                1 
            </status-code> 
            <status-desc> 
                Command executed successfully. 
            </status-desc> 
        </stream> 
        <stream name="livestream3"> 
            <status-code> 
                1 
            </status-code> 
            <status-desc> 
                Command executed successfully. 
            </status-desc> 
        </stream> 
</controlplane-response>

Mark a stream as in-progress

Description

Signals that a stream is still in progress. You send this as an HTTP POST request with the stream name specified in the body. This request undoes a /done request.

Endpoint

inProgress

Result

Upon receiving this request, the control plane module deletes any existing .done file in the associated streams directory and returns an XML formatted response indicating success or failure for each stream.

Sample request

[POST] http://www.example-server.com/ctrlplane/livepkgr/_definst_/livestream.f4m/inProgress

<controlplane-request> 
    <stream name="livestream1"></stream> 
</controlplane-request>

Sample response

<?xml version="1.0" encoding="UTF-8"?> 
<controlplane-response xmlns="http://ns.adobe.com/hds/controlplane/status/1.0"> 
    <manifest-file> 
        /livepkgr/_definst_/livestream.f4m 
    </manifest-file> 
    <stream name="livestream1"> 
        <status-code> 
            1 
        </status-code> 
        <status-desc> 
            Command executed successfully. 
        </status-desc> 
    </stream> 
</controlplane-response>