GStreamer Training

Gstreamer Overview

IP TV-Gstreamer Overview

GStreamer Abstraction Layer & Plugin

AAMP has two modes of operation - gstreamer player (when JS/Native API's are used by the application) and gstreamer element (when application manages playbin as in the case of html5 video playback)

AAMP as Gstreamer player

In this mode of operation, AAMP manages the gstreamer pipeline. AAMP creates one playbin for each stream ( Audio or Video) and adds these playbins to pipeline. AAMP manages the pipeline state as well as injects data to the playbins through gstreamer appsrc element. AAMP sets caps of appsrc src pads based on the format of injected data. Buffers and events are sent to gstreamer using these appsrc objects.

AAMP as Gstreamer element

In this mode of operation, AAMP is a gstreamer element. Usually, application manages the lifecycle of the playbin which has aamp as one element. There is a dummy element aampsrc to handle protocol "aamp://" and "aamps://" which is replaced to http internally. aamp gstreamer element manages state of AAMP engine as well as gets media data and injects it to next element. aamp gstreamer element exposes dynamic pads which have caps corresponding to the format of injected data on that srcpad. 

...

For HLS, AAMP uses software demux which is a part of AAMP code by default. If codec info is not in the manifest, AAMP does not demux the TS fragments, which causes h/w demux gstreamer element of the platform to be used by playbin to demux the TS fragments. In summary, AAMP outputs elementary streams on using s/w demux, else output mpeg TS buffers which results in hardware demux to be used.

HLS on

...

RPI

HLS on RPI (UVE and S/W Demux)

AAMP manages the pipeline. For audio and video there are separate playbins. AAMP provides audio/video elementary stream to the playbins using appsrc. Playbin connects platform decoder and sink elements using the capability of the  appsrc src pads.

Pipeline graph

Image Removed

HLS on

...

RPI (UVE, H/W Demux)

AAMP manages the pipeline. AAMP provides  TS  to the playbins using appsrc. Playbin connects platform ts-demux, decoder and sink elements.

Pipeline graph of HLS playback :

Image RemovedImage Added

DASH Widevine on

...

RPI(UVE)

AAMP manages the pipeline. For audio and video there are separate playbins. AAMP provides audio/video fragmented mp4 buffers to the playbins using appsrc. Playbin connects qtdemux, platform decoder and sink elements using the capability of the src pads.

...

0009-qtdemux-aamp-tm.patch

https://gerrit.teamccp.com/plugins/gitiles/rdk/yocto_oe/layers/meta-rdk-ext/+/refs/heads/stable2/recipes-multimedia/gstreamer/files/0009-qtdemux-aamp-tm.patch

This patch adds support for PTS restamping of outgoing buffers from qtdemux based on  a custom event aamp_override. App can override the base PTS which  is to be used as base for restamping using property "basePTS". If it is not proved, PTS of first fragment will be used as base PTS. 

...

0021-qtdemux-aamp-tm-multiperiod.patch

https://gerrit.teamccp.com/plugins/gitiles/rdk/yocto_oe/layers/meta-rdk-ext/+/refs/heads/stable2/recipes-multimedia/gstreamer/files/0021-qtdemux-aamp-tm-multiperiod.patch

On period transition during trickmode, PTS of the new fragment changes. To ensure smooth trickplay experience, PTS restamping should be continuous. This patch ensures this using aamp-tm-disc event. Fragments after this event has PTS very different from previous fragments. So base PTS has to be recalculated to have continuous trick play experience at this point. PTS offset is calculated by adding last restamped PTS prior to the event and expected delay between Iframes. Base PTS is reset. PTS offset is added to the recalculated PTS on next fragments. 

...

If content is encrypted with Widevine or Playready DRM, aamp decryptor elements )aampplayreadydecryptor or aampwidevinedecryptor) are automatically plugged in after qtdemux by playbin. These elements decrypt samples using opencdm interface.

Pipeline graph of DASH stream:

Image RemovedImage Added

HTML5 Video Tag based playback on Broadcom

...

Device

Here Webkit manages the pipeline. aampsrc is a dummy gstreamer source just to handle aamp:// protocol. aamp element fetches manifets/playlists and fragments, demuxes TS fragment in case of HLS, injects buffers to the downstream element using srcpads. RDK WPE uses Westeros sink gstreamer element as video sink.

Pipeline graph

Image Removed

Progressive Support in Aamp

...

In order to support progressive playback, aamp by default uses souphttpsrc as the source element instead of appsrc which is used for adaptive streams.Aamp decides from the url extension that the stream is progressive.Once this is confirmed, aamp would set the url to uri property of playbin.Aamp would then take its natural course and manage the pipeline.

Pipeline Graph (used by default)

Image Removed

Aamp can however choose appsrc over souphttpsrc by toggling a runtime configuration switch termed as "useAppSrcForProgressivePlayback". This switch gets enabled when the keyword "appSrcForProgressivePlayback" is added in aamp.cfg of the box.

When "appSrcForProgressivePlayback" is enabled in aamp.cfg, aamp will choose appsrc as the source element and complete its pipeline.The pipeline graph then, will look like below.

Pipeline Graph

Image Removed

UVE playback on Intel STB

Playback is similar to Broadcom platform, but uses Intel ismd gstreamer plugins to decode/render. For DASH normal playback,  audio and video are re-stamped since start-time  of segment events are not properly supported in Intel platform.

Pipeline graph

Image Removed

HTML5 Video Tag based playback on Intel STB

Playback is similar to Broadcom platform, but uses Intel ismd gstreamer plugins to decode/render.

Pipeline graph

Image Removed

Gstreamer SOC element properties used by AAMP

AAMP Gstreamer player uses custom properties of SOC elements to achieve requirements.

Broadcom SOC element properties used by AAMP

...

Element

...

Property

...

Description

...

Intel gstreamer element properties used by AAMP

...

Element

...

Property

...

Description

...

Intel SOC (PaceXG1V1) specific changes

INTELCE macro is enabled for Intel SOC. Key differences from Broadcom based STB's are

...

.