...

The shared memory buffer shall be split into two regions for each playback session, one for a video stream, and the other for audio (only one concurrent audio track supported, for audio track selection previous audio must be removed first). Each region must be big enough to accommodate the largest possible frame of audio/video data plus associated decryption parameters and metadata. At the end there is also a separate section for webaudio regions (common for all playback sessions). The buffer shall initially be sized to 8Mb per playback session to allow some overhead + 10kB * number of webaudio regions. There can be 0 or more Web Audio regions per Rialto Client.

For apps that can support more than one concurrent playback the shared memory buffer shall be sized accordingly and partitioned into different logical areas for each playback session. The partitions need not necessarily be equally sized, for example if an app supports one UHD and one HD playback the 'HD' partition may be smaller. There can be 0 or more Web Audio regions per Rialto Client.

draw.io Diagram

border	true
diagramName	Shared memory partitioning for multiple playbacks
simpleViewer	false
width	links	auto
tbstyle	top
lbox	true
diagramWidth	1261
revision	2

...

draw.io Diagram

width

border	true
diagramName	Shared memory buffer layout for AV stream
simpleViewer	false
links	auto
tbstyle	top
lbox	true
diagramWidth	791
revision	1

...

draw.io Diagram

border	true
diagramName	Metadata format v1
simpleViewer	false
width	links	auto
tbstyle	top
lbox	true
diagramWidth	431
revision	1

...

draw.io Diagram

border	true

diagramName	Shared memory buffer layout v2
simpleViewer	false
width
links	auto
tbstyle	top
lbox	true
diagramWidth	1462
revision	13

V2 metadata uses protobuf to serialise the frames' properties to the shared memory buffer. This use of protobuf aligns with the IPC protocol but also allows support for optional fields and for fields to be added and removed without causing backward/forward compatibility issues. It also supports variable length fields so the MKS ID, IV & sub-sample information can all be directly encoded in the metadata, avoiding the complexities of interleaving them with the media frames and referencing them with offsets/lengths as used in the V1 metadata format.

enum SegmentAlignment {

  ALIGNMENT_UNDEFINED = 0;
  ALIGNMENT_NAL       = 1;
  ALIGNMENT_AU        = 2;
}

enum CipherMode {

  CIPHER_MODE_UNDEFINED = 0;
  CIPHER_MODE_CENC      = 1;  /* AES-CTR scheme */
  CIPHER_MODE_CBC1      = 2;  /* AES-CBC scheme */
  CIPHER_MODE_CENS      = 3;  /* AES-CTR subsample pattern encryption scheme */
  CIPHER_MODE_CBCS      = 4;
}message MediaSegmentMetadata {/* AES-CBC subsample pattern encryption scheme */
}


message MediaSegmentMetadata {

    optional     required uint32                 length               = 1;             /* Number of bytes in sample */

     required optional sint64                 time_position        = 2;             /* Position in stream in nanoseconds */

     required optional sint64                 sample_duration      = 3;             /* Frame/sample duration in nanoseconds */

     required optional uint32                 stream_id            = 4;             /* stream id (unique ID for ES, as defined in attachSource()) */

    optional uint32                 sample_rate          = 5;             /* Samples per second for audio segments */

    optional uint32                 channels_num         = 6;             /* Number of channels for audio segments */

    optional uint32                 width                = 7;             /* Frame width in pixels for video segments */

    optional uint32                 height               = 8;             /* Frame height in pixels for video segments */

    optional SegmentAlignment       segment_alignment    = 9;             /* Segment alignment can be specified for H264/H265, will use NAL if not set */

    optional bytes                  extra_data           = 10;            /* Buffer containing extradata */

    optional bytes                  media_key_session_id = 11;            /* Buffer containing key session ID to use for decryption */

    optional bytes                  key_id               = 12;            /* Buffer containing Key ID to use for decryption */

    optional bytes                  init_vector          = 13;            /* Buffer containing the initialization vector for decryption */

    optional uint32                 init_with_last_15    = 14;            /* initWithLast15 value for decryption */

    optional repeated SubsamplePair sub_sample_info      = 15;            /* If present, use gather/scatter decryption based on this list of clear/encrypted byte lengths. */

                                                                          /* If not present and content is encrypted then entire media segment needs decryption. (unless    */

      */    optional bytes                  codec_data           = 16;            /* Buffer containing updated codec data for video segments */    optional bytes       /* cipher_mode indicates pattern encryption in which case crypt/skip byte block cipher_modevalue specify    */

     = 17;            /* Block cipher mode of operation when common encryption used */    optional uint32                 crypt_byte_block     = 18;            /* Crypt byte block value for AES-CBC/cbcs* commonthe encryption */pattern)    optional uint32                 skip_byte_block      = 19;            /* Skip byte block value for AES-CBC/cbcs common encryption */

message SubsamplePair

{

    required uint32_t               */

  num_clear_bytes   optional bytes  = 1;             /* How manycodec_data of next bytes in sequence are clear */   = required16; uint32_t           /* Buffer containing updated num_encrypted_bytescodec data =for 2;video segments */

    optional CipherMode            /* cipher_mode How many of next bytes in sequence are encrypted */

}

Playback Control

Rialto interactions with Client & GStreamer

Start/Resume Playback

...

format	SVG
title	Start/resume playback

@startuml

autonumber

box "Container" #LightGreen
participant Client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

...

opt Gstreamer pipeline paused
rialtoServer -> GStreamer_server: Set pipeline state PLAYING
GStreamer_server --> rialtoServer: status
else opt Gstreamer pipeline playing
rialtoServer -> rialtoServer: status=OK
note right: Silently ignore
else
rialtoServer -> rialtoServer: status=ERROR
end

rialtoServer --> rialtoClient: status
rialtoClient --> Client: status

...

opt Pending playback rate change (see Set Playback Rate)
rialtoServer -> GStreamer_server: Set pipeline playback speed to pending rate
GStreamer_server --> rialtoServer: status
note left: No notification to client on failure
end

loop While pipeline not prerolled
rialtoServer -> GStreamer_server: check preroll status
GStreamer_server --> rialtoServer:
end
rialtoServer -/ rialtoClient: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PLAYING)
rialtoClient -/ Client: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PLAYING)
else opt Error starting playback
rialtoServer -/ rialtoClient: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
rialtoClient -/ Client: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
else No pipeline state change required
note over rialtoServer: No event required
end

@enduml

        = 17;            /* Block cipher mode of operation when common encryption used */

    optional uint32                 crypt_byte_block     = 18;            /* Crypt byte block value for CBCS cipher mode pattern */

    optional uint32                 skip_byte_block      = 19;            /* Skip byte block value for CBCS cipher mode pattern */
    optional Fraction               frame_rate           = 20;            /* Fractional frame rate of the video segments */

message SubsamplePair

{

    optional uint32_t               num_clear_bytes      = 1;             /* How many of next bytes in sequence are clear */

    optional uint32_t               num_encrypted_bytes  = 2;             /* How many of next bytes in sequence are encrypted */

}

Playback Control

Rialto interactions with Client & GStreamer

Start/Resume

...

Playback

PlantUML Render Macro

format	SVG
title	Pause Start/resume playback

@startuml

autonumber

box "Container" #LightGreen
participant Client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

Client -> rialtoClient: pauseplay(pipeline_session)
rialtoClient -> rialtoServer: pauseplay(pipeline_session)

opt Gstreamer pipeline in PLAYING statepaused
rialtoServer -> GStreamer_server: Set pipeline state PAUSEDPLAYING
GStreamer_server --> rialtoServer: status
else opt Gstreamer pipeline in PAUSED stateplaying
rialtoServer -> rialtoServer: status=OK
note right: Silently ignore
else
rialtoServer -> rialtoServer: status=ERROR
end
end

rialtoServer --> rialtoClient: status
rialtoClient --> Client: status

opt Pipeline state changed to PLAYING

opt Pending playback rate change (see Set Playback Rate)
rialtoServer -> GStreamer_server: Set pipeline playback speed to pending rate
GStreamer_server --> rialtoServer: status
note left: No notification to client on failure
end

loop While pipeline not prerolled
rialtoServer - -> rialtoClient: GStreamer_server: check preroll status
rialtoClient GStreamer_server --> Client: statusnote over Client, rialtoServer
Note that playback state notification may occur before pause() call returns
end noteopt Pipeline state changed to PAUSED
rialtoServer:
end
rialtoServer -/ rialtoClient: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PAUSEDPLAYING)
rialtoClient -/ Client: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PAUSEDPLAYING)
else opt Error pausing pipelinestarting playback
rialtoServer -/ rialtoClient: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
rialtoClient -/ Client: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
else No pipeline state change required
note over rialtoServer: No event required
end

@enduml

...

Pause Playback

PlantUML Render Macro

format	SVG
title	StopPause playback

@startuml

autonumber

box "Container" #LightGreen
participant Client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

Client Client -> rialtoClient: stoppause(pipeline_session)
rialtoClient -> rialtoServer: stoppause(pipeline_session))

opt Gstreamer pipeline in PLAYING state
rialtoServer -> GStreamer_server: Set pipeline state PAUSED
GStreamer_server --> rialtoServer: status
else opt Gstreamer pipeline in PAUSED state
rialtoServer -> rialtoServer: Stop requesting new samples from attached sourcesstatus=OK
note right: Silently ignore
else
rialtoServer -> GStreamer_server: Set pipeline state NULL
GStreamer_server --> rialtoServer rialtoServer: status=ERROR
end

rialtoServer --> rialtoClient: status
rialtoClient --> Client: status

note over Client, rialtoServer
Note that playback state notification may occur before stoppause() call returns
end note

opt Playback stopped successfullyPipeline state changed to PAUSED
rialtoServer -/ rialtoClient: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_STOPPEDPAUSED)
rialtoClient -/ Client: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_STOPPEDPAUSED)
else opt Error stopping playbackpausing pipeline
rialtoServer -/ rialtoClient: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
rialtoClient -/ Client: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
else No pipeline state change required
note over rialtoServer: No event required
end

@enduml

...

Stop

PlantUML Render Macro

format	SVG
title	Set Playback RateStop

@startuml

autonumber

box "Container" #LightGreen
participant Client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

Client Client -> rialtoClient: setPlaybackRatestop(pipeline_session, rate)
rialtoClient -> rialtoServer: setPlaybackRatestop(pipeline_session, rate)

opt rate != 0.0

)
rialtoServer -> rialtoServer: Stop requesting new samples from attached sourcesopt Pipeline in PLAYING state && seek not pending
rialtoServer -> GStreamer_server: Set pipeline playback speed to 'rate'
note right: See Starboard PlayerImpl::SetRate() for example implementation
state NULL
GStreamer_server --> rialtoServer: status

else Pipeline not in PLAYING state || seek pending

rialtoServer rialtoServer --> rialtoClient: status
rialtoClient --> rialtoServerClient: Store pending rate change
note right: When pipeline reaches playing state this rate must be applied

end

status

note over Client, rialtoServer
Note that playback state notification may occur before stop() call returns
end note

opt Playback stopped successfully
rialtoServer -/ rialtoClient: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_STOPPED)
rialtoClient -/ Client: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_STOPPED)
else Error stopping playbackelse rate == 0.0
rialtoServer --> rialtoServer: status=ERROR
note right: Client should call pause()
end
rialtoServer --> rialtoClient/ rialtoClient: statusnotifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
rialtoClient --> Client/ Client: statusnotifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
end

@enduml

Cobalt Integration

Play/Pause/Set speed

Set Playback Rate

PlantUML Render Macroplantuml

format	SVG
title	Cobalt play/pause/set speedSet Playback Rate

@startuml

autonumber

box

"Container"

#LightGreen
participant Client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

Client -> rialtoClient: setPlaybackRate(pipeline_session, rate)
rialtoClient -> rialtoServer: setPlaybackRate(pipeline_session, rate)

opt rate != 0.0

opt Pipeline in PLAYING state && seek not pending

rialtoServer -> GStreamer_server: Set pipeline playback speed to 'rate'
note right: See Starboard PlayerImpl::SetRate() for example implementation
GStreamer_server --> rialtoServer: status

else Pipeline not in PLAYING state || seek pending

rialtoServer --> rialtoServer: Store pending rate change
note right: When pipeline reaches playing state this rate must be applied

end

else rate == 0.0
rialtoServer --> rialtoServer: status=ERROR
note right: Client should call pause()
end

rialtoServer --> rialtoClient: status
rialtoClient --> Client: status

@enduml

Cobalt Integration

Play/Pause/Set speed

PlantUML Macro

format	SVG
title	Cobalt play/pause/set speed

@startuml

autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant GStreamer_client
participant rialtoClient
end box


Cobalt             ->  Starboard:        SbPlayerSetPlaybackRate(player, rate)
note over GStreamer_client
For now we assume max 1 pipeline session in
GStreamer_client and therefore store its
Rialto handle in a local variable. This
will need to be fixed to support dual
playback.
end note

opt rate == 0
Starboard  
participant Cobalt
participant Starboard
participant GStreamer_client
participant rialtoClient
end box


Cobalt             ->  Starboard:        SbPlayerSetPlaybackRate(player, rate)
note over GStreamer_client
For now we assume max 1 pipeline session in
GStreamer_client and therefore store its
Rialto handle in a local variable. This
will need to be fixed to support dual
playback.
end note

opt rate == 0
Starboard          ->  GStreamer_client: Set pipeline state PAUSED
GStreamer_client   ->  rialtoClient:     pause(pipeline_session)
else rate != 0
Starboard          ->  GStreamer_client: Set pipeline state PLAYING
GStreamer_client   ->  rialtoClient:     play(pipeline_session)
end
rialtoClient       --> GStreamer_client: status
GStreamer_client   --> Starboard:        status

Starboard          ->  GStreamer_client: Set pipeline playback rate
GStreamer_client   ->  rialtoClient:     setPlaybackRate(rate)
rialtoClient       --> GStreamer_client: status
GStreamer_client   --> Starboard:        status

Starboard          --> Cobalt:        ->   status



opt Pause->play successful
rialtoClientGStreamer_client: Set pipeline state PAUSED
GStreamer_client   ->  rialtoClient:  -/  GStreamer_client: notifyPlaybackStatepause(pipeline_session, PLAYBACK_STATE_PLAYING)
else Play->pause successful
rialtoClient rate != 0
Starboard          -/>  GStreamer_client: Set notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PAUSED)
else Play<->pause state change failed
rialtoClient    PLAYING
GStreamer_client   -/>  GStreamer_clientrialtoClient: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
end

@enduml

Render Frame (Video Peek)

Render frame may be called whilst playback is paused either at the start of playback or immediately after a seek operation. The client must wait for the readyToRenderFrame() callback first.

PlantUML Macro

format	SVG
title	Render Frame

@startuml

autonumber

box "Container" #LightGreen
participant Netflix
participant DPI
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

Netflix        play(pipeline_session)
end
rialtoClient       --> GStreamer_client: status
GStreamer_client   --> Starboard:        status

Starboard          ->  GStreamer_client: Set pipeline playback rate
GStreamer_client   ->  rialtoClient:     setPlaybackRate(rate)
rialtoClient       -->  DPIGStreamer_client: status
GStreamer_client   --> Starboard:         renderFrame()
DPIstatus

Starboard          --> Cobalt:      ->  rialtoClient:   status



opt  renderFrame()Pause->play successful
rialtoClient       ->/  rialtoServerGStreamer_client:     renderFrame()
opt Frame renderable
rialtoServernotifyPlaybackState(pipeline_session, PLAYBACK_STATE_PLAYING)
else Play->pause successful
rialtoClient       ->/  GStreamer_serverclient: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PAUSED)
else Play<->pause state change failed
rialtoClient       -/  GStreamer_client: notifyPlaybackState(pipeline_session, PLAYBACK_STATE_FAILURE)
end

@enduml

Render Frame (Video Peek)

Render frame may be called whilst playback is paused either at the start of playback or immediately after a seek operation. The client must wait for the readyToRenderFrame() callback first.

PlantUML Macro

format	SVG
title	Render Frame

@startuml

autonumber

box "Container" #LightGreen
participant Netflix
participant DPI
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

Netflix      Trigger rendering of frame

opt Frame rendered successfully

note across: It is a Netflix requirement to call updatePlaybackPosition() after renderFrame()
rialtoServer       --> rialtoClient:     notifyPosition(position)
rialtoClient       -->  DPI:              notifyPositionrenderFrame(position)
DPI                -->  NetflixrialtoClient:     renderFrame()
rialtoClient       ->  updatePlaybackPosition(pts)
rialtoServer:     renderFrame()
opt Frame renderable
rialtoServer       --> rialtoClient GStreamer_server: Trigger rendering   status=true
elseof frame
rialtoServer       --> rialtoClient:     status=falsetrue
end

else renderFrame() called in bad state
rialtoServer       --> rialtoClient:     status=false
end

rialtoClient       --> DPI:              status
DPI                --> Netflix:          status

@enduml

...

PlantUML Macro

format	SVG
title	Cobalt pushing media frames

@startuml

autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant GStreamer_client_appsrc
participant GStreamer_client_rialto_sink
participant ocdmProxy
end box


Cobalt

autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant GStreamer_client_appsrc
participant decrypter_element
participant ocdmProxy
end box


Cobalt                        ->  Starboard:                    SbPlayerWriteSample2(player, sample[])
note right: Currently sample array size must be 1
Starboard                     ->  Starboard:                    Create GstBuffer and add media data from sample  ->  Starboard:to it

opt Sample encrypted
Starboard                     SbPlayerWriteSample2(player, sample[])
note right: Currently sample array size must be 1->  GStreamer_client_appsrc:      gst_buffer_add_protection_meta(gst_buffer, decrytion_params)
end


Starboard                     ->  StarboardGStreamer_client_appsrc:      gst_app_src_push_buffer(app_src, gst_buffer)
GStreamer_client_appsrc             Create GstBuffer and add media data from sample to it

opt Sample encrypted
Starboard    --> decrypter_element:            data flows through client pipeline
decrypter_element             ->  decrypter_element:    ->  GStreamer_client_appsrc:      gst_buffer_addget_protection_meta(gst_buffer, decrytion_params)
end


Starboard
opt Implementation before CBCS support added
decrypter_element             ->  ocdmProxy:            ->  GStreamer_client_appsrc:      gstopencdm_gstreamer_appsession_srcdecrypt_push_buffer(app_src, gst_buffer)
GStreamer_client_appsrcex(key_session, buffer, sub_samples, iv, kid, init_with_last_15, caps)
ocdmProxy        --> GStreamer_client_rialto_sink: data flows through client pipeline
GStreamer_client_rialto_sink  ->  GStreamer_client_rialto_sink: gst_buffer_get_protection_meta(buffer)
GStreamer_client_rialto_sink  ->  ocdmProxy:                    opencdm_gstreamer_session_decrypt(key_session, buffer,Create gst struct containing encryption data decrytion_params)
ocdmProxy  
else Implementation after CBCS support added
decrypter_element             ->  ocdmProxy:     ->  ocdmProxy:             opencdm_gstreamer_session_decrypt_buffer(key_session, buffer, caps)
end

ocdmProxy     Create gst struct containing encryption data decrytion_params
ocdmProxy          ->  decrypter_element:          ->  GStreamer_client_rialto_sink: gst_buffer_add_protection_meta(buffer, metadata)
note left
Decryption is deferred until the data is sent to Rialto so
attach the required decryption parameters to the media frame
which are then ready to be passed to Rialto when it requests
more data.
end note

@enduml

...

PlantUML Macro

format	SVG
title	Decrypt Frames on Gstreamer server pipeline

@startuml

autonumber

box "Platform" #LightGreen
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
participant Ocdm
end box

GStreamer_server   ->  rialtoServer:      decrypt(buffer)

rialtoServer	   ->  GStreamer_server:  gst_buffer_get_protection_meta(buffer)
 
opt Protection Meta exists (Frame encrypted)
 
rialtoServer       ->  rialtoServer:      Extract frame's metadata

opt media_keys.key_system == "com.netflix.playready"
rialtoServer       ->  Ocdm:              opencdm_select_key_id(ocdm_session, kid)
end


opt Implementation before CBCS support added
rialtoServer       ->  Ocdm:              opencdm_gstreamer_session_decrypt_ex(ocdm_session, gst_buffer, subsample_info, iv, key, init_with_last_15)
end iv, key, init_with_last_15, caps)
else Implementation after CBCS support added
rialtoServer       ->  Ocdm:              opencdm_gstreamer_session_decrypt_buffer(ocdm_session, gst_buffer, caps)
end

end

@enduml

Playback State

Position Reporting

...

PlantUML Macro

format	SVG
title	Underflow

@startuml

autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant GStreamer_client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

== Initialisation - register for callbacks ==

opt Video source attached
rialtoServer       ->  GStreamer_server:     g_signal_connect(video_decoder, getVideoUnderflowSignalName_soc(), video_underflow_cb, user_data);
GStreamer_server   --> rialtoServer:         video_handler_id
end

opt Audio source attached
rialtoServer       ->  GStreamer_server:     g_signal_connect(audio_decoder, getAudioUnderflowSignalName_soc(), audio_underflow_cb, user_data);
GStreamer_server   --> rialtoServer:         audio_handler_id
end


== Termination - unregister for callbacks ==

opt Video source removed
rialtoServer       ->  GStreamer_server:     g_signal_handler_disconnect(video_decoder, video_handler_id);
GStreamer_server   --> rialtoServer:
end

opt Audio source removed
rialtoServer       -> GStreamer_server:     g_signal_handler_disconnect(audio_decoder, audio_handler_id);
GStreamer_server   --> rialtoServer:
end


== Underflow ==

opt Data starvation in server AV pipeline
GStreamer_server   -/  rialtoServer:         video_underflow_cb() or audio_underflow_cb()
rialtoServer       ->  rialtoServer:         Set pipeline state to paused
rialtoServer       -/  rialtoClient:         notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PAUSED)
rialtoClient       -/  GStreamer_client:     notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PAUSED)
rialtoServer       -/  rialtoClient:         notifyNetworkState(pipeline_session, NETWORK_STATE_STALLED)
rialtoClient       -/  GStreamer_client:     notifyNetworkState(pipeline_session, NETWORK_STATE_STALLED)

note over Starboard, GStreamer_client
Starboard does not have any support for underflow
so the event can be ignored for this integration.
end note

note across
There will be one ore more pending need data requests at this point which if serviced will allow playback to resume
end note

end

== Recovery ==

opt rialtoServer detects that any need media data requests pending at point of underflow are now serviced and pushed to GStreamer_server || EOS signalled for any underflowed sources
note across
It is likely that underflow is due to one source becoming starved whilst data is buffered for other sources, so waiting until pending request(s) are serviced should allow playback to resume.
There are also some YT conformance tests that delay signalling EOS for an underflowed source whilst the other continues to stream hence the EOS condition to allow streaming to resume for the valid source.
end note

rialtoServer       ->  rialtoServer:         Set pipeline state to playing
rialtoServer       -/  rialtoClient:         notifyNetworkState(pipeline_session, NETWORK_STATE_BUFFERED
note across
underflow_enabled: Underflow is enabled when we're in playing state and source is attached.
underflow_cancelled: Underflow may be cancelled when haveData is called between notification from GStreamer and Underflow task handling.
end note
opt underflow_enabled && !underflow_cancelled
rialtoServer       -/  rialtoClient:         notifyBufferUnderflow(source_id)
rialtoClient       -/  GStreamer_client:     notifyBufferUnderflow(source_id)
GStreamer_client   -/  Starboard:            emit video_underflow_cb() or audio_underflow_cb()

note over Starboard, GStreamer_client
Starboard does not have any support for underflow
so the event can be ignored for this integration.
end note

end

note across
There will be one re more pending need data requests at this point which if serviced will allow playback to resume
end note

end

@enduml

Non-fatal Playback Failures

Decryption: Any encrypted frames that fail to decrypt are dropped, and an error notification is propagated to the rialto-gstreamer, at which point a decryption error is raised on the sink.

PlantUML Macro

format	SVG
title	Non-fatal Errors

@startuml

autonumber

box "Container" #LightGreen
participant Application
participant rialtoGstreamer
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server 
end box

== Decryption ==
GStreamer_server   ->  rialtoServer:      decrypt(buffer)

rialtoServer	   ->  GStreamer_server:  MediaKeyErrorStatus::Fail
GStreamer_server   ->  GStreamer_server:  GST_BASE_TRANSFORM_FLOW_DROPPED 

note over GStreamer_server
Frame is dropped but playback is unaffected.
end note 

GStreamer_server   ->  rialtoServer:      GST_MESSAGE_WARNING(src, GST_STREAM_ERROR_DECRYPT)

rialtoServer       -/  rialtoClient:      notifyPlaybackError(MediaSourceType, PlaybackError::DECRYPTION)
rialtoClient       -/  GStreamer_client rialtoGstreamer:     notifyNetworkState(pipeline_session, NETWORK_STATE_BUFFERED)
rialtoServer       -/  rialtoClient:         notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PLAYING)
rialtoClient       -/  GStreamer_client:     notifyPlaybackState(pipeline_session, PLAYBACK_STATE_PLAYING)

endnotifyPlaybackError(MediaSourceType, PlaybackError::DECRYPTION)
 
note over rialtoGstreamer
Posting an error message on the sink make the\n
sink unable to continue playing back content.
end note   
rialtoGstreamer    -/  Application:       GST_MESSAGE_ERROR(sink, GST_STREAM_ERROR_DECRYPT)

@enduml

RDK Resources

[RDK Preferred]

Code Management Facility

RDK Forums

[RDK Conferences]

RDK Support

Archives

Versions Compared

Old Version 13

New Version Current

Key

Playback Control

Rialto interactions with Client & GStreamer

Start/Resume Playback

Playback Control

Rialto interactions with Client & GStreamer

Start/Resume

Playback

Pause Playback

Stop

Cobalt Integration

Play/Pause/Set speed

Set Playback Rate

Cobalt Integration

Play/Pause/Set speed

Render Frame (Video Peek)

Render Frame (Video Peek)

Playback State

Position Reporting

Non-fatal Playback Failures

RDK Resources

[*RDK Preferred*]

Code Management Facility

RDK Forums

[RDK Conferences]

RDK Support

Archives

Page History

Versions Compared

Old Version 13

New Version Current

Key

Playback Control

Rialto interactions with Client & GStreamer

Start/Resume Playback

Playback Control

Rialto interactions with Client & GStreamer

Start/Resume

Playback

Pause Playback

Stop

Cobalt Integration

Play/Pause/Set speed

Set Playback Rate

Cobalt Integration

Play/Pause/Set speed

Render Frame (Video Peek)

Render Frame (Video Peek)

Playback State

Position Reporting

Non-fatal Playback Failures

[RDK Preferred]