@startuml

autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant GStreamer_client_appsrc
participant decrypter_element
participant ocdmProxy
end box


Cobalt       --> DPI:              status
DPI  ->  Starboard:            --> Netflix:       SbPlayerWriteSample2(player, sample[])
note right: Currently  status

@enduml

@startuml

autonumber

box "Container" #LightGreen
participant Netflix
participant DPI
participant rialtoClient
end box

rialtoClient      -/  DPI:              opencdm_gstreamer_session_decrypt_buffer(key_session, buffer, caps)
end

ocdmProxynotifyNeedMediaData(pipeline_session, sourceId, frame_count, need_data_request_id, shm_info)
DPI               --> rialtoClient:

opt Cached segment stored from previous  ->  decrypter_element:need data request
DPI               ->  rialtoClient:     gstaddSegment(need_bufferdata_add_protection_meta(buffer, metadata)
note left
Decryption is deferred until the data is sent to Rialto so
attach the required decryption parameters to the media frame
which are then ready to be passed to Rialto when it requests
more data.
end note

@enduml

@startuml

autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant rialtoClient
end box

== Initialisation ==

Cobalt            -> OK
else Errror
rialtoClient Starboard:       --> DPI: ERROR
rialtoClient SbPlayerGetMaximumNumberOfSamplesPerWrite(player, sample_type)
Starboard      -/  DPI: --> Cobalt:            notifyPlaybackState(PLAYBACK_STATE_FAILURE)
end

opt First video frame at start of playback or after seek ready for rendering
opt notifyFrameReady not currently implementedmax_samples=1
note right: Specify that Cobalt only send 1 sample at a time


== Write samples ==

rialtoClient      -/  DPIStarboard:        notifyNeedMediaData(pipeline_session, sourceId, frame_count,    notifyFrameReady(time_position)
else
rialtoClientneed_data_request_id, shm_info)
Starboard         --/>  DPI:              notifyPlaybackState(PLAYBACK_STATE_PAUSED)
end
DPIrialtoClient:

opt Cached segment stored from previous need data request
Starboard         ->  rialtoClient:     -/  NetflixaddSegment(need_data_request_id, cached_media_segment)
rialtoClient      --> Starboard:        status
note  readyToRenderFrame(pts=time_position)
end

@enduml

@startuml

autonumber

box "Container" #LightGreen
participant GStreamer_client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
end box

Note across
Whenever the audio or video region of the shared memory buffer is empty a refill should be initiated (unless the source is at EOS). The buffer may be empty for various reasons,
such as new attachSource occurred, buffered frames all consumed or flush due to seek.
end note

rialtoServer      -/  rialtoClient:     notifyNeedMediaData(pipeline_session, sourceId, frameCount, needDataRequestId, shmInfo)
note left
shmInfo specifies the size and offset of the region to
populate with the media data in the shm buffer. This
data is used by the Rialto client to write the shm buffer.
end note
rialtoClient      ->  rialtoClient:     Store shmInfo & frameCount with request ID
rialtoClient      --> rialtoServer:
rialtoClient      -/  GStreamer_client: notifyNeedMediaData(pipeline_session, sourceId, frameCount,\n\t\t\t\t\tneedDataRequestId, shmInfo)
note right
The shmInfo should be set to invalid in this event as it is not used by the
application. It is only present to allow a common interface on client and server.
end note
GStreamer_client  --> rialtoClient:

loop While (framesFetched < frameCount) && (addSegment() returns OK)
GStreamer_client  ->  GStreamer_client: Pull frame from pipeline (use cached frame first if available)
note right: See below "Cache excess frame"
GStreamer_client  ->  GStreamer_client: Get any encryption metadata from media frame buffer
GStreamer_client  ->  GStreamer_client: Create media segment from sample data, sample\nmetadata and, if relevant, decryption metadata
GStreamer_client  ->

@startuml

autonumber

box "Container" #LightGreen
participant GStreamer_client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
end box

Note across
Whenever the audio or video region of the shared memory buffer is empty a refill should be initiated (unless the source is at EOS). The buffer may be empty for various reasons,
such as new attachSource occurred, buffered frames all consumed or flush due to seek.
end note

rialtoServer      -/  rialtoClient:     notifyNeedMediaData(pipeline_session, sourceId, frameCount, addSegment(needDataRequestId, shmInfosegment)
noteopt left
shmInfoneedDataRequestId specifiesis thevalid size&& andenough offset of the region to
populate with the media data in the shm buffer. This
data is used by the Rialto client to write the shm buffer.
end notespace to write segment and its metadata to shm region && client not trying to send too many frames
rialtoClient      ->  rialtoClient:     StoreCopy shmInfosegment & frameCount with metadata to shm buffer based on previously stored shmInfo for this request ID
rialtoClient      --> rialtoServerGStreamer_client: OK
rialtoClientelse Not enough space in shm -/region || GStreamer_client: notifyNeedMediaData(pipeline_session, sourceId, frameCount,\n\t\t\t\t\tneedDataRequestId, shmInfo)
note right
The shmInfo should be set to invalid in this event as it is not used by the
application. It is only present to allow a common interface on client and server.
end note
GStreamer_client  --> rialtoClient:

loop While (framesFetched < frameCount) && (addSegment() returns OK) trying to send too many frames
rialtoClient      --> GStreamer_client: NO_SPACE
else needDataRequestId not found
rialtoClient      --> GStreamer_client: OK
note right: Silently ignore calls with invalid request ID as this is possible due to race conditions
end
end
note right
If not enough frames are available do not wait but
return whatever data is available immediately.
end note

opt Last call to addSegment() returned NO_SPACE
GStreamer_client  ->  GStreamer_client: PullCache frame from pipeline (use cached frame first if available)excess frame
end
note right:
In Seeprevious belowloop "Cacheit excess frame"
GStreamer_client  ->  GStreamer_client: Get any encryption metadata from media frame bufferis possible that the final frame pulled from the pipeline would not
fit within the shm region. In this case that frame should be cached and sent when the
next notifyNeedMediaData() request arrives for the source.
end note

opt No frames available AND not EOS not reached
GStreamer_client  ->  GStreamer_client: Createstatus media segment from sample data, sample\nmetadata and, if relevant, decryption metadata= NO_AVAILABLE_SAMPLES
else EOS reached (with or without any frames available)
GStreamer_client  ->  rialtoClientGStreamer_client: status = EOS
else Samples addSegment(needDataRequestIdavailable, segment)
opt needDataRequestId is valid && enough space to write segment and its metadata to shm region && client not trying to send too many frames
rialtoClient    not EOS
GStreamer_client  ->  GStreamer_client: status = OK
else
GStreamer_client  ->  GStreamer_client: status = ERROR
end

GStreamer_client  ->  rialtoClient:     Copy segment & metadata to shm buffer based on previously stored shmInfo for thishaveData(pipeline_session, status, needDataRequestId)

opt needDataRequestId is valid - i.e. it matches the last sent notifyNeedMediaData request ID
rialtoClient for an attached source

rialtoClient      -->  GStreamer_clientrialtoServer: OK
else Not enough space in shm region || client trying to send too many frames
rialtoClient      --> GStreamer_client: NO_SPACE
else needDataRequestId not found
rialtoClient haveData(pipeline_session, status, needDataRequestId)

opt version in metadata is supported by Rialto Server && needDataRequestId is valid

opt At least one frame available
rialtoServer      -->  GStreamer_clientrialtoServer: OK
note right: Silently ignore callsTrigger withalgorithm invalidto requestpush IDdata asto this is possible due to race conditions
end
end
note right
If not enough frames are available do not wait but
return whateverGstreamer
note left: This must not block, it notifies the worker thread that new data is available immediately.
end note

opt Last call to addSegment() returned NO_SPACE
GStreamer_client  
else
rialtoServer      ->  GStreamer_clientrialtoServer: Cache excess frame
end
note right
In previousSet looptimer itto issend possiblenew thatneedData the final frame pulled from the pipeline would not
fit within the shm region. In this case that frame should be cached and sent when the
next notifyNeedMediaData() request arrives for the source.
end note

opt No frames available AND not EOS not reached
GStreamer_client  ->request
note left: A timer is used to prevent a needData/haveData message\nstorm when client has no samples to send
end

opt Frames pushed for all attached sources && buffered not sent
rialtoServer      -/  rialtoClient:     notifyNetworkState(NETWORK_STATE_BUFFERED)
rialtoClient      -/  GStreamer_client: status = NO_AVAILABLE_SAMPLES
else EOS reached (with or without any frames available)
GStreamer_client  ->  GStreamer_client: status = EOS
else Samples available, not EOS
GStreamer_client  -> notifyNetworkState(NETWORK_STATE_BUFFERED)
end

rialtoServer      --> rialtoClient:     OK
rialtoClient      --> GStreamer_client: status = OK
else
GStreamer_client metadata ->version  GStreamer_client: status = ERROR
end

GStreamer_client  -> unsupported
rialtoServer      --> rialtoClient:     haveData(pipeline_session, status, needDataRequestId)

opt needDataRequestId is valid - i.e. it matches the last sent notifyNeedMediaData request ID for an attached source

ERROR
rialtoClient      --> GStreamer_client: ERROR
rialtoServer      -/  rialtoClient:     notifyPlaybackState(PLAYBACK_STATE_FAILURE)
rialtoClient      ->/  rialtoServerGStreamer_client:     haveData(pipeline_session, status, needDataRequestId)

opt version in metadata is supported by Rialto Server && needDataRequestId is valid

opt At least one frame available
rialtoServer      ->  rialtoServer:     Trigger algorithm to push data to Gstreamer
note left: This must not block, it notifies the worker thread that new data is available
else
rialtoServernotifyPlaybackState(PLAYBACK_STATE_FAILURE)
else needDataRequestId not valid
rialtoServer      --> rialtoClient:     OK
note right
There are various race conditions, especially when seeking,
that can cause the request ID to not match the cached
value. This should be logged as a warning for
troubleshooting purposes but not treated as an error.
end note
rialtoClient      -->  rialtoServer:     Set timer to send new needData request
note left: A timer is used to prevent a needData/haveData message\nstorm when client has no samples to send
end

opt Frames pushed for all attached sources && buffered not sent
rialtoServerGStreamer_client: OK
end

else needDataRequestId is not valid
rialtoClient      -/  rialtoClient-> GStreamer_client: OK
note right
See note  notifyNetworkState(NETWORK_STATE_BUFFERED)
rialtoClient      -/  GStreamer_client: notifyNetworkState(NETWORK_STATE_BUFFERED)
endabove
end note
end

@enduml

@startuml

autonumber

box "Platform" #LightBlue
participant client
participant rialtoServer
end box

rialtoServer      -->/  rialtoClientclient:     OK
rialtoClient      --> GStreamer_client: OK
else metadata version unsupported
rialtoServernotifyNeedMediaData(pipeline_session, sourceId, frameCount, maxBytes, needDataRequestId, shmInfo)
client            -->  rialtoClientclient:     ERROR
rialtoClient      --> GStreamer_client: ERROR
rialtoServerIgnore shmInfo
client       -/  rialtoClient:   --> rialtoServer:

loop While notifyPlaybackState(PLAYBACK_STATE_FAILURE)
rialtoClient      -/  GStreamer_client: notifyPlaybackState(PLAYBACK_STATE_FAILURE)
else needDataRequestId not valid
rialtoServer(framesFetched < frameCount) && (addSegment() returns OK)
client            -->  rialtoClientclient:     OK
note right
There are various race conditions, especially when seeking,
that can cause the request ID to not match the cached
value. This should be logged as a warning for
troubleshooting purposes but not treated as an error.
end note
rialtoClient      --> GStreamer_client: OK
end

else needDataRequestId is not valid
rialtoClient   Get next frame & any decryption metadata
client            ->  rialtoServer:     addSegment(needDataRequestId, segment)

opt needDataRequestId is valid && enough space to write segment and its metadata to shm region && client not trying to send too many frames
rialtoServer      -->  GStreamer_clientrialtoServer: OK
note right
See note above
end note
end

@enduml

The code for populating the shm buffer from the parameters to addSegment() will be common on the client & server side so this should be stored in a common location to be used by both implementations.

@startuml

autonumber

box "Platform" #LightGreen
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
participant Ocdm
end box

note across
Gstreamer app source uses 2 signals, need-data and enough-data,
to notify its client whether it needs more data. Rialto server
should only push data when the appsrc indicates that it is in
the need-data state.
end note


loop While appsrc needs data && appsrc data available in shm buffer

rialtoServer       ->  rialtoServer:      Extract frame's metadata from shm
opt Frame encrypted

rialtoServer	   ->  GStreamer_server:  gst_buffer_add_protection_meta(buffer, meta)

end

rialtoServer       ->  GStreamer_server:  Set width/height caps
opt new codec_data in frame
rialtoServer       ->  GStreamer_server:  Set codec_data caps
end

rialtoServer       ->  GStreamer_server:  gst_app_src_push_buffer(src, gst_buffer)
rialtoServer       ->  rialtoServer:      'Remove' frame from shm

opt First video frame pushed at start of playback / after seek
note across: Not currently implemented

@startuml

autonumber

box "Platform" #LightGreen
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
participant Ocdm
end box

note across
Gstreamer app source uses 2 signals, need-data and enough-data,
to notify its client whether it needs more data. Rialto server
should only push data when the appsrc indicates that it is in
the need-data state.
end note


loop While appsrc needs data && appsrc data available in shm buffer

rialtoServer       ->  rialtoServer:      Extract frame's metadata from shm
opt Frame encrypted

rialtoServer	--/ rialtoClient:      notifyFrameReady(frame_timestamp)
end

opt Appsrc data exhausted from shm
opt (status == EOS) for this appsrc
rialtoServer       ->   GStreamerGStreamer_server:    gst_buffer_add_protection_meta(buffer, meta)

end
  notify EOS
else Not EOS
rialtoServer       ->  GStreamer_server:  Set width/height caps
opt new codec_data in frame
rialtoServer    --/ rialtoClient:      notifyNeedMediaData(...)
end
end

end

@enduml

@startuml

autonumber

box "Platform" #LightGreen
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
participant Ocdm
end box

GStreamer_server   ->  GStreamer_serverrialtoServer:    Set codec_data caps
end decrypt(buffer)

rialtoServer  	     ->  GStreamer GStreamer_server:  gst_appbuffer_srcget_pushprotection_meta(buffer(src, gst_buffer)
)
 
opt Protection Meta exists (Frame encrypted)
 
rialtoServer       ->  rialtoServer:      'Remove'Extract frame's from shmmetadata

opt First video frame pushed at start of playback / after seek
note across: Not currently implemented
rialtoServermedia_keys.key_system == "com.netflix.playready"
rialtoServer       ->  Ocdm:       --/ rialtoClient:      notifyFrameReady(frame_timestampopencdm_select_key_id(ocdm_session, kid)
end


opt AppsrcImplementation databefore exhaustedCBCS fromsupport shm
opt (status == EOS) for this appsrc
rialtoServeradded
rialtoServer       ->  Ocdm:              ->  GStreamer_server:      notify EOS
else Not EOSopencdm_gstreamer_session_decrypt_ex(ocdm_session, gst_buffer, subsample_info, iv, key, init_with_last_15, caps)
else Implementation after CBCS support added
rialtoServer       --/>  rialtoClientOcdm:            notifyNeedMediaData(...)
end   opencdm_gstreamer_session_decrypt_buffer(ocdm_session, gst_buffer, caps)
end

end

@enduml

@startuml

autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant GStreamer_client: notifyPosition(pipeline_session, position)
note over GStreamer_client: Not used by Cobalt as some conformance\ntests require very high position accuracy


== Get position ==

Cobalt    
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

opt End of content reached
GStreamer_server   -/  rialtoServer:         GST_MESSAGE_EOS
rialtoServer       ->/  StarboardrialtoClient:         SbPlayerGetInfo2(player)
Starboard notifyPlaybackState(pipeline_session, END_OF_STREAM)
rialtoClient       ->/  GStreamer_client: Get position    notifyPlaybackState(pipeline_session, END_OF_STREAM)
note left
This should notify all attached sinks of EOS
end note
GStreamer_client   ->/  rialtoClientStarboard:     getPosition(pipeline_session)
rialtoClient     ->  rialtoServer:GST_MESSAGE_EOS
Starboard     getPosition(pipeline_session)
rialtoServer     ->/  GStreamer_serverCobalt: Get position from pipeline
GStreamer_server --> rialtoServer:     position
rialtoServer    PlayerStatus(player, kSbPlayerStateEndOfStream)
end

@enduml

@startuml

autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant GStreamer_client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box

== Initialisation - register for callbacks ==

opt Video source attached
rialtoServer --> rialtoClient:     position
rialtoClient     --> GStreamer_client: position
GStreamer_client --> Starboard:        position
Starboard        ->  StarboardGStreamer_server:     g_signal_connect(video_decoder, getVideoUnderflowSignalName_soc(),  Set player_info.pos = position
Starboard        video_underflow_cb, user_data);
GStreamer_server   --> CobaltrialtoServer:           player_info

@enduml

@startuml

autonumber

box "Container" #LightGreen
participant Application
participant rialtoGstreamer
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server 
end box

== Decryption ==
GStreamer_server   -->  rialtoServer:
end

opt Audio source removed
rialtoServer          decrypt(buffer)

rialtoServer	   ->  GStreamer_server:  MediaKeyErrorStatus::Fail
GStreamer_server   ->  GStreamer_server:     g_signal_handler_disconnect(audio_decoder, audio_handler_id);
GStreamer_server   --> rialtoServer:
end


== Underflow ==

opt Data starvation in server AV pipelineGST_BASE_TRANSFORM_FLOW_DROPPED 

note over GStreamer_server
Frame is dropped but playback is unaffected.
end note 

GStreamer_server   -/>  rialtoServer:         video_underflow_cb() or audio_underflow_cb()GST_MESSAGE_WARNING(src, GST_STREAM_ERROR_DECRYPT)

rialtoServer          -/  rialtoClient rialtoClient:         notifyBufferUnderflow(source_id notifyPlaybackError(MediaSourceType, PlaybackError::DECRYPTION)
rialtoClient       -/  GStreamer_client:     notifyBufferUnderflow(source_id)
GStreamer_client   -/  Starboard:            emit video_underflow_cb() or audio_underflow_cb()
 rialtoGstreamer:   notifyPlaybackError(MediaSourceType, PlaybackError::DECRYPTION)
 
note over Starboard, GStreamer_client
Starboard does not have any support for underflow
so the event can be ignored for this integrationrialtoGstreamer
Posting an error message on the sink make the\n
sink unable to continue playing back content.
end note

note across
There will be
rialtoGstreamer one re more pending-/ need Application: data  requests  at this point which if serviced will allow playback to resume
end note

end  GST_MESSAGE_ERROR(sink, GST_STREAM_ERROR_DECRYPT)

@enduml