Versions Compared

Old Version 14

changes.mady.by.user Philip Proudman

Saved on

compared with

New Version 15

changes.mady.by.user Philip Proudman

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

PlantUML Macro
formatSVG
titleQuery audio/video capabilities
@startuml

autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box


Cobalt            ->  Starboard:        SbIsMediaSupported(video_codec, audio_codec, key_system)
note right: key_system ignored


Starboard         ->  rialtoClient:     createMediaPipelineCapabilities()
rialtoClient      ->  rialtoServer:     createMediaPipelineCapabilities()
rialtoServer      ->  GStreamer_server: Get decoder capabilities
GStreamer_server  ->  rialtoServer:     Decoder caps
rialtoServer      ->  rialtoServer:     Convert caps to list of supported MIME\ntypes: 'supportedMimeTypes'
note right
Rialto has pre-defined list of its known MIME types.
The list of supported MIME types is the subset of
MIME types that are in the pre-defined  list AND
are supported by the underlying decoders.
end note
rialtoServer      ->  rialtoClient:     media_pipeline_caps
rialtoClient      ->  Starboard:        media_pipeline_caps

Starboard         ->  Starboard:        Convert video_codec to MIME type
Starboard         ->  rialtoClient:     media_pipeline_caps.isMimeTypeSupported(mimeType)
rialtoClient      ->  rialtoServer:     media_pipeline_caps.isMimeTypeSupported(mimeType)
rialtoServer      ->  rialtoServer:     Check mimeType against supportedMimeTypes
rialtoServer      ->  rialtoClient:     result
rialtoClient      ->  Starboard:        result

opt video_codec supported
Starboard         ->  Starboard:        Convert audio_codec to MIME type
Starboard         ->  rialtoClient:     media_pipeline_caps.isMimeTypeSupported(mimeType)
rialtoClient      ->  rialtoServer:     media_pipeline_caps.isMimeTypeSupported(mimeType)
rialtoServer      ->  rialtoServer:     Check mimeType against supportedMimeTypes
rialtoServer      ->  rialtoClient:     result
rialtoClient      ->  Starboard:        result

opt audio_codec supported
Starboard         --> Cobalt:           true
else
Starboard         --> Cobalt:           false
end

else
Starboard         ->  Cobalt:           false
end



== Get Supported Mime Types ==
note across: An alternative implementation could use the getsupporteMimeTypes() API

Starboard         ->  rialtoClient:     media_pipeline_caps.getSupportedMimeTypes(sourceType)
note right: sourceType indicates audio or video
rialtoClient      ->  rialtoServer:     media_pipeline_caps.getSupportedMimeTypes(sourceType)
rialtoServer      ->  rialtoServer:     Extract relevant MIME types from\nsupportedMimeTypes based on sourceType
rialtoServer      ->  rialtoClient:     mimeTypes[]
rialtoClient      ->  Starboard:        mimeTypes[]

@enduml

Output Control

Set Video Window Size & Position

API for setting the position and size of the video output window for a pipeline session.

Get supported properties

This is not something that a user of rialto-gstreamer needs to be concerned about (it is done automatically). This API is to query if a specified set of properties are supported on the RialtoServer. This enables RialtoGstreamer to determine the set of properties to present, and support, for the user of rialto-gstreamer. This will happen upon creation of a pipeline (when a sink is created).

PlantUML Macro
formatSVG
titleSet Video Output WindowGet Supported Properties
@startuml


autonumber

box "Container" #LightGreen
participant CobaltGStreamer_client
participant Starboard
participant GStreamer_clientrialtoGstreamer
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box


CobaltGStreamer_client  ->  rialtoGstreamer:  a sink is requested
rialtoGstreamer   ->  StarboardrialtoClient:         SetBounds(z, x, y, w, h)
Starboard     getSupportedProperties(pipeline_session, media_type, property_names)
rialtoClient      ->  GStreamer_clientrialtoServer: set  video sink geometry   getSupportedProperties(pipeline_session, xmedia_type, y, w, hproperty_names)
GStreamer_clientrialtoServer  ->  rialtoClient:  ->   setVideoWindow(pipeline_session, x, y, w, h)
rialtoClient    GStreamer_server: g_object_class_find_property(called once for each property name)
GStreamer_server  ->  rialtoServer:     property setVideoWindow(pipeline_session, x, y, w, h)(or null if not supported) for each property name
rialtoServer      ->  GStreamer_server: set video sink geometry (pipeline, x, y, w, h)
GStreamer_server  --> rialtoServer:     status
rialtoServer      --> rialtoClient:     status
rialtoClient      -->rialtoClient:     list of supported properties
rialtoClient      ->  rialtoGstreamer:  list of supported properties
rialtoGstreamer   ->  GStreamer_client: status
GStreamer_clienta sink --> Starboard:
Starboard         --> Cobalt:is created that supports the properties

@enduml

Quality of Service

The notifyQos() callback is used to notify clients of dropped audio/video data in the server side pipeline.


Output Control

Set Video Window Size & Position

API for setting the position and size of the video output window for a pipeline session.


PlantUML Macroplantumlrender
formatSVG
titleQOSSet Video Output Window
@startuml
autonumberbox 



autonumber

box "Container" #LightGreen


participant Cobalt


participant Starboard


participant GStreamer_client


participant rialtoClient


end box


box "Platform" #LightBlue


participant rialtoServer


participant GStreamer_server


end
box== During pipeline creation ==

rialtoServer      ->  GStreamer_server: Subscribe for QOS event

== During playback ==

opt Frames dropped or element changed processing strategy

note across

This event is generate whenever GStreamer generates a QoS event which is currently for these reasons:

 - The element dropped a buffer because of QoS reasons.

 - An element changed its processing strategy because of QoS reasons. This could include a decoder that

   decided to drop every B frame to increase its processing speed or an effect element switching to a lower

   quality algorithm.

end note

GStreamer_server  -/  rialtoServer:     GstBusMessage(qos_event)
rialtoServer      ->  rialtoServer:     Extract required data from event
rialtoServer      -/  rialtoClient:     notifyQos(pipeline_session, source_id, qos_info)
rialtoClient      -/  GStreamer_client: notifyQos(pipeline_session, source_id, qos_info)
GStreamer_client  ->  GStreamer_client: Create GST_EVENT_QOS message from qos_info
GStreamer_client  -/  Starboard:        GST_MESSAGE_QOS(qos_event)

note right
Only number of processed & dropped frames/
samples is sent by Rialto so the QOS event
sent by GStreamer_client will have less
valid properties set than the original server
side event.
end note

note over Cobalt, Starboard
QOS events are not propogated up to
Cobalt but can be queried anytime
by calling SbPlayerInfo2()
end note

end

@enduml

Get supported properties

This is not something that a user of rialto-gstreamer needs to be concerned about (it is done automatically). This API is to query if a specified set of properties are supported on the RialtoServer. This enables RialtoGstreamer to determine the set of properties to present, and support, for the user of rialto-gstreamer. This will happen upon creation of a pipeline (when a sink is created).

 box


Cobalt            ->  Starboard:        SetBounds(z, x, y, w, h)
Starboard         ->  GStreamer_client: set video sink geometry (pipeline, x, y, w, h)
GStreamer_client  ->  rialtoClient:     setVideoWindow(pipeline_session, x, y, w, h)
rialtoClient      ->  rialtoServer:     setVideoWindow(pipeline_session, x, y, w, h)
rialtoServer      ->  GStreamer_server: set video sink geometry (pipeline, x, y, w, h)
GStreamer_server  --> rialtoServer:     status
rialtoServer      --> rialtoClient:     status
rialtoClient      --> GStreamer_client: status
GStreamer_client  --> Starboard:
Starboard         --> Cobalt:

@enduml


Quality of Service

The notifyQos() callback is used to notify clients of dropped audio/video data in the server side pipeline.


PlantUML Render Macro
formatSVG
titleQOS

@startuml


autonumber

box "Container" #LightGreen
participant Cobalt
participant Starboard
participant GStreamer_client
participant rialtoClient
end box

box "Platform" #LightBlue
participant rialtoServer
participant GStreamer_server
end box


== During pipeline creation ==

rialtoServer      ->  GStreamer_server: Subscribe for QOS event


== During playback ==

opt Frames dropped or element changed processing strategy

note across

This event is generate whenever GStreamer generates a QoS event which is currently for these reasons:

 - The element dropped a buffer because of QoS reasons.

 - An element changed its processing strategy because of QoS reasons. This could include a decoder that

   decided to drop every B frame to increase its processing speed or an effect element switching to a lower

   quality algorithm.

end note


GStreamer_server  -/  rialtoServer:     GstBusMessage(qos_event)
rialtoServer      ->  rialtoServer:     Extract required data from event
rialtoServer      -/  rialtoClient:     notifyQos(pipeline_session, source_id, qos_info)
rialtoClient      -/  GStreamer_client: notifyQos(pipeline_session, source_id, qos_info)
GStreamer_client  ->  GStreamer_client: Create GST_EVENT_QOS message from qos_info
GStreamer_client  -/  Starboard:        GST_MESSAGE_QOS(qos_event)

note right
Only number of processed & dropped frames/
samples is sent by Rialto so the QOS event
sent by GStreamer_client will have less
valid properties set than the original server
side event.
end note

note over Cobalt, Starboard
QOS events are not propogated up to
Cobalt but can be queried anytime
by calling SbPlayerInfo2()
end note

end

PlantUML Macro
formatSVG
titleGet Supported Properties
@startuml autonumber box "Container" #LightGreen participant GStreamer_client participant rialtoGstreamer participant rialtoClient end box box "Platform" #LightBlue participant rialtoServer participant GStreamer_server end box GStreamer_client -> rialtoGstreamer: a sink is requested rialtoGstreamer -> rialtoClient:     getSupportedProperties(pipeline_session, media_type, property_names) rialtoClient -> rialtoServer:     getSupportedProperties(pipeline_session, media_type, property_names) rialtoServer -> GStreamer_server: g_object_class_find_property(called once for each property name) GStreamer_server -> rialtoServer: property (or null if not supported) for each property name rialtoServer -> rialtoClient: list of supported properties rialtoClient -> rialtoGstreamer: list of supported properties rialtoGstreamer -> GStreamer_client: a sink is created that supports the properties

@enduml

Volume

API for setting and getting the volume for a pipeline session.

...