Description

The MSE (Media Source Extensions) player sample app demos how to put everything together for a more real world example. The app will show how to build a gstreamer pipeline that can be fed raw H264 and AAC frames asynchronously. The sample content contains three separate fragments of the same video to show how to simulate a seek by flushing the video pipeline and providing new samples at different period in time. The app also uses the essos library which simplifies setting up wayland for graphics and keyboard/remote input. The following key features are demonstrated:

How to setup and use a more complicated gstreamer pipeline with a custom source
How to setup and register an object with rt
How to handle suspend/resume

Data Structures
struct	AVFrame

class	MediaSourcePipeline

Macros
#define	HANDLE_EINTR_EAGAIN(x)

Typedefs
typedef void(*	PipeSourceCallback) (void *ctx)

Enumerations
enum	ReadStatus

enum	PipelineType

enum	AVType

Functions
GSource *	pipe_source_new (int pipefd[2], PipeSourceCallback cb, void *ctx)
	This API creates a new event source "rtRemoteSource". More...

Variables
constexpr int	eintr_maximum_attempts =100

constexpr int	PIPE_LISTEN = 0

constexpr int	PIPE_WRITE = 1

Data Structure Documentation

◆ AVFrame

struct AVFrame

Definition at line 45 of file mediasourcepipeline.h.

Collaboration diagram for AVFrame:

Data Fields
guint8 *	data_
int32_t	size_
int64_t	timestamp_us_

◆ MediaSourcePipeline

class MediaSourcePipeline

Definition at line 51 of file mediasourcepipeline.h.

Inheritance diagram for MediaSourcePipeline:

Collaboration diagram for MediaSourcePipeline:

Public Member Functions
	rtDeclareObject (MediaSourcePipeline, rtObject)

	rtMethodNoArgAndNoReturn ("suspend", suspend)

	rtMethodNoArgAndNoReturn ("resume", resume)

	MediaSourcePipeline (std::string frame_files_path)

virtual bool	Start ()

virtual void	HandleKeyboardInput (unsigned int key)

rtError	suspend ()

rtError	resume ()

gboolean	HandleMessage (GstMessage *message)
	This API handles different message types. More...

void	StartFeedingAppSource (GstAppSrc *p_src)
	This signal callback is called when appsrc needs data. More...

void	StopFeedingAppSource (GstAppSrc *p_src)
	This signal callback is called when appsrc have enough-data. More...

void	SetNewAppSourceReadPosition (GstAppSrc *p_src, guint64 position)
	This API is to seek the app's position and set the position. More...

void	OnAutoPadAddedMediaSource (GstElement element, GstPad pad)
	This API links auto pad added elements in audio pipeline or video pipeline depending upon the element audio/video. More...

void	OnAutoElementAddedMediaSource (GstElement *element)
	This API checks for the dynamic addition of real audio sink. More...

gboolean	ReadVideoFrame ()
	This API reads data from both the audio/video timestamp file, raw frames file and push the details to appsrc. More...

gboolean	ReadAudioFrame ()
	This API reads the audio frame details and push the details to appsrc. More...

gboolean	StatusPoll ()
	This API queries the playback position. More...

gboolean	ChunkDemuxerSeek ()
	This API is used by the mse source to perform its own seek before starting the read data again. More...

void	sourceChanged ()
	This signal callback is called when "source-setup" is emitted.

Private Member Functions
bool	Build ()

void	Init ()

void	Destroy ()

void	StopAllTimeouts ()

void	CloseAllFiles ()

void	PerformSeek ()

ReadStatus	GetNextFrame (AVFrame *frame, AVType type)

bool	PushFrameToAppSrc (const AVFrame &frame, AVType type)

bool	ShouldBeReading (AVType av)

void	SetShouldBeReading (bool is_reading, AVType av)

void	CalculateCurrentEndTime ()

bool	ShouldPerformSeek ()

int64_t	GetCurrentStartTimeMicroseconds () const

bool	IsPlaybackOver ()

void	AddPlaybackPositionToHistory (int64_t position)

bool	IsPlaybackStalled ()

bool	HasPlaybackAdvanced ()

void	ResetPlaybackHistory ()

void	DoPause ()

void	finishPipelineLinkingAndStartPlaybackIfNeeded ()

Private Attributes
std::string	frame_files_path_

int32_t	current_file_counter_

FILE *	current_video_file_

FILE *	current_video_timestamp_file_

FILE *	current_audio_file_

FILE *	current_audio_timestamp_file_

bool	seeking_

GstElement *	pipeline_

GstAppSrc *	appsrc_source_video_

GstAppSrc *	appsrc_source_audio_

GstElement *	video_sink_

GstElement *	audio_sink_

std::vector< GstElement * >	ms_video_pipeline_

std::vector< GstElement * >	ms_audio_pipeline_

bool	should_be_reading_ [2]

float	playback_position_secs_

float	current_end_time_secs_

guint	video_frame_timeout_handle_

guint	audio_frame_timeout_handle_

guint	status_timeout_handle_

int32_t	current_playback_history_cnt_

std::vector< int64_t >	playback_position_history_

bool	playback_started_

bool	is_playing_

PipelineType	pipeline_type_

GstElement *	source_

GstCaps *	appsrc_caps_video_

GstCaps *	appsrc_caps_audio_

bool	pause_before_seek_

bool	is_active_

int64_t	seek_offset_

Member Function Documentation

◆ HandleMessage()

gboolean MediaSourcePipeline::HandleMessage ( GstMessage * message )

This API handles different message types.

Types can be errors, warnings, End of stream, state changed.

Parameters

[in] message Indicates the message type.

Returns: Returns true on success, appropriate error code otherwise.

Definition at line 136 of file mediasourcepipeline.cpp.

◆ StartFeedingAppSource()

void MediaSourcePipeline::StartFeedingAppSource ( GstAppSrc * p_src )

This signal callback is called when appsrc needs data.

Parameters

[in] p_src App source.

Definition at line 412 of file mediasourcepipeline.cpp.

◆ StopFeedingAppSource()

void MediaSourcePipeline::StopFeedingAppSource ( GstAppSrc * p_src )

This signal callback is called when appsrc have enough-data.

Parameters

[in] p_src App source.

Definition at line 438 of file mediasourcepipeline.cpp.

◆ SetNewAppSourceReadPosition()

void MediaSourcePipeline::SetNewAppSourceReadPosition	(	GstAppSrc *	p_src,
		guint64	position
	)

This API is to seek the app's position and set the position.

Parameters

[in]	p_src	App source.
[in]	position	Position to set.

Definition at line 454 of file mediasourcepipeline.cpp.

◆ OnAutoPadAddedMediaSource()

void MediaSourcePipeline::OnAutoPadAddedMediaSource	(	GstElement *	element,
		GstPad *	pad
	)

This API links auto pad added elements in audio pipeline or video pipeline depending upon the element audio/video.

Parameters

[in]	element	Source element
[in]	pad	Capabilities

Returns: Returns RT_OK on success, appropriate error code otherwise.

Definition at line 457 of file mediasourcepipeline.cpp.

◆ OnAutoElementAddedMediaSource()

void MediaSourcePipeline::OnAutoElementAddedMediaSource ( GstElement * element )

This API checks for the dynamic addition of real audio sink.

Parameters

[in] element Source element

Returns: Returns RT_OK on success, appropriate error code otherwise.

Definition at line 496 of file mediasourcepipeline.cpp.

◆ ReadVideoFrame()

gboolean MediaSourcePipeline::ReadVideoFrame ( )

This API reads data from both the audio/video timestamp file, raw frames file and push the details to appsrc.

if it fails reading,assume end of file is reached and need to peform a seek.

Returns: Returns TRUE on success, appropriate error code otherwise.

Definition at line 353 of file mediasourcepipeline.cpp.

◆ ReadAudioFrame()

gboolean MediaSourcePipeline::ReadAudioFrame ( )

This API reads the audio frame details and push the details to appsrc.

Returns: Returns TRUE on success, appropriate error code otherwise.

Definition at line 382 of file mediasourcepipeline.cpp.

◆ StatusPoll()

gboolean MediaSourcePipeline::StatusPoll ( )

This API queries the playback position.

Resets the file counter back to before beginning if it is complete.

Returns: Returns RT_OK on success, appropriate error code otherwise.

Definition at line 306 of file mediasourcepipeline.cpp.

◆ ChunkDemuxerSeek()

gboolean MediaSourcePipeline::ChunkDemuxerSeek ( )

This API is used by the mse source to perform its own seek before starting the read data again.

Returns: Returns False.

Definition at line 778 of file mediasourcepipeline.cpp.

Macro Definition Documentation

◆ HANDLE_EINTR_EAGAIN

#define HANDLE_EINTR_EAGAIN ( x )

Value:

  ({            \
  unsigned char _attempts = 0;               \
                                             \
  decltype(x) _result;                       \
                                             \
  do                                         \
  {                                          \
    _result = (x);                           \
  }                                          \
  while (_result == -1                       \
         && (errno == EINTR ||               \
             errno == EAGAIN ||              \
             errno == EWOULDBLOCK)           \
         && _attempts++ < eintr_maximum_attempts); \
                                             \
  _result;                                   \
})

Definition at line 31 of file glib_tools.h.

Function Documentation

◆ pipe_source_new()

GSource* pipe_source_new	(	int	pipefd[2],
		PipeSourceCallback	cb,
		void *	ctx
	)

This API creates a new event source "rtRemoteSource".

This API also adds a file descriptor polled for this source, sets whether a source can be called recursively.

Parameters

[in]	pipefd	Pipe File descriptor
[in]	cb	Callback Function
[in]	ctx	Context

Returns: Returns RT_OK on success, appropriate error code otherwise.

Definition at line 81 of file glib_tools.cpp.

Description

Data Structures

Macros

Typedefs

Enumerations

Functions

Variables

Data Structure Documentation

◆ AVFrame

◆ MediaSourcePipeline

Public Member Functions

Private Member Functions

Private Attributes

Member Function Documentation

◆ HandleMessage()

◆ StartFeedingAppSource()

◆ StopFeedingAppSource()

◆ SetNewAppSourceReadPosition()

◆ OnAutoPadAddedMediaSource()

◆ OnAutoElementAddedMediaSource()

◆ ReadVideoFrame()

◆ ReadAudioFrame()

◆ StatusPoll()

◆ ChunkDemuxerSeek()

Macro Definition Documentation

◆ HANDLE_EINTR_EAGAIN

Function Documentation

◆ pipe_source_new()