Versions Compared

Old Version 17

changes.mady.by.user James Cooley

Saved on

compared with

New Version Current

changes.mady.by.user James Cooley

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Introduction Introduction

CMF perform fully automated sanity testing of multiple video reference platforms on both Gerrit and Github contributions and on nightly branches.

...

CMF Sanity Tests are command line driven (e.g. gst-launch, aampcliaamp-cli) as well as UI driven. We have UI tests for the following UI's:

...

CMF currently support the following platforms and profiles for contribution and nightly branch CI.


PlatformProfilesTestingComment
RPI
3 Hybrid 

Video Accelerator UI

LGI Refapp2

Contribution

Nightly

Contribution

Nightly

RPI 3 MediaclientVideo Accelerator UI

Contribution

Nightly

RPI 3 IP Client
4 IP Client Mixed Mode ACKVideo Accelerator UINightly
RPI 4 Hybrid
32 Bit
Mixed Mode ACKVideo Accelerator UI

Nightly

Contribution


RPI 4 Mediaclient
32 Bit
Mixed Mode ACKVideo Accelerator UINightly
BRCM972180HB

Video Accelerator UI


LGI Refapp2

Contribution

Nightly

Contribution

Nightly

contribution builds for Video Accelerator profile are meta layers only
BRCM972180HBDTVKit DVB UI
Nightly

Currently Not Run:  

Jira
serverJIRA - 2
serverId11deff04-0380-3a3d-a916-0849d4e573f7
keyBCM72180-135

On Demand
SAGEMCOM (Broadcom)Video Accelerator UINightly
AMLOGIC AH212 IP ClientVideo Accelerator UI

Contribution

Nightly

contribution builds are meta layers only
AMLOGIC AT301 TV ProfileVideo Accelerator UI

Contribution

Nightly

contribution builds are meta layers only
SKYWORTH  (Amlogic)Video Accelerator UINightly
REALTEKVideo Accelerator UI
WIP

Contribution

Nightly

contribution builds are meta layers only
SERCOM (Blade Realtek)Video Accelerator UINightly

Automated Sanity Flows

Each sanity test flow follows the same principle:

  • Look for a free DUT resource (Jenkins Lock)
  • Retrieve the automated build image
  • Deploy the image on the DUT
    • includes deploying any test related scripts 
  • Verify the image has flashed successfully
    • check version.txt 
    • key service and process checks e.g. wpeframework, rmfstreamer etc
    • IP address has been assigned
  • Retrieve the DUT logs pre test
    • includes a long set of commands CMF define
    • retrieve the actual device logs (/opt/logs)
  • Perform the video command line and UI testing
    • Screenshots and other useful info is captured here
  • Retrieve the DUT logs post test
  • Prepare a test report (for contributions this is sent to Gerrit/Github and Verified/Platform-Verified status is set accordinlyaccordingly)
  • Power down the device and free it for the next request

...

Gallery
excludecnn-example-part-1.png, cnn-example-part-2.png, gerrit-messag-example.png, sample-flow-archive.png, lab-1.jpg, lab-2.jpg, lab-3.jpg, lab-4.jpg, lab-5.jpg, lab-6.jpg, lab-7.jpg, lab-8.jpg, lab-9.jpg
titleAutomated Flow Examples

...

TestShort DescriptionRPI 3 HybridRPI 3 Mediaclient

RPI 3
IPClient 

RPI 3
Refapp2		RPI 4
Hybrid 32 Bit		RPI 4
Mediaclient 32 Bit		BRCM
VA		SAGEMCOM
VA 		BRCM
Refapp2		AMLOGIC AH212AMLOGIC AT301SKYWORTH
VA		REALTEKSERCOM
VA
gst-play-1.0play a H264 or MPEG 2 local transport stream 













gst-launch uri=aamptest playback using gst-launch with aamp uri using HLS manifest













aamp-cli HLSHLS (.m3u8) video streaming using aamp-cli 













aamp-cli DASHDASH (.mpd) video streaming using aamp-cli 













aamp-cli PLAYREADY_DRMEncrypted Playready Playback with aamp-cli






WIP
WIPWIP


aamp-cli WIDEVINE_DRMEncrypted Widevine Playback with aamp-cli








WIPWIP


rmfApptest streaming from hybrid to client













rdkShellServicesCITest RDK Shell CI tests which test RDK Shell/Services API's













aampCI Tests Launches AAMP based unified player URL in a webkit instance and runs the embedded tests 













...

Video Accelerator UI Tests

TestShort DescriptionRPI 3 HybridRPI 3 Mediaclient

RPI 3
IPClient 

RPI 4
Hybrid 32 Bit		RPI 4
Mediaclient 32 Bit		BRCM
RefBoard VA		SAGEMCOM
VA		AMLOGIC AH212 RefBoard VAAMLOGIC AT301 RefBoard VASKYWORTH
VA		REALTEK
RefBoard VA		SERCOM
VA
vaHomeScreenTestcheck resident UI app has loaded
WIPWIP












vaCNNVideo1Testtest CNN app launch and video playback
WIPWIP












vaVODVideoTesttest VOD playback from resident UI
WIPWIP












vaVimeoVideo1Testtest Vimeo app launch and playback
WIPWIP












vaYoutubeVideoTesttest YouTube launch and playback
WIP
WIP












vaSettingsScreenCaptureTestcapture va settings screens
WIPWIP












vaEuroNewsVideoTesttest euronews launch and playback
WIPWIP












vaAccuWeatherTesttest Accuweather app launch and check Cork weather
WIPWIP












vaWSJVideoTesttest WSJ app launch and playback
WIPWIP












  • Note not all VA tests are run on contributions, key tests run are vaHomeScreenTest, vaCNNVideo1Test & vaVODVideoTest

...

The video logs contain serial output from the device, logging from the test framework and screen grabs so you can actually see what was happening on the device during the test.


For example jenkins sanity flow archive:


example fullLog.txt:

Code Block
Flash BCMREF
Flash BCM Reference r05s12 Sequence Start
  get BOLT> prompt [SUCCESS]
  boot initrd image [SUCCESS]
  flash bcm rootfs [SUCCESS]
  get BOLT> prompt [SUCCESS]
  boot bcm reference image [SUCCESS]
  Check Service rmfstreamer [SUCCESS]
  Check Service wpeframework [SUCCESS]
  Check Ip Addr 10.5.25.72 [SUCCESS]
Flash BCM Reference r05s12 Sequence [SUCCESS]
Play BCMREF
"Testing Sequence Started" 
  HLS Video Streaming using aamp-cli [SUCCESS]
  DASH Video Streaming using aamp-cli [SUCCESS]
  PLAYREADY_DRM Video Streaming using aamp-cli [SUCCESS]
  Video Accelerator Home Screen Test [SUCCESS]
  Video Accelerator CNN App Video Detection Test [SUCCESS]
  Video Accelerator VOD Video Detection Test [SUCCESS]
  AAMP CI Tests [SUCCESS]
	AAMP CI: WebApp launch passed
	AAMP CI: 1 tests passed
	AAMP CI: 0 tests failed
		AAMP CI: HTML Web App Tests Passed
Testing Sequence [SUCCESS]


gerrit/github message:


example snippet from the StormTest video log for the CNN APP Video Detection Test:

Sanity Test Framework Architecture

View file
nameCMF-Sanity-Framework-V2.pptx
height250
 


CMFLAB Pictures 

Gallery
includelab-1.jpg, lab-2.jpg, lab-3.jpg, lab-4.jpg, lab-5.jpg, lab-6.jpg, lab-7.jpg, lab-8.jpg, lab-9.jpg,
titleLab Pictures