Introduction 

CMF perform fully automated sanity testing of multiple video reference platforms on both Gerrit and Github contributions and on nightly branches.

Testing can also be performed on tip dunfell/rdk-next on demand.

CMF Sanity Tests are command line driven (e.g. gst-launch, aampcli) as well as UI driven. We have UI tests for the following UI's:

• RDK Video Accelerator UI (all platforms)
  • Accelerator Home UI - v3
• LGI Refapp2 (brcm97218 and RPI-3)
  • RDK reference image
• DTVKit DVB UI (brcm97218)
  • DTVKit

CMF Video Sanity Tests use the StormTest^TM product to do image comparison, video and audio detection, OCR etc..

CMF currently support the following platforms and profiles for contribution and nightly branch CI.

Automated Sanity Flows

Each sanity test flow follows the same principle:

• Look for a free DUT resource (Jenkins Lock)
• Retrieve the automated build image
• Deploy the image on the DUT
  • includes deploying any test related scripts 
• Verify the image has flashed successfully
  • check version.txt 
  • key service and process checks e.g. wpeframework, rmfstreamer etc
  • IP address has been assigned
• Retrieve the DUT logs pre test
  • includes a long set of commands CMF define
  • retrieve the actual device logs (/opt/logs)
• Perform the video command line and UI testing
  • Screenshots and other useful info is captured here
• Retrieve the DUT logs post test
• Prepare a test report (for contributions this is sent to Gerrit/Github and Verified/Platform-Verified status is set accordinly)
• Power down the device and free it for the next request

Aim is to keep each sanity test flow to <= 30 minutes.

All interaction with the DUT is done via serial and ssh.

Some examples of the automated flows:

CMF Sanity Tests

Tests are split between command line and UI based tests.

A global appState variable is used to determine if the app should be stopped/started or is running based on the test being run.

Command Line Tests

Video Accelerator UI Tests

• Note not all VA tests are run on contributions, key tests run are vaHomeScreenTest, vaCNNVideo1Test & vaVODVideoTest

LGI Refapp 2 UI Tests

DTVKit DVB UI Tests

Test Reporting

Each CMF Sanity Test Flow logs all the stages from image deployment, flashing, testing. This along with the DUT logs and video test reports are stored in Jenkins and in S3 buckets.

The video logs contain serial output from the device, logging from the test framework and screen grabs so you can actually see what was happening on the device during the test.

For example:

fullLog.txt:

Flash BCMREF
Flash BCM Reference r05s12 Sequence Start
  get BOLT> prompt [SUCCESS]
  boot initrd image [SUCCESS]
  flash bcm rootfs [SUCCESS]
  get BOLT> prompt [SUCCESS]
  boot bcm reference image [SUCCESS]
  Check Service rmfstreamer [SUCCESS]
  Check Service wpeframework [SUCCESS]
  Check Ip Addr 10.5.25.72 [SUCCESS]
Flash BCM Reference r05s12 Sequence [SUCCESS]
Play BCMREF
"Testing Sequence Started" 
  HLS Video Streaming using aamp-cli [SUCCESS]
  DASH Video Streaming using aamp-cli [SUCCESS]
  PLAYREADY_DRM Video Streaming using aamp-cli [SUCCESS]
  Video Accelerator Home Screen Test [SUCCESS]
  Video Accelerator CNN App Video Detection Test [SUCCESS]
  Video Accelerator VOD Video Detection Test [SUCCESS]
  AAMP CI Tests [SUCCESS]
	AAMP CI: WebApp launch passed
	AAMP CI: 1 tests passed
	AAMP CI: 0 tests failed
		AAMP CI: HTML Web App Tests Passed
Testing Sequence [SUCCESS]

gerrit/github message:

example snippet from the StormTest video log for the CNN APP Video Detection Test: