Introduction

CMF performs automated sanity testing for RDKB  on RPI-3, RPI-4 and Turris GW platforms on both Gerrit and Github contributions and on nightly branches.

TDK testing is also performed on nightly CI branches for RPI-3.

Testing can also be performed on tip dunfell/rdk-next on demand.

Automated Sanity Flows

Example rdkb sanity flows for nightly branch testing, the same test flow jobs are used for contributions.


Sanity Tests

For RPI the following sanity tests are performed

  •  Image Validation
    • essential processes (e.g. ccsp) and services are running:
    • ccspwifiagent, hostapd for 2.4/5 Ghz
    • GW has erouter IP
  • Retrieve DUT Logs
    • includes a long set of commands CMF define
    • device logs are retrieved before and after running the sanity tests
    • serial console logs are also stored 

  • Wifi Access & Connectivity

    • Using Raspberry PI Ubuntu:

      • WiFi 2.4Ghz hotspot testing

      • WiFi 5Ghz hotspot testing

      • WiFi connection performed using wpa_supplicant 


  • Web UI Admin GUI
    • Login 
    • Menu Navigation
    • Parameter Modification
    • WiFi SSID/password change and re-connection verification
    • Change admin password
    • Selenium Webdriver used to navigate and verify WEB UI screenshots


For Turris GW:

  •  Image Validation
    • essential processes (e.g. ccsp) and services are running:
    • ccspwifiagent, hostapd for 2.4/5 Ghz
    • GW has erouter IP
  • Retrieve DUT Logs
    • includes a long set of commands CMF define
    • device logs are retrieved after validating the image
    • serial console logs are also stored 

Test Reporting

Each CMF Sanity Test Flow logs all the stages from image deployment, flashing, testing. This along with the DUT logs and test reports are stored in Jenkins and in S3 buckets.



example RPI FullfLow log:

###############################
RDKB RPI_R03S01
Deploy Sequence r03s01 Start
Deploy Sequence r03s01 [SUCCESS]
Flash RPI r03s01 Sequence Start
  flash SdCard [SUCCESS]
  get to sd prompt [FAILURE], remaining retries 3
  get to sd prompt [SUCCESS]
  Check Version nightly/20230113-dunfell [SUCCESS]
  Check Service ccspwifiagent [SUCCESS]
  Check Service hostapd [SUCCESS]
  Check Process hostapd0 [SUCCESS]
  Check Process hostapd1 [SUCCESS]
  Check Ip Addr 10.5.25.101 [FAILURE]
Flash RPI r03s01 Sequence [SUCCESS]
###############################


###############################
Ubuntu Wifi Client 2.4GHz
RPI r03s11 ready. no redeploy
RPI r03s11 ready. no flashing
Testing Sequence Started
  Create wpa.conf for SSID=r03s01_31147-AP0 [SUCCESS]
  connect wifi [SUCCESS]
  Acquire IP address [SUCCESS]
  WebBrowser sequence start
    setup virtual display [SUCCESS]
    get portal [SUCCESS]
    WebUI login [SUCCESS]
    browse and change wifi ssid/password (2.4G) [SUCCESS]
  WebBrowser sequence finished [SUCCESS]
  ping 10.0.0.1 failed as expected [SUCCESS]
  Reboot Wifi client [SUCCESS]
  Create wpa.conf for SSID=r03s01_31147_updt_2.4 [SUCCESS]
  connect wifi [SUCCESS]
  Acquire IP address [SUCCESS]
  WebBrowser sequence start
    setup virtual display [SUCCESS]
    get portal [SUCCESS]
    WebUI login [SUCCESS]
  WebBrowser sequence finished [SUCCESS]
Testing Sequence [SUCCESS]
###############################


###############################
Ubuntu Wifi Client 5GHz
RPI r03s11 ready. no redeploy
RPI r03s11 ready. no flashing
Testing Sequence Started
  Create wpa.conf for SSID=r03s01_31147-AP1 [SUCCESS]
  connect wifi [SUCCESS]
  Acquire IP address [SUCCESS]
  WebBrowser sequence start
    setup virtual display [SUCCESS]
    get portal [SUCCESS]
    WebUI login [SUCCESS]
    browse and change wifi ssid/password (5G) [SUCCESS]
  WebBrowser sequence finished [SUCCESS]
  ping 10.0.0.1 failed as expected [SUCCESS]
  Reboot Wifi client [SUCCESS]
  Create wpa.conf for SSID=r03s01_31147_updt_5 [SUCCESS]
  connect wifi [SUCCESS]
  Acquire IP address [SUCCESS]
  WebBrowser sequence start
    setup virtual display [SUCCESS]
    get portal [SUCCESS]
    WebUI login [SUCCESS]
  WebBrowser sequence finished [SUCCESS]
Testing Sequence [SUCCESS]
###############################


example Fullflow Turris log:

###############################
RDKB Turris TOM_R01S03
Boot Turris r01s03 OpenWRT Sequence Start
  Set Turris/OpenWrt mmcboot [SUCCESS]
  Get to rootfs prompt [SUCCESS]
  Check Version r16325-88151b8303 [SUCCESS]
  Check Ip Addr 192.168.21.103 [SUCCESS]
Boot Turris r01s03 OpenWRT Sequence [SUCCESS]
Deploy Sequence 192.168.21.103 Start
  Copy artifacts _to_ device 192.168.21.103 [SUCCESS]
  Deploy artifacts _on_ device 192.168.21.103 [SUCCESS]
Boot Turris r01s03 RDKB Sequence Start
  Set Turris/RDKB mmcboot [SUCCESS]
  Get to RDKB rootfs prompt [SUCCESS]
  Check Version nightly [SUCCESS]
  Check Service ccspwifiagent [SUCCESS]
  Check Service hostapd [SUCCESS]
  Check Ip Addr 192.168.21.103 [SUCCESS]
Boot Turris r01s03 RDKB Sequence [SUCCESS]
###############################


example sequence of WEB UI screenshots for RPI 2.4 and 5 Ghz testing is attached here:  example-rpi-webui-screenshot-sequence-selenium.tgz


CMFLAB Pictures


Some example pics of CMFLAB RDKB Racks:

  • No labels