Test Planning
CMFLAB Dashboard
CMFLAB Backlog
Template Tickets (these are cloned for each task)
Task | Template Ticket |
---|
TDK Intake | Jira |
---|
server | RDK |
---|
serverId | 11deff04-0380-3a3d-a916-0849d4e573f7 |
---|
key | CMFLAB-1125 |
---|
|
|
RDK-B Iteration/Release | Jira |
---|
server | RDK |
---|
serverId | 11deff04-0380-3a3d-a916-0849d4e573f7 |
---|
key | CMFLAB-552 |
---|
|
|
RDK-V Iteration/Release | Jira |
---|
server | RDK |
---|
serverId | 11deff04-0380-3a3d-a916-0849d4e573f7 |
---|
key | CMFLAB-365 |
---|
|
|
RDK-B CODEMGMT Iteration | Jira |
---|
server | RDK |
---|
serverId | 11deff04-0380-3a3d-a916-0849d4e573f7 |
---|
key | CODEMGMT-238 |
---|
|
|
RDK-B CODEMGMT Quarterly Release | Jira |
---|
server | RDK |
---|
serverId | 11deff04-0380-3a3d-a916-0849d4e573f7 |
---|
key | CODEMGMT-425 |
---|
|
|
RDK-V CODEMGMT Iteration | Jira |
---|
server | RDK |
---|
serverId | 11deff04-0380-3a3d-a916-0849d4e573f7 |
---|
key | CODEMGMT-180 |
---|
|
|
RDK-V CODEMGMT Quarterly Release | Jira |
---|
server | RDK |
---|
serverId | 11deff04-0380-3a3d-a916-0849d4e573f7 |
---|
key | CODEMGMT-554 |
---|
|
|
Sanity Test Frameworks
Sanity Test Descriptions
Sanity Test Build Trend
RefApp Documentation
Emulator and RPI Reference Platform Documentation
draw.io Diagram |
---|
border | true |
---|
viewerToolbar | true |
---|
| |
---|
fitWindow | false |
---|
diagramName | Sanity Test Framework |
---|
simpleViewer | false |
---|
width | |
---|
diagramWidth | 1055 |
---|
revision | 1 |
---|
|
Sanity Test Flows
TDK Test Frameworks
TDK Documentation and Releases
- Contains all the TDK Documentation you need from
- Setting up the TDK manager (note we have automated this in CMF via vagrant/puppet)
- TDK additional framework setup (e.g. aamp, spark, xconf, RDKB-E2E frameworks etc etc)
- TDK Video and Broadband Release Documentation
- Lots of other useful TDK information
Emulator and RPI Reference Platform Documentation
- Contains setup and build instructions for emulator and RPI reference platforms
CMF Test Framework Design
draw.io Diagram |
---|
border | true |
---|
viewerToolbar | true |
---|
| |
---|
fitWindow | false |
---|
diagramName | TDK Test Frameworks |
---|
simpleViewer | false |
---|
width | |
---|
diagramWidth | 1768 |
---|
revision | 4 |
---|
|
CMF TDK Testing is done using 3 frameworks (2 in active use):
All of which are kicked off with a set of input parameters to their various Jenkins jobs. Currently supporting 4 'RUNTYPE' options – 'release', 'nightly', 'nightly-full', and 'master'.
There is also a dry-rinBelow is each framework's instruction to run a flow job for the various RUNTYPE options and the differences those options make in regards to each phase of each flow.
CMF TDK Merged Reports Tool
- This produces a comparison report per platform on TDK test runs on nightly and release branches, this it the main tool we use to track TDK test results and identify regressions
- Report can also be run for rdk-dev or rdk-next only based nightly branches
- note to be added to the daily email this tool automatically generates add your email to:
TDK Merged Reports Tool
- This produces a comparison report per platform on a configurable number of TDK Reference Releases
- Useful on TDK Intake to see any change in test status or tests removed/added etc
TDK Test Metrics
Test and Git
CMF Gerrit
Comcast Gerrit
CMF Code Contribution Process
- Very useful doc to explain how to setup git and push changes for review to CMF Gerrit
CMF Test Git Repositories and Useful Info
Code Block |
---|
# clone the test repo
git clone "https://code.rdkcentral.com/r/cmf/test"
# gittiles link (must be logged into gerrit)
https://code.rdkcentral.com/r/plugins/gitiles/cmf/test
# TDK test results repo (soon to be deprecated, used to store TDK reference results)
git clone "https://code.rdkcentral.com/r/tata/test-results"
# clone the jenkins dsl test repo where all our test jenkins jobs are maintained
git clone "https://code.rdkcentral.com/r/cmf/jenkins-dsl/jobs/test"
# setup the commit hook if you want push changes for review in CMF Gerrit
cd <repo>
gitdir=$(git rev-parse --git-dir); curl -o ${gitdir}/hooks/commit-msg https://code.rdkcentral.com/r/tools/hooks/commit-msg ; chmod +x ${gitdir}/hooks/commit-msg
# useful git commands/workflow sequence if/when working on branches rather than pushing changes directly on master
git checkout master
git pull —rebase
git checkout -b <branch name>
… make changes
git add <files>
git commit -m "CMFLAB-XXX <commit msg>"
git push origin <branch name>
if single commit:
git checkout master
git cherry-pick <commit ID>
git push origin HEAD:refs/for/master
if multiple commits:
git rebase -i HEAD~<# of commits from HEAD>
… pick top commit; squash all others
git log (**check to see that squashed commit has Change-ID)
if no Change-ID in commit: git commit —amend; exit editor (Change-ID should now be there)
git push origin HEAD:refs/for/master |
Test AWS S3 Buckets
Test Development Builds
The test team maintain their own build scripts which we use to build and test debug images for defect investigations. These scripts are run on personal build slaves and can be used to build on any branch we support in CMF.
The scripts will upload the images to an s3 bucket and can also be used to automatically smoke test the image.
The scripts are maintained in the cmf test git repository https://code.rdkcentral.com/r/plugins/gitiles/cmf/test/+/master/cmf_testbed/tools/builds/
1) launch build slave via https://jenkins.cmf.code.rdkcentral.com/job/jenkins-reserve-slave
2) ssh to build slave (see jenkins console o/p for IP address)
3) set up build scripts on your build slave (if not already setup)
Code Block |
---|
cd ~/jenkinsroot/workspace
git clone "https://code.rdkcentral.com/r/cmf/test"
ln -s test/cmf_testbed/tools/builds/ builds |
4) build your image, e.g.
Code Block |
---|
|
cd ~/jenkinsroot/workspace/builds
./build_emub.sh <branch> <prepare|build>
Note follow the prompts to select what image(s) you want to build and whether you want to automatically smoke test the image |
- Notes
- if you want to apply patches or do some reverts or cherrypicks do the build in two stages
- run script with prepare first (this will do repo init and repo sync)
- make code changes
- then run script with build
- if you just want to build a branch then just run script with no prepare or build options
Debugging RDKB
Code Block |
---|
|
ifconfig
iwconfig
systemctl | grep -i ccsp
systemctl -l status <service>
journalctl -u <service>
ps -axf | grep hostapd <- check hotspot |
- If groups of tests failing, likely issue with one service
- Some services have dependancies so not unusual for a few of them to be in bad state
- Can take a few minutes for all services to start
Debugging RDKV
Code Block |
---|
|
systemctl
systemctl -l status <service>
journalctl -u <service>
ps -axf | grep rmfstreamer |
- If E2E/RMF tests failing check if rmfStreamer is running
TDK EMU-B VM
on griffen:
Code Block |
---|
|
get_vpc_key -i 192.168.32.74 -u <username> griffen
sudo su jenkins
# general info commands
vboxmanage list vms
vboxmanage list runningvms
vboxmanage showvminfo "rdkb-emu-tdk"
vboxmanage list usbhost
# restart vm
vboxmanage startvm "rdkb-emu-tdk" --type headless
# if above restart doesn't work power it off first
vboxmanage controlvm rdkb-emu-tdk poweroff
vboxmanage startvm "rdkb-emu-tdk" --type headless
# destroy VM - mechanism 1
vboxmanage controlvm rdkb-emu-tdk poweroff
vboxmanage unregistervm "rdkb-emu-tdk" --delete
# destroy VM - mechanism 2
vboxmanage startvm rdkb-emu-tdk --type emergencystop
vboxmanage unregistervm --delete rdkb-emu-tdk
# rdp session for RDK-B Emulator, run following ssh command then on RDP client connect to localhost:3388
ssh -o "PasswordAuthentication=no" -L 3388:192.168.32.74:3388 -i ~/.griffen-<username>.rsa <username>@192.168.32.74 |
Access EMU-V TDK VM's on browser
Code Block |
---|
|
# do get vps on heron first
get_vpc_key -i 192.168.32.76 -u <username> heron
# ssh and tunnel through local port for the TDK Manager your interested in
ssh -L 8085:192.168.21.31:8080 -i ~/.heron-jamescol.rsa jamescol@192.168.32.76
ssh -L 8086:192.168.21.32:8080 -i ~/.heron-jamescol.rsa jamescol@192.168.32.76
ssh -L 8087:192.168.21.33:8080 -i ~/.heron-jamescol.rsa jamescol@192.168.32.76
ssh -L 8088:192.168.21.34:8080 -i ~/.heron-jamescol.rsa jamescol@192.168.32.76
ssh -L 8089:192.168.21.35:8080 -i ~/.heron-jamescol.rsa jamescol@192.168.32.76
ssh -L 8090:192.168.21.36:8080 -i ~/.heron-jamescol.rsa jamescol@192.168.32.76
ssh -L 8091:192.168.21.38:8080 -i ~/.heron-jamescol.rsa jamescol@192.168.32.76
ssh -L 8092:192.168.21.39:8080 -i ~/.heron-jamescol.rsa jamescol@192.168.32.76
# now you can access any of the 8 TDK EMU-V TDK Managers in your browser via
http://localhost:<port>/rdk-test-tool/execution/create
# e.g.
ssh -L 8085:192.168.21.31:8080 -i ~/.heron-jamescol.rsa jamescol@192.168.32.76
http://localhost:8086/rdk-test-tool/execution/create |
TDK Manger Tips
To connect to TDK Manager VM's used in ATF testing: EMU-B TDK Manager VM, RPI-B TDK VM, RPI-V TDK VM you can use one of two methods either via vagrant ssh or ssh directly using it’s IP
( i ) via vagrant
Code Block |
---|
|
# first connect to griffen
get_vpc_key -i 192.168.32.74 -u <username> griffen
sudo su jenkins
# get list of VM's running and their status
vagrant global-status
# output will look like
vagrant global-status
id name provider state directory
------------------------------------------------------------------------------------------------------------
f6380e9 nfs-server-rdkb virtualbox running /home/jenkins/cmflab-vagrants/nfs-server/rdkb
0885fe0 tdk-emulator-rdkb virtualbox running /home/jenkins/cmflab-vagrants/rdkb/emulator/tdk
5e18def tdk-emulator-rdkb-rpi virtualbox running /home/jenkins/cmflab-vagrants/rdkb/rpi/tdk
f3fdd9a tdk-emulator-raven-atf-rpi-rdkv virtualbox running /home/jenkins/cmflab-vagrants/rdkv/rpi/tdk
81a242c tdk-emulator-rdkb-rpi-e2e virtualbox running /home/jenkins/cmflab-vagrants/rdkb/rpi/tdk-e2e
1b99642 rdkb-webserver virtualbox running /home/jenkins/cmflab-vagrants/rdkb/webserver
# run command `vagrant ssh <id>`
vagrant ssh 0885fe0
# some other vagrant commands
vagrant status <id>
vagrant halt <id>
vagrant up <id>
|
( ii ) via ssh
Code Block |
---|
|
# first connect to griffen or some other lab server machine (e.g. heron/corvus/raven)
get_vpc_key -i 192.168.32.74 -u <username> griffen
sudo su jenkins
# ssh to TDK VM using it's IP
# TDK Manager Emulator
ssh -o "StrictHostKeyChecking=no" vagrant@10.5.25.40
ssh -o "StrictHostKeyChecking=no" vagrant@192.168.32.63
http://192.168.32.63/rdk-test-tool/execution/create
# TDK Manager RPI-B
ssh -o "StrictHostKeyChecking=no" vagrant@10.5.25.41
ssh -o "StrictHostKeyChecking=no" vagrant@192.168.32.69
http://192.168.32.69/rdk-test-tool/execution/create
# TDK Manager RPI-V
ssh -o "StrictHostKeyChecking=no" vagrant@10.5.25.43
ssh -o "StrictHostKeyChecking=no" vagrant@192.168.32.70
http://192.168.32.70/rdk-test-tool/execution/create |
To connect to EMU-V VM's
Code Block |
---|
|
# first connect to heron
get_vpc_key -i 192.168.32.76 -u <uname> heron
sudo su jenkins
# get list of VM's running and their status
vagrant global-status
# output will list all emu-v VM's and the following TDK VM's
vagrant global-status
id name provider state directory
-------------------------------------------------------------------------------------------------------------------------------------------------
eeacce1 tdk-emulator-1 virtualbox running /home/jenkins/manager-vagrant
c4c70a1 tdk-emulator-2 virtualbox running /home/jenkins/manager-vagrant
148b8d3 tdk-emulator-3 virtualbox running /home/jenkins/manager-vagrant
09ef373 tdk-emulator-4 virtualbox running /home/jenkins/manager-vagrant
e778206 tdk-emulator-5 virtualbox running /home/jenkins/manager-vagrant
9633df6 tdk-emulator-6 virtualbox running /home/jenkins/manager-vagrant
9e17573 tdk-emulator-7 virtualbox running /home/jenkins/manager-vagrant
ae67a6f tdk-emulator-8 virtualbox running /home/jenkins/manager-vagrant
# run command `vagrant ssh <id>`
vagrant ssh eeacce1
# some other vagrant commands
vagrant status <id>
vagrant halt <id>
vagrant up <id>
# if vagrant-global status does not work (can be buggy) then cd into dir where vagrants are run vagrant status
cd /home/jenkins/manager-vagrant
vagrant status
Current machine states:
tdk-emulator-1 running (virtualbox)
tdk-emulator-2 running (virtualbox)
tdk-emulator-3 running (virtualbox)
tdk-emulator-4 running (virtualbox)
tdk-emulator-5 running (virtualbox)
tdk-emulator-6 running (virtualbox)
tdk-emulator-7 running (virtualbox)
tdk-emulator-8 running (virtualbox)
vagrant ssh tdk-emulator-1
# to ssh using IP , cat the VagrantFile to get the IP for the manager you want
cat /home/jenkins/manager-vagrant/Vagrantfile
tdk_managers = [
{:hostname => 'tdk-emulator-1', :ip => '192.168.21.31', :box => 'ubuntu/trusty64'},
{:hostname => 'tdk-emulator-2', :ip => '192.168.21.32', :box => 'ubuntu/trusty64'},
{:hostname => 'tdk-emulator-3', :ip => '192.168.21.33', :box => 'ubuntu/trusty64'},
{:hostname => 'tdk-emulator-4', :ip => '192.168.21.34', :box => 'ubuntu/trusty64'},
{:hostname => 'tdk-emulator-5', :ip => '192.168.21.35', :box => 'ubuntu/trusty64'},
{:hostname => 'tdk-emulator-6', :ip => '192.168.21.36', :box => 'ubuntu/trusty64'},
{:hostname => 'tdk-emulator-7', :ip => '192.168.21.38', :box => 'ubuntu/trusty64'},
{:hostname => 'tdk-emulator-8', :ip => '192.168.21.39', :box => 'ubuntu/trusty64'}
]
ssh -o "StrictHostKeyChecking=no" vagrant@192.168.21.31 <-- this is currently looking for password, need to check with ALAN
|
Some General TDK Manager VM Commands
Code Block |
---|
|
# check if tftp server is running on the TDK Manager VM
vagrant@tdk-emulator-rdkb:~$ ps -ef | grep tftp
vagrant 12660 12630 0 12:17 pts/0 00:00:00 grep --color=auto tftp
root 13444 13389 0 Sep18 ? 00:00:00 sudo python /var/lib/tomcat6/webapps/rdk-test-tool/fileStore/tftp_server.py 69 /var/lib/tomcat6/webapps/rdk-test-tool/logs/logs/
root 13446 13444 0 Sep18 ? 00:00:05 python /var/lib/tomcat6/webapps/rdk-test-tool/fileStore/tftp_server.py 69 /var/lib/tomcat6/webapps/rdk-test-tool/logs/logs/
# Manually start TDK TFTP Server on a TDK VM (tftp server is used on manager to transfer agent logs from DUT)
sudo python /var/lib/tomcat6/webapps/rdk-test-tool/fileStore/tftp_server.py 69 /var/lib/tomcat6/webapps/rdk-test-tool/logs/logs/ &
# kill tftp serrver
sudo pkill -f tftp
# to manually set the route on TDK Manager VM (this is needed if you can't connect to the webui of VM via your browser), e.g. can happen when VM was restarted abnormally
route del default
sudo route del default; sudo route add default gw 192.168.32.1 dev eth1
# to change interface or mechanism for agent log transfer (can be set to tftp/REST) edit tm.config file and restart tomcat
vi /var/lib/tomcat6/webapps/rdk-test-tool/fileStore/tm.config
sudo pkill -f tftp
sudo service tomcat6 stop
sudo service tomcat6 start
|
Sometimes it's useful to access the platform/DUT directly for defect investigation etc for emulators we can ssh to the device, for RPI's we can connect to the devices either by serial (via telnet on iolan) or by ssh'ing directly to the device.
For RPI Rack Setup and IP addresses, see
Code Block |
---|
|
# rdkb-emu TDK VM
get_vpc_key -i 192.168.32.74 -u <username> griffen
sudo su jenkins
ssh -o "StrictHostKeyChecking=no" root@10.5.25.100
# RPI via telnet
# telnet <iolan ip> <port>
telnet 10.5.25.5 10002
# use CTRL-] to quit
# tip if you accidently hit CTRL-C while in telnet session it will stop responding, quit and do following to recover
# echo 03 | xxd -r -p | nc <iolan ip> <port>
echo 03 | xxd -r -p | nc 10.5.25.5 10015
# RPI via ssh
# ssh -o "StrictHostKeyChecking=no" root@<IP>
# e.g. 2 RPI-B TDK
ssh -o "StrictHostKeyChecking=no" root@10.5.25.102
ssh -o "StrictHostKeyChecking=no" root@10.5.25.108
# RDKV-EMU TDK Hybrids and Clients
# these device IP's are not static so you either get it's IP from the console output of the
# https://jenkins.cmf.code.rdkcentral.com/view/TEST/job/test-emu-prepare-dut/ job are use vagrant to connect on heron
get_vpc_key -i 192.168.32.76 -u <username> heron
sudo su jenkins
vagrant global-status
id name provider state directory
-------------------------------------------------------------------------------------------------------------------------------------------------
7a9a0d5 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-hyb1
c8902cd default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-mc1
11caf80 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-hyb2
a0e30e0 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-mc2
bee0188 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-hyb3
e1fc8c3 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-mc3
318efaa default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-hyb4
c0ef362 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-mc4
1e66545 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-hyb5
2f90e49 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-mc5
d51c759 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-hyb6
3bb2290 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-mc6
186b64d default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-hyb7
1ad3bc7 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-mc7
972eb8b default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-hyb8
b96a896 default virtualbox running /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-mc8
# vagrant ssh <id>
vagrant ssh 7a9a0d5
# or also cd into relevant dir and ssh directly
cd /home/jenkins/jenkins2root/workspace/test-emu-prepare-dut/cmf_testbed/emulator/functional/em-tdk-hyb1
vagrant status
vagrant ssh
|
To format SD card using SD Formatter
To flash SD card
Some RPI Setup links: