Automatics - Introduction

Automatics is a test setup for verifying RDK builds. It can be integrated with CI flow to validate new changes checked in by user. Also, user can trigger automated tests against given RDK builds.

Automatics helps in faster release of RDK builds by validating builds using automated test execution. The results of test execution can be viewed from results page of Automatics.

Automatics Setup

Setup details will guide the user to setup Automatics system in their environment.  It includes following details.

• Setup details of Automatics Orchestration
• Setup for Automatics Core
• Configuration of Jenkins jobs for test execution
• Setup details of Automatics Property

Execution Environment

• OS - Linux
• Jenkins

Automatics Orchestration

Software Requirement

• JDK 1.8
• MySql 5.0.6
• Tomcat 7.0.92
• Maven 3

Mysql Configuration           

• Create database with name ‘automatics’.
• Execute the script 'Automatics_DB.sql' that is available with Orchestration source at 'automatics\resources\'. Now the Automatics tables are created with basic configuration data.

WAR Generation

• Pull the latest Automatics Orchestration tool project from the repository (rdk/tools/automatics) with branch “rdk-next”.
• After taking the pull, do “mvn clean install” then war file will be generated and will be present inside Automatics/release/Automatics-v0.1/ folder inside the project.
• Rename war to Automatics.war.

War Deployment

• Copy the Automatics.war file to the apache-tomcat/webapps folder.
• Copy the restartTMR.sh file which is inside Automatics/config to apache-tomcat/bin folder.
• Copy “hibernate.cfg.xml”(inside Automatics/config folder) and “log4j.properties”(inside Automatics) files to any specific location,
• Update “hibernate.cfg.xml” with database user name and password. And the password updating in "hibernate.cfg.xml” should be Base64 encoded. And, the DB name should be 'automatics'.
• Add the following params to JAVA_OPTS inside apache-tomcat/bin/catalina.sh file   

-DAutomatics -DhibernateUI.config.file = {path to hibernate config file} -DloggerUI.properties.file = {path to log properties file} 

• Create empty files childUI.jmd and mainUI.jmd in apache-tomcat/bin/. This is required by orchestration for job scheduling.
• Navigate to apache-tomcat/bin directory and start the server by executing command.

./startup.sh

• To verify the server logs, navigate to logs directory (apache-tomcat/logs) under tomcat. 
  • catalina.out - server start up logs
  • automatics_{time_stamp} - Automatics logs
  • traces/trace_automatics_{time_stamp} - Automatics trace logs
• Automatics Orchestration can be launched from “{protocol}://{host:port}/Automatics/login.htm

• To stop the server type ./shutdown.sh.

Configuration

• Launch  Automatics Orchestration.
• For the first time, login with username as "admin" without password using Self Authentication.
• After a successful login, we need to setup the following configuration.

  
  

System Configuration

• From Automatics UI, go to Settings -> System Configuration page.

           The system configuration page has following parameters and partner has to configure values for their environment. The description of each parameter is provided below which helps to configure them.

Device Models

• From Automatics UI, navigate to Settings -> Manage Scripts -> Run on Models

User needs to configure all device models that is going to execute from Automatics. 

Automatics identifies a device model from its build name. For this, admin user has to configure device model and its corresponding build name. It requires only to map the device model and initial starting sub string of build name.

• • Click on Add Device Name
  • Enter the Device Name.
  • Enter the Image Prefix Name. The build name prefix which will be unique to identify a device model. Automatics identify a device model from the build name.
  • Select the device category.
  • Check-in isClientDevice, if the device is a RDKV client device.

Device Group

• From Automatics UI, navigate to Settings -> Device Groups

For Automatics to use devices from partner's inventory, device groups have to be configured. The device group name in inventory should be added here so that user can execute tests on devices from configured device groups only. Admin user can map device groups to users so that only those users mapped to device group can execute tests on device.

• • Go to Settings -> Device Groups
  • Click on Add New Device Group Name
  • Enter the Device Group Name. Devices should be present in this Group Name.

Resource Details

• From Automatics UI, navigate to Settings ->  Resource Details

User can configure Jenkins details in resource details. The test execution happens at jenkins.

• • Click on Add Resource Details.
  • Enter the Jenkins resource base URL
  • Select the Category type from the dropdown (Select if jenkins is configured for RDKV or RDKV execution).
  • Maximum Parallel Jobs :- It represents the number of jobs configured in Jenkins for Automatics. It also requires, the configured jenkins jobs should have pre-defined name which is configurable from system property 'JOB_NAME_PREFIX'. For, eg: If "JOB_NAME_PREFIX" is set to AUTO_JOB, then in jenkins, jobs dedicated for Automatics execution should have names AUTO_JOB1, AUTO_JOB2, AUTO_JOB3. Here, the Maximum Parallel Jobs configured will be 3.

Source Code Repository

            User can configure source repository of test project in Automatics. Test project repository path and branch are passed to jenkins during execution. Jenkins will clone the repository and execute the tests. It requires user to add details from database.

• Update the Source Code Repository Details, by executing sql command on 'source_code_details' table in database.
  • Ex: insert into source_code_details (NAME, CATEGORY_TYPE, SOURCE_REPO, SOURCE_BRANCH, DELETED_DATE, UPDATED_BY, UPDATED_DATE) values ('{generic category name}','{category type}',{repo details of the project been deployed}','{branch name of the project where it is pulled}','NULL', 'admin', now());

 Job Settings

• From Automatics UI, navigate to Settings ->  Job Settings

            For each test type in Automatics, job will be created during test execution. The jobs are created based on job templates. For each test type, a job template need to be created.

• • Click on Add New Job
  • Enter Job Name: Enter the Job Name with the prefix of AUTO followed by TEST_TYPE and JOB, all are separated by “_”.                                

Eg: AUTO_1HOUR_JOB

• • And fill all the necessary details.
  • Click on Save.

Admin Only Features

Following features should be given permission only to Admin.

Upgrade Automatics

Admin can upgrade the Automatics by going to Settings -> Upgrade Automatics

Automatics provides support for automated deployment.

• Click on Browse button, now select the following Automatics.war file and click on Open.
• Click on Upgrade button to upgrade the Automatics.

User Management

• From Automatics UI, navigate to Settings ->  User Management

Admin can manage all the users of the Automatics.

Admin can Add/Delete the user details and groups. Also, manages the user group module access.

• Click on Add New User
  • Enter Preferred User Id: “guest”
  • Enter the user name: “guest”
  • Enter email: guest@automatics.com
  • Select user group: GUEST
• Click on Add New User Group
  • Enter User Group name: GUEST_1
• Click on Delete User Group
  • Select the User Group, you want to delete and click on Delete
• Click on User Group Module Access
  • Select the User Group and click on Fetch Access Rights.
  • Select the access right checkbox of the Module Name, you want to provide access.
  • Click on Save.

Automatics Core

Automatics core is the base part of the system that performs the test execution.  It performs following functionality

• Initialize devices for execution
• Identify tests and create testng suite at runtime
• Perform Pre-Test initialization
• Execute testng tests
• Update test results to Automatics orchestration
• Perform Post-Test cleanup

Software Requirement

• JDK 1.8
• Maven 3

Setup

• Pull the Automatics Core project from the repository (rdk/tools/automatics/automatics-core) with branch (“rdk-next”)
• In pom.xml and update the url in distributionManagement with the respective artifact repository url in the place as mentioned below. Automatics core is required to be deployed to partner's repository. Partner implementation project requires core to be added as maven dependency for using its APIs.

<distributionManagement>

             <repository>

                          <url></url>

            </repository>

 </distributionManagement>     

• Execute “mvn clean deploy” on project. This will deploy the Core project to the partner's repository.

Automatics Generic Tests

This project contains the generic test cases which will be executed by Automatics. The automation id and manual id in automated tests should be matching with the automation Id and manual Id present in Automatics orchestration. During test execution the execution status of each test step will be updated to Automatics orchestration. And, at the end final execution status will be updated to Automatics.

Software Requirement

• JDK 1.8
• Maven 3

Setup

• Pull the latest AutomaticsGeneric Test project from the repository (rdk/tools/automatics/generic-automation-tests) with branch (“rdk-next”).
• Update pom.xml with partner project as a dependency.
• Merge the changes so that it will be available during tests execution.

Automatics Properties

This project contains the properties as a key value pair, which will be used to run the Automatics Core/Partner Implementation/Rest Implementation Projects for executing the test cases. 

Automatics Props Set up:

Automatics system holds configuration data required for test execution in Automatics Props application.

There are two types of configuration data in Automatics Props

• Automatics properties - key value pair data used for configuring data required by Automatics system.
• Device Props – Json file containing device configuration for each model.

Automatics properties file

• Launch {protocol}//{host:port}/AutomaticsProps/automatics/props.
• User will be redirected to automatics properties page where it contains all the Properties which can be changed based on the requirements.
• Automatics Core/Partner Implementation/Rest Implementation Projects will read configuration values from this file.
• To edit the values, please click on “unlocked” button. Update the values and click on submit. This will save the configuration values.

 Device properties file

• Launch {protocol}// {host:port}/AutomaticsProps/deviceProps.
• User will be redirected to device json page, where it contains the device model based configuration.
• To edit device properties, user has to update the file device_config.json deployed in tomcat server at path apache-tomcat/webapps/automatics/

Software Requirement

• JDK 1.8
• Tomcat 7.0.92
• Maven 3

Setup

• Clone AutomaticsProps project from the repository (rdk/tools/automatics/automatics-props) with branch (“rdk-next”).
• Execute “mvn clean install”, and war file will be generated at target directory.
• Copy automaticsProps.war file to apache-tomcat/webapps directory
• Create a directory named “back_up”, in apache-tomcat and place the sample “automatics.properties” file.  When ever user edits automatics properties, a backup file will be generated inside this directory.
• Create a directory “automatics” inside apache-tomcat/webapps.
• Copy “automatics.properties”, "device_config.json" and “config.properties” from automatics-props\src
• Configure“config.properties” with username and password required for Automatics Properties UI login.
•  Start the server using command ./startup.sh.

Automatics Partner Implementation

• Add the “Automatics Core” project as the dependency to the pom.xml of Partner Project.
• After partner project development is completed, deploy the project to partner repository by executing “mvn clean deploy”.

Partner XML configuration file

• Partner can inject their implementation classes to Automatics core using Spring Dependency Injection. 
• Automatics core expects spring configuration file partner-applicationContext.xml to be provided by partner project and it should be available at classpath during test execution.
• All Java API implementations to be injected should be configured in partner-applicationContext.xml.
• partner-applicationContext.xml file contains predefined bean id with respective implementation class. All beans uses "lazy-init" set to true so that bean instance will be created on demand only. And, for beans that requires separate instance for each device, property "scope" is set to value "prototype".

Automatics Rest Implementation

• Partner needs to provide REST based implementation for APIs in device management that is consumed by Automatics orchestration.
• Other device management APIs consumed by Automatics core can be implemented using Java API or REST.
• Automatics core provides rest clients for device management and hardware providers like Power Provider. Partner can provide REST implementation for them based on API specification. Automatics core, by default, will use rest implementation. However, if partners prefers to go with Java API implementation, then in Automatics Properties, update the following properties as shown.

"partner.impl.deviceManager=true”  - When set to true, Automatics expects Java based implementation of device management and it should be configured in partner-applicationContext.xml against bean "deviceProvider".

“partner.impl.powerProvider=true”  - When set to true, Automatics expects Java based implementation of power provider and it should be configured in partner-applicationContext.xml against bean "powerProvider".

Jenkins 

Automatics performs test execution in Jenkins. User can view the status of Jenkins execution from Job Manager page of Automatics. The final execution status will be updated back to Automatics.

Setup

Steps to be taken care during Jenkins configuration

• Install Jenkins
• Configure Java 1.8 and Maven 3
• Select Manage Jenkins and select Configure System
• Create jenkins job with name configured in Automatics orchestration.

• Within job, in General section

  • Check the “Discard old builds”.
  • Select Log Rotation and we can keep the days of build and Max # of builds.
  • Please configure following String parameters

filterTestIds, filterTestType, updateRdkPortal, settopList, JMD_ID, BUILD_NAME, executionMode, gr,grb 

• • Under Source Code Management, Select GIT and specify Repository base url. During execution, the test project repository path and branch will be available from 'gr' and 'grb' parameters.
  • Under Pre Steps in Build section, please add below

1. In home directory, the path to be set up as Jenkins installed path in VM.
2. Inside Maven Configuration, local maven repository to be set
3. Inside Jenkins location, Jenkins URL and respective email address to be set.
4. Under Global properties, we have to set the required list of environment variable.

 Configuring Jenkins Job

Steps to be taken care during job configuration.

• From Jenkins select “New Item”, Specify name of the new job to be created.
• If we want to clone from existing job, we can give name of the existing job in the Copy from text box present in the bottom and select “OK”

In General section

• Check the “Discard old builds”.
• Select Log Rotation and we can keep the days of build and Max # of builds.
• Please configure String parameters filterTestIds, filterTestType, updateRdkPortal, settopList, JMD_ID, BUILD_NAME, executionMode, gr,grb in job
• Under Source Code Management, Select GIT and specify Repository base url.
• Under Pre Steps in Build section, please add below

• Click on Apply and Save button.

Automatics Orchestration - User Manual

1       Automatics Login

User can login into Automatics through any one of the two authentication modes. One is Self-Authentication and another is LDAP Authentication.

By giving the respective credentials such as username & password and selecting either of the Self Authentication/LDAP Authentication modes from the drop down, user can login to Automatics Tool.

User will be able to see the below Homepage after a successful login.

User can see the Builds Processed Details for a default time period of 30 days. User can change the date and click on filter to see the required details.

User can also see the Execution Details of Last 4.

2       Manage Scripts

To perform test execution, user needs to add test details to Automatics system. This can be done from Manage Scripts page.

2.1   Add Script

Script Details

To add a new script, user can click on “Add New” link on the right top of page. Field “Automation Id” should be unique name that identifies an automated test. This id should match with the automation id in automated test case.  Automatics considers RDK builds in any of the 3 categories ‘Sprint’, “Stable” and “Release”. A test case can be mapped to execute in any or all of the build types.

On Next page, Environment can be selected. In the screen shown below, the test is RDKV device related. So mapping the test against RDKV environment. Next applicable device model/s can be selected. End point “RACK_DEVICE” refers to devices managed by rack system and “DESK DEVICE” refers to desk devices.  Head end field refers to device head end. If head end is provided, then during execution, Automatics can request for devices from given head end.

On Next page, user can select the feature required by device during execution of test. Here, as we are adding MOCA related test case, feature “MOCA” is selected. During execution of this test, Automatics will request for device with feature “MOCA” from inventory. If a test case can mapped to a particular RDK component testing, then it can be mapped with Field “RDK Component”.  Then Click on “Save”.

Script Steps

Using filter option, display the newly added test case and click on “Update Step Details“.

We can add test steps from here and to add new step click on “Add Row” button.

After adding test steps, the scripts should be enabled. Only enabled scripts will be considered for execution by Automatics.

Tick mark shows that the script is enabled.

3       Trigger Test Execution

User can trigger a test execution from “Trigger Execution Manually” page.

Service Name “FUNCTIONAL VERIFICATION” refers to functional testing. “CI_VERIFICATION” refers to CI testing. In Build Name field, user can provide the build name that is currently loaded in device. End Point “RACK_DEVICE” and “DESK_DEVICE” which indicates Rack device and Desk device. From Environment Type, user can select values like “RDKV” or “RDKB” and test type like “1HOUR”, “4HOUR” etc.

When “Is Quick Test Required”, it means before executing tests like 1HOUR or others, an image upgrade is to be done on device. QT refers to “Quick Test” which is a special test that will load the device with image mentioned in Build Name field. Only if QT is success, other type of tests will be executed by Automatics. If QT box is unchecked, then image upgrade is not performed.

Navigate to next page. In 2^nd page, user can select test cases to be executed.

Next, navigate to 3^rd page Device Details. User can provide device details from here. Automatics gets devices for execution either from user or via pool based mechanism.

Select the option, ‘Specify Device MAC Address’ for user to provide the mac address. If option ‘Take Boxes from pool’ is selected, then devices will be fetched by Automatics from partner’s inventory. The device needed for execution are identified from test script features added in Manage Script page.

‘Build Appender’ field has different functions in job execution. It is used in 2 different ways.

1. Build Appender – execution grouping.

During execution, when user provides a custom appender value, then in job manager page the jobs are displayed in separate tab with appender name. In results page, the results will be displayed in against buildname with appender so that execution results for the same build with and without appender are displayed as 2 entries.

1. Build Appender – pre-test initializer.

Partner can configure pre-defined appender values in orchestration via System config. If user selects any of the pre-defined appender, then partner can add support at partner side implementation to perform device initialization based on appender value. The Automatics core sends appender to partner during initialization. This helps user to enable specific configurations in device before test execution.

When execution is triggered successfully, below pop message will be displayed.

4       Job Manager

User can verify the status of the job executions from Job Manager page. Also, user can cancel or re-trigger a job, view child jobs, navigate to Jenkins executions are other features provided by Job Manager.

Navigate to Manage Test Trigger, click on Job Manager under RDKV/RDKB.

Here, user can find the status of job executions. Quick Test(QT) is always the parent job of any other jobs. All other job types will be created under QT as child jobs. If user selected QT, then QT will be executed first on all selected devices and only when QT execution status is “COMPLETED” child jobs will start executing. If QT job status is “FAILURE”, then all child jobs become “CANCELLED”. Click on Action to view child jobs.

In scenario where user skips QT, then QT job with dummy QT test will be created and child jobs will be added under it.  User can find the details of job by hovering on the job name.

Details of JOB status are

Initially, the status will be QUEUED.When the job is triggered in Jenkins it changes to SCHEDULED and when job execution is going on, then status will become IN PROGRESS.Once, it is completed you can see the status as COMPLETED.

Action tab on the page supports features in the order as “Go To Jenkins Job”, “Re-trigger”, “View Execution Results”, “View Child Jobs” and “More Options”. User can cancel a job from “More Options”.

5       Reports

User will be to view and download the various reports of Build Execution Results, Device Usage, Script Health & Script Execution Time reports from the Reports section.

5.1 Execution Results

From execution results page, user can view the results of the triggered build executions. It shows the total count of test steps and its Pass, Fail, NT(Not Tested), NR(Not Run) and NA(Not Applicable) counts.

Now, click on Reports and go to Execution Results under RDKV/RDKB.

Following details will be displayed as shown below:

Now, click on the Build name under the Image Name column to see the Test Case Results as below.

You can verify the required information from the Execution Result Details for that particular build.

5.2 Execution Results – TC

User can get an idea on the execution results from test case count via this page.

Now, click on Reports and go to Execution Results - TC under RDKV/RDKB.

Select the date for the required result details.

Following details will be displayed as shown below:

5.3Device Usage Report

User will be able to generate the usage reports of a particular device within the required dates.

Now, click on Reports and go to Device Usage Report.

Enter the following details of Device Mac Address, Date and Time as shown below.

After the entering the details click on Generate.

The following data/details of that particular device will be generated as shown below.

5.4Script Health Report

User can generate the Health Report of the test scripts.

Click on Reports and go to Script Health Report.

Now, enter the following required details of Select Start Date & End Date followed by Select Category Type as shown below and click on Generate.

After clicking on Generate the following data/details will be generated as show below

Also, by clicking on Download Report you can download the report to your local in order to see the data offline.

5.5Script Execution Time Report

User can generate the Execution Time Report of the test scripts for the required dates.

Click on Reports and go to Script Execution Time Report.

Now, enter the following details of Select Category Type, Select Start Date, End Date followed by Select Test Type as shown below and click on Generate.

After clicking on Generate the following data/details will be generated as show below.

Also, by clicking on Download Report you can download the report to your local in order to see the data offline.

6       RDK E2E Manager

RDK/CI portal sends execution requests to Automatics orchestration. Automatics based on filters configured decides if execution to be performed or not.

RDK E2E Manager has 2 sections.

• E2E RDK Request
• E2E RDK Request Filter

6.1 E2E RDK Request

User will be able to see the incoming RDK Requests from the RDK Portal from this page.

Navigate to RDK E2E Manager and click on E2E RDK Request under RDKV/RDKB.

If the request status is accepted, then execution will be triggered by Jenkins. Or else if the status is REJECTED, then no execution will be triggered.

6.2 E2E RDK Request Filter

User can configure RDK filters so that when request comes from RDK portal, Automatics can decide based on it, if execution should trigger or not.

Now, navigate to RDK E2E Manager and click on E2E RDK Request Filter under RDKV/RDKB.

User will be able to see the existing RDK Request Filters, if any present already like the above screen having.

We can Enable/Disable the Request Filters by clicking on the below Enable Selected Records/ Disable Selected Records. Only enabled filters will be considered by Automatics.

Also, we can show/hide the params of the request filter by clicking on Show/Hide Filter Params button above.

Edit Request Filter Details

We can Edit the Request Filter, by clicking on Edit Request Filter Details under the Action column.

Edit the required details of the request filter and click on Save Results.

Add new Request Filter Details

We can add a new Request Filter, by clicking on Add Request Filter Details button above.

Now, enter the required Settings details & Automation ID details of the filter and click on Save Results.

Automatics API Specification