Setup
If you already have an existing setup of Automatics you can follow the steps in this page for upgrading the software's Automatics Technology Stack - Upgrade
Software Requirement
Please note that even though the recommended version of Orchestration is JDK17, it is reverse compatible with JDK11 also. So code also be compiled with JDK11. Follow Steps below to install Java 17 on CentOS/linux.
- Update before installing any new program
sudo yum install java-17-openjdk
Sample output:
Follow below steps to install and configure MariaDB on CentOS/linux
All the packages need to be up-to-date. Use the following command to update your packages.
Create a new repo file for the latest version.
vi /etc/yum.repos.d/MariaDB.repo
Copy and paste the following contents into the /etc/yum.repos.d/mariadb.repo file.
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.11/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
To exit and save the Vim file, type :wq and press Enter.
- Install the Latest Version of MariaDB
Install 10.11 of MariaDB and the most common packages, using the command below.
sudo yum install MariaDB-server galera-4 MariaDB-client MariaDB-shared MariaDB-backup MariaDB-common
- Start MariaDB
Start the MariaDB service with this command.
sudo systemctl start mariadb
To set MariaDB to start automatically when the VM boots up, use the following command.
sudo systemctl enable mariadb
Check the current status of MariaDB using this command.
sudo systemctl status mariadb
Check the currently installed MariaDB version with this command.
Tomcat 9.0.XX
Follow below Steps to setup Tomcat server on CentOs/linux.
wget -c https://downloads.apache.org/tomcat/tomcat-9/v9.0.85/bin/apache-tomcat-9.0.85.tar.gz
- Extract the Tomcat archive to the directory in which you want to deploy the Automatics Tools
sudo tar -xvf apache-tomcat-9.0.85.tar.gz -C /opt/automatics/
- Navigate to the tomcat file path and run ./bin/startup.sh cmd to start the tomcat server.
Sample Output:
Follow below steps to setup maven on Centos/linux.
- Add maven bin folder path using cmd "export PATH=<MAVEN_FOLDER_PATH>:$PATH.
Example:
- Verfiy Maven is installed by giving "mvn -v" cmd.
- SMTP server (optional, if email feature in Automatics to be enabled)
MySQL Configuration
- Create database with name ‘automatics’.
- Execute the script 'Automatics_DB.sql' that is available with Orchestration source at 'automatics\resources\'. Now the Automatics tables are created with basic configuration data.
- Execute the script 'Automatics_RDKB' that is available with Orchestration source at 'automatics\resources\'. Now the Automatics tables are updated with basic Automatics_RDKB data.
- Execute the script 'Automatics_MySqlProcedures.sql' that is available with Orchestration source at 'automatics\resources\'. This is a stored procedure invoked by orchestration to update test pass/fail count in DB.
WAR Generation
- Pull the latest Automatics Orchestration tool project from the repository (rdk/tools/automatics) with branch “rdk-next”.
- After taking the pull, do “mvn clean install” then war file will be generated and will be present inside Automatics/release/Automatics-v<version>/ folder inside the project.
- Rename war to Automatics.war.
War Deployment
- Copy the Automatics.war file to the apache-tomcat/webapps folder.
- Copy the restartTMR.sh file which is inside Automatics/config to apache-tomcat/bin folder.
- Copy “hibernate.cfg.xml”(inside Automatics/config folder) and “log4j.properties”(inside Automatics) files to any specific location,
- Update “hibernate.cfg.xml” with database user name and password. And the password updating in "hibernate.cfg.xml” should be Base64 encoded. And, the DB name should be 'automatics'.
- Add the following params to JAVA_OPTS inside apache-tomcat/bin/catalina.sh file
-DAutomatics -DhibernateUI.config.file = {path to hibernate config file}
- Log4j Configuration
- Orchestration <= v3.0
- Orchestration >= v3.1
- Log4j configuration is optional.
- log4j.properties has been replaced with log4j2-test.xml and is available at Automatics/src/main/resources/. By default, log4j2-test.xml logging configuration will be taken and no explicit configuration is required. However, if custom logging is preferred then it can be configured with JVM property
- Add the following param to JAVA_OPTS inside apache-tomcat/bin/catalina.sh file
-Dlog4j.configurationFile={path_to_custom_log4j2-test.xml}
- Create empty files childUI.jmd and mainUI.jmd in apache-tomcat/bin/. This is required by orchestration for job scheduling.
- Navigate to apache-tomcat/bin directory and start the server by executing command.
./startup.sh
- To verify the server logs, navigate to logs directory (apache-tomcat/logs) under tomcat.
- catalina.out - server start up logs
- automatics_{time_stamp} - Automatics logs
- traces/trace_automatics_{time_stamp} - Automatics trace logs
- Automatics Orchestration can be launched from “{protocol}://{host:port}/Automatics/login.htm
- To stop the server type ./shutdown.sh.
Configuration
- Launch Automatics Orchestration.
- For the first time, login with username as "admin" without password using Self Authentication.
- After a successful login, we need to setup the following configuration. Some of the configurations are visible only to admin user. Also, it is recommended to keep configuration related access to be retained only with admin. (eg Access to page 'System Configuration' ).
System Configuration
- From Automatics UI, go to Settings -> System Configuration page.
The system configuration page has following parameters and partner has to configure values for their environment. The description of each parameter is provided below which helps to configure them.
System Config Param | Description |
DEFAULT_SYNDICATION_PARTNER | Default syndication partner. Eg: Company name |
SYNDICATION_PARTNERS | Syndication partner names in comma separated format. Eg: Company name1, Company name2 |
DEVICE_INVENTORY_BASE_URL | Base URL which is having the rest implementation of device management |
EXECUTION_ENVIRONMENT_TYPES | Execution environment types in comma separated format. Supported values are RDKV, RDKB |
JUN | Jenkins Username |
JUP | Jenkins Password |
LDAP | LDAP Configuration values |
Test_Types | Test Types supported by Automatics, QUICK, QUICK_CI, 1HOUR, 4HOUR, 2DAYS, FAST_QUICK, FAST_QUICK_CI. New test types can also be created here using comma separated format. |
EXECUTION_PRIORITY | Test cases will be executed based on the priority. Default provided values in comma separated format for execution priority are P0,P1,P2,P3 |
JOB_NAME_PREFIX | Prefix of the Jenkins job name. Depending on the test environment partner can configure multiple jenkins jobs for test execution and they all should have name that is configured here with an index. Default provided value is “GENERIC_RDKM_JOB”. eg: GENERIC_RDKM_JOB1, GENERIC_RDKM_JOB2 ...GENERIC_RDKM_JOBN |
MINIMUM_FAIL_PERCENTAGE_TO_RETRY_TEST | After test execution, orchestration will calculate the percentage of failed test steps in a test script. If the failure percentage is beyond the limit configured here, then test will be automatically selected for re-trigger. Default value is 20 |
USER_DEFAULT_MODULES | Values of the default modules that can be accessed by the guest user. Default provided values separated by hyphen -3-5-6-8-9-13- |
MASTER | Comma separated string keywords to identify a build name as master |
SPRINT | Comma separated string keywords to identify a build name as sprint |
STABLE | Comma separated string keywords to identify a build name as stable |
CI_BUILD | Comma separated string keywords to identify a build name as CI build |
TM_TYPE | Automatics deployment environment. Valid values DEV or PROD |
HEAD_ENDS | Head ends to which devices are connected. Values added in comma separated format |
SERVICE_CI_VERIFICATION_VALUE | Value for service CI_VERFICATION. Supported value CI_VERIFICATION. |
SERVICE_FUNCTIONAL_VERIFICATION_VALUE | Value for service FUNCTIONAL_VERIFICATION. Supported value FUNCTIONAL _VERIFICATION. |
REASON_TYPES | Reasons for adding/updating/deleting scripts via 'Manage Script' page. Values added in comma separated format |
MAX_DAILY_ACCEPTABLE_GRT_BUILD_FROM_PORTAL | Ideally automatics process one time only the same CI build for a device. If process/execution to be done multiple time on same CI build for same device this property can be configured. |
RDK_PORTAL_BASE_URL | RDK Portal base url; Required only for integration with Partner's CI Portal |
RDK_PORTAL_AUTH | Authentication token to access the RDK portal. Currently, this is not used by Orchestration for communication |
ISSUE_MANAGEMENT_URL | Issue/Defect Management service base url; Required only for automated defect creation |
AUTOMATICS_BASE_URL | Base URL of Automatics Orchestration Application |
MICROSERVICE_URL | Base URL of Scriptless Service |
TEST_TYPE_MAP | Test types enums are removed from orchestration and added here as key value pairs. Use the insert script from Automatics_DB.sql mentioned with a comment as 'Changes for Test type configuration' to insert this record. |
Device Models
- From Automatics UI, navigate to Settings -> Manage Scripts -> Run on Models
User needs to configure all device models that is going to execute from Automatics.
Automatics identifies a device model from its build name. For this, admin user has to configure device model and its corresponding build name. It requires only to map the device model and initial starting sub string of build name. The starting few letters(image prefix) in build name should be unique to a device model and this should be added in 'Image Name Prefix' text.
- Click on Add Device Name
- Enter the Device Name.
- Enter the Image Prefix Name. The build name prefix which will be unique to identify a device model. Automatics identify a device model from the build name.
- Select the device category.
- Check-in isClientDevice, if the device is a RDKV client device.
Device Group
- From Automatics UI, navigate to Settings -> Device Groups
For Automatics to use devices from partner's inventory, device groups have to be configured. The device group name in inventory should be added here so that user can execute tests on devices within these groups only. Admin user can map device groups to users so that only those users mapped to device group can execute tests on device.
- Go to Settings -> Device Groups
- Click on Add New Device Group Name
- Enter the Device Group Name. Devices should be present in this Group Name.
Resource Details
- From Automatics UI, navigate to Settings -> Resource Details
User can configure Jenkins details in resource details. The test execution happens at jenkins. The jenkins job name should be configured in 'System Configuration' page for param 'JOB_NAME_PREFIX'.
- Click on Add Resource Details.
- Enter the Jenkins resource base URL
- Select the Category type from the dropdown (Select if jenkins is configured for RDKV or RDKB execution).
- Maximum Parallel Jobs :- It represents the number of jobs configured in Jenkins for Automatics. It also requires, the configured jenkins jobs should have pre-defined name which is configurable from system property 'JOB_NAME_PREFIX'. For, eg: If "JOB_NAME_PREFIX" is set to AUTO_JOB, then in jenkins, jobs dedicated for Automatics execution should have names AUTO_JOB1, AUTO_JOB2, AUTO_JOB3 etc. Here, the Maximum Parallel Jobs configured is 10. So at a time Automatics can perform 10 job executions and remaining jobs will be in Queued state in Automatics waiting for free jenkins job.
Source Code Repository
Admin user has to configure source repositories of test projects in Automatics Orchestration.
User can update source repo details via DB. UI support is not available now.
Source Repo Configuration for RDKB
Following are the source repository details to be configured for RDKB. Please execute following SQL Queries.
Configure generic core tests
insert into source_code_details (NAME, CATEGORY_TYPE, SOURCE_REPO, SOURCE_BRANCH, DELETED_DATE, UPDATED_BY, UPDATED_DATE) values ('RDKB_CORE_TESTS', 'RDKB', 'generic-automation-tests', 'rdk-next', NULL, 'admin', now());
Configure RDKB tests
insert into source_code_details (NAME, CATEGORY_TYPE, SOURCE_REPO, SOURCE_BRANCH, DELETED_DATE, UPDATED_BY, UPDATED_DATE) values ('RDKB_TESTS', 'RDKB', 'rdkb-tests', 'rdk-next', NULL, 'admin', now());
Source Repo Configuration for RDKV
Following are the source repository details to be configured for RDKV. Please execute following SQL Queries.
Configure generic core tests
insert into source_code_details (NAME, CATEGORY_TYPE, SOURCE_REPO, SOURCE_BRANCH, DELETED_DATE, UPDATED_BY, UPDATED_DATE) values ('RDKV_CORE_TESTS', 'RDKV', 'generic-automation-tests', 'rdk-next', NULL, 'admin', now());
Job Settings
From Automatics UI,User can navigate to the page Settings -> Job Settings.
And verify (AUTO_QUICK_TEST_JOB, AUTO_QUICK_TEST_JOB_CI, AUTO_1HOUR_TEST_JOB, AUTO_4HOUR_TEST_JOB,AUTO_2DAYS_TEST_JOB,AUTO_COMPONENT_TEST_JOB,) jobs are created in the UI.
If job is not created please follow this Url Automatics Orchestration User Manual - Preferred Program - RDK Central Wiki
If the jobs not configured you can follow the UI.
For the below test types configured in Automatics, user has to create a Job template from Job Settings page. The Job names and test types names should be configured as below.
Job templates are created . If it is not not created you can follow Automatics Orchestration User Manual - Preferred Program - RDK Central Wiki
User can create a new test type, map the test cases to the newly created test type and execute the test cases. To create a new test type add the test type using system configuration tab
under 'Test_Types' configuration name. If a new test type is created, then a new job as to be created based on test type name. The job name should be in the format of 'AUTO_'+'Test type name'+'_TEST_JOB'.
For example If a test type is created as 'REGRESSION', then the job name should be 'AUTO_REGRESSION_TEST_JOB'. Do not create new test types by using the use the test type name that are already present
in 'Test_Types' and 'TEST_TYPE_MAP' config values from the system_config table.
Test Type | Template Job Name |
---|
QUICK, FAST_QUICK | AUTO_QUICK_TEST_JOB |
QUICK_CI, FAST_QUICK_CI | AUTO_QUICK_TEST_JOB_CI |
1HOUR | AUTO_1HOUR_TEST_JOB |
4HOUR | AUTO_4HOUR_TEST_JOB |
2DAYS | AUTO_2DAYS_TEST_JOB |
Tests not mapped to any group | AUTO_COMPONENT_TEST_JOB |
Email Notification
- From Automatics UI, navigate to Settings -> System Configuration.
Automatics Orchestration has the support to update test execution details and other updates to users via automated email notification.
To enable this feature, admin user has to configure following fields in System Configuration page:
System Config Param | Description |
---|
MAIL_FROM | Email sender id |
SMTP_HOST | SMTP server host details eg: {smtpHost:port} |
Following features should be given permission only to Admin.
Upgrade Automatics
Admin can upgrade the Automatics by going to Settings -> Upgrade Automatics
Automatics provides support for automated deployment.
- Click on Browse button, now select the Automatics.war file in releases/automatics-v{pom_version} folder and click on Open.
- Click on Upgrade button to upgrade the Automatics.
User Management
- From Automatics UI, navigate to Settings -> User Management
Admin can manage all the users of the Automatics.
Admin can Add/Delete the user details and groups. Also, manages the user group module access.
- Click on Add New User
- Enter Preferred User Id: “guest”
- Enter the user name: “guest”
- Enter user email id
- Select user group: GUEST
- Click on Add New User Group
- Enter User Group name: GUEST_1
- Click on Delete User Group
- Select the User Group, you want to delete and click on Delete
- Click on User Group Module Access
- Select the User Group and click on Fetch Access Rights.
- Select the access right checkbox of the Module Name, you want to provide access.
- Click on Save.
14 Comments
Unknown User (duraisamyc)
>> Copy “hibernate.cfg.xml”(inside Automatics/config folder) and “log4j.properties”(inside Automatics) files to any specific location.
The "log4j.properties" file is not available under automatics/ directory. Can you specify the correct directory path?
>> Update “hibernate.cfg.xml” with database user name and password. And the password updating in "hibernate.cfg.xml” should be Base64 encoded. And, the DB name should be 'automatics'.
Do we need to update this .xml at automatics/config path? or at the new specific location path where we copied this file?
Unknown User (rsudhakaran)
Hi Unknown User (duraisamyc)
Thanks for the queries.
log4j.properties is required in versions till v3.0 of Orchestration and was available at Automatics/src/main/resources/. And, log4j configuration was mandatory till this version.
With v3.1 onwards, log4j.properties has been replaced with log4j2-test.xml and is available at Automatics/src/main/resources/. By default, log4j2-test.xml logging configuration will be taken and no explicit configuration is required. However, if custom logging is preferred then it can be configured with JVM property
-Dlog4j.configurationFile={path_to_custom_log4j2-test.xml}
hibernate.cfg.xml - We have to copy the file to specific location in deployment VM and update the credentials to this file. And, the path to this file will be updated in property -DhibernateUI.config.file={path_to_hibernate_config_file}
Updated log4j configuration details in wiki page.
Unknown User (duraisamyc)
Hi Unknown User (rsudhakaran)
Thanks for the clarifications. I forgot to add additional comments, yes I skipped log4j.properties and was able to bring up the orchestration UI.
Unknown User (lpiskunowicz)
When i execute
insert into source_code_details (NAME, CATEGORY_TYPE, SOURCE_REPO, SOURCE_BRANCH, DELETED_DATE, UPDATED_BY, UPDATED_DATE) values ('RDKB_CORE_TESTS', 'RDKB', 'generic-automation-tests', 'rdk-next', 'NULL', 'admin', now());
the value stored for DELETED_DATE="0000-00-00 00:00:00"
the server errors out with:
SEVERE: Servlet.service() for servlet [spring] in context with path [/Automatics] threw exception [Request processing failed; nested exception is org.hibernate.exception.GenericJDBCException: Value '0000-00-00 00:00:00' can not be represented as java.sql.Timestamp] with root cause
Unknown User (gilles.printemps)
Is there a way to update the different scripts for a latest version of mysql (version 8.0)?
version 5.6 is very (very...) old !!!
Moreover, a space is missing in the "Automatics_DB.sql", which is causing an error with version 8.0:
"--alter for python framework integration" >> "-- alter for python framework integration"
Unknown User (shameemm.s)
Hi,
Thank you for pointing it out, fix is released
Regards,
Shameem
Unknown User (gilles.printemps)
I have an issue with the "War generation"
automatics/src/main/java/com/automatics/objects/StbDetailsList.java:[30,33] package javax.xml.bind.annotation does not exist.
The library
javax.xml.bind
has been removed from Java 11 release. Can someone update "automatics" accordingly?Unknown User (gilles.printemps)
1) Adding the following dependency in the pom.xml seems finding the issue (At least, "Automatics.war" is generated")
Can someone confirm?
2) Renaming war to Automatics.war
in my case and with the above fix, "Automatics.war" is automatically generated.
Is it normal as the documentation is saying "Rename war to Automatics.war"?
Unknown User (gilles.printemps)
After following the different steps (compilation, deployment...) and restarting Tomcat, nothing is working anymore:
- Tomcat manager is no more answering
- DeviceManager is no more answering
if the following lines from catalina.sh, Tomcat manager and DeviceManager are working again
When 2 lines are added, "Catalina.out" shows that Tomcat seems not loading everything...
automatics_error.log is empty
automatics.log
automatics_trace.log
Any idea? What's happening?
As currently only Automatics Orchestration and DeviceManager war files are deployed, is there a dependency on another app?
Unknown User (gilles.printemps)
Same result with the following components deployed:
- Automatics.war
- AutomaticsProps.war
- DeviceManager.war
- DeviceManagerUI.war
Unknown User (rsudhakaran)
Hi Unknown User (gilles.printemps)
The required version for Automatics is Java 8. Could you please deploy Automatics using Java 8.
Also, Automated deployment support is available for Automatics. Could you please try that Automatics Automated Build & Deployment
Unknown User (gilles.printemps)
Hi,
Can you explain why Java 8 as it is a very (very....) old version and no more supported on latest Linux version?
Can you imagine installing a 2014 version on a production environment server?
Java Releases
Regarding your automatic deployment, you have to be aware that, today, i386 architecture is not the only one available.
As example all new Apple MBP are based on arm!!! So, your automated deployment might no work on all computers.
Moreover, if you want to understand correctly, I think that manual install is the best way!!!
BR
Unknown User (gilles.printemps)
FYI, the issue highlighted before was NOT related to the JDK version but to the installation of MySQL.
From recent Linux versions, MySQL 5.6 is no more available as it it very old (Released in 2013).
As of today (March 2023), MySQL 8 seems the default one...
Connection to a latest version of MySQL seems not working using the source code provided by "Automatics".
Unknown User (shameemm.s)
Hi,
We are currently working on upgrading Database as well as JDK and Jenkins
Regards,
Shameem