RDK Resources
[*RDK Preferred*]
Code Management Facility
RDK Forums
[RDK Conferences]
RDK Support
Archives
Papers & Presentations Archive
In RDK Telemetry, the process of ingestion refers to the collection of telemetry data from various sources within the RDK ecosystem, such as applications, subsystems, and network interfaces. This data is then brought into a central system for processing and analysis. The transformation aspect involves converting this raw telemetry data into a format that is useful for analysis. This may include structuring, filtering, and enriching the data to make it more meaningful for specific uses, like performance monitoring or issue diagnosis. The transformed data can then be used for various purposes, including troubleshooting, network optimization, and improving user experiences. This process ensures that the vast amounts of data generated by RDK devices are effectively utilized to enhance service delivery and operational efficiency.
This reference setup is a simpler solution to upload and view telemetry data from RDK devices based on the Spring boot based Microservice – Elasticsearch DB – Kibana architecture which are used in data processing, data storage and data visualization, respectively.
1.Real-Time Monitoring: Telemetry dashboards provide real-time monitoring of systems and processes. This allows for immediate detection and response to any issues or anomalies, enhancing operational efficiency and reducing downtime.
2.Data Visualization: They convert complex data sets into easy-to-understand visual formats like graphs, charts, and maps. This aids in quicker comprehension and better decision-making by presenting data in a user-friendly manner.
3.Performance Analysis: By tracking various metrics and KPIs, telemetry dashboards enable detailed performance analysis. This can lead to insights into trends, patterns, and potential areas for improvement.
VM : Ubuntu 20.04 +
You can either deploy the EK setup in the same VM or use another VM. For reference setup, we have used two separate ubuntu server instances
Given below is the VM Configuration for the RDKM reference setup, the configs needs to be modified as per the requirement
VM Configuration
$ sudo apt update $ sudo apt install openjdk-8-jdk
2. Check Java installation by running the below command
$ java -version
3. The output should look something like this:
openjdk version "1.8.0_252" OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1ubuntu1-b09) OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
sudo apt-get install nginx
2. If prompted, type y
and hit Enter for the process to finish.
Elastic repositories enable access to all the open-source software in the ELK stack. To add them, start by importing the GPG key.
1 . Enter the following into terminal window to import the PGP key for Elastic:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
2.The system should respond with OK,
3. Next, install the apt-transport-https package:
sudo apt-get install apt-transport-https
4. Add the Elastic repository to your system’s repository list:
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee –a /etc/apt/sources.list.d/elastic-7.x.list
5. Prior to installing Elasticsearch, update the repositories by entering:
sudo apt-get update
6. Install Elasticsearch with the following command:
sudo apt-get install elasticsearch
7. Elasticsearch uses a configuration file to control how it behaves. Edit the file.
sudo nano /etc/elasticsearch/elasticsearch.yml
8. You should see a configuration file with several different entries and descriptions. Scroll down to find the following entries:
#network.host: 192.168.0.1
#http.port: 9200
9.Uncomment the lines by deleting the hash (#) sign at the beginning of both lines and replace 192.168.0.1 with localhost.
network.host: localhost http.port: 9200
10 . Just below, find the Discovery section. We are adding one more line, as we are configuring a single node cluster:
discovery.type: single-node
11. By default, JVM heap size is set at 1GB. It is recommended setting it to no more than half the size of your total memory. Open the following file for editing:
sudo nano /etc/elasticsearch/jvm.options
12. Find the lines starting with -Xms
and -Xmx and set the value. In reference setup we set it as 4gb
13. Start the Elasticsearch service by running a systemctl command:
It may take some time for the system to start the service. There will be no output if successful.
sudo systemctl start elasticsearch.service
14. To enable Elasticsearch to start on boot:
sudo systemctl enable elasticsearch.service
15. Use the curl
command to test your configuration. Enter the following:
curl -X GET "localhost:9200"
16. The name of your system should display, and elasticsearch for the cluster name. This indicates that Elasticsearch is functional and is listening on port 9200.
1. Run the following command to install Kibana:
sudo apt-get install kibana
2. Allow the process to finish. Once finished, it’s time to configure Kibana. Next, open the kibana.yml configuration file for editing:
sudo nano /etc/kibana/kibana.yml
3. Delete the #
sign at the beginning of the following lines to activate them:
#server.port: 5601
#server.host: "your-hostname"
The above-mentioned lines should look as follows:
#elasticsearch.hosts: ["http://localhost:9200"]
4.
server.port: 5601 server.host: "localhost" elasticsearch.hosts: ["http://localhost:9200"]
5. Save the file
6. Start the Kibana service:
sudo systemctl start kibana
7. Next, configure Kibana to launch at boot if you want
sudo systemctl enable kibana
8. If the UFW firewall is enabled on your Ubuntu system, you need to allow traffic on port 5601 to access the Kibana dashboard. In a terminal window, run the following command:
sudo ufw allow 5601/tcp
9. To access Kibana, open a web browser and browse to the following address:
http://<serverIP>:5601
10 .If you receive a “Kibana server not ready yet” error, Wait for some time ,check if the Elasticsearch and Kibana services are active.
curl -X PUT "localhost:9200/rdkb-telemetry" -H 'Content-Type: application/json' -d' { "mappings": { "properties": { "Time": { "type": "date" , "format": "yyyy-MM-dd HH:mm:ss" } } } } '
To create RDKV index in elastic search DB, run the following command
curl -X PUT "localhost:9200/rdkv-telemetry" -H 'Content-Type: application/json' -d' { "mappings": { "properties": { "Time": { "type": "date" , "format": "yyyy-MM-dd HH:mm:ss" } } } } '
3. To create an index from the Kibana
PUT /rdkb-telemetry { "mappings": { "properties": { "Time": { "type": "date" , "format": "yyyy-MM-dd HH:mm:ss" } } } }
PUT /rdkv-telemetry { "mappings": { "properties": { "Time": { "type": "date" , "format": "yyyy-MM-dd HH:mm:ss" } } } }
2. If you want to setup RDKB telemetry dashboard, add name as rdkb-telemetry, Select 'Time' from the drop down menu, You will get a message like this - 'Your index pattern matches 1 source.' Click on 'Create index pattern'
If you want to setup RDKV telemetry dashboard, add name as rdkv-telemetry, Select 'Time' from the drop down menu, You will get a message like this - 'Your index pattern matches 1 source.' Click on 'Create index pattern'.
VM : Ubuntu 20.04 +
Given below is the VM Configuration for the RDKM reference setup, the configs needs to be modified as per the requirement
VM Configuration
$ sudo apt update $ sudo apt install openjdk-8-jdk
2. Check Java installation by running the below command
$ java -version
3. The output should look something like this:
openjdk version "1.8.0_252" OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1ubuntu1-b09) OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
sudo apt-get -y install maven
2. Verify the Apache Maven version using the following command:
mvn -version
Check out the code from : https://github.com/rdkcentral/telemetry-data-collector
There are upload APIS for RDKB and RDKV devices separately in the microservice. So we need to setup elastic search URL and indexes corresponding to rdkb and rdkv in telemetry-collector\src\main\resources\application.properties. Sample application.properties with reference setup Elastic search given. Change the elastic search URL corresponding to your elastic search-Kibana setup
rdkv.index=rdkv-telemetry rdkb.index=rdkb-telemetry elasticsearch.url=http://52.71.72.93:9200/ <Change this accordingly> server.port=8080
2. Go to telemetry-collector and run the below maven command
mvn clean install
3. War file- telemetry-collector.war will be created in the folder - telemetry-collector\target
wget -c https://downloads.apache.org/tomcat/tomcat-9/v9.0.85/bin/apache-tomcat-9.0.85.tar.gz
2. Extract the Tomcat archive to the directory in which you want to deploy the Automatics Tools
sudo tar -xvf apache-tomcat-9.0.85.tar.gz -C /opt/automatics/
./startup.sh
3. Sample output is given below
4. Logs can be found in the file - apache-tomcat<>/logs/catalina.out
The APis support upload of both telemetry (Single profile) and telemetry(Multi Profile) which are in different JSON schema/format
RDKB telemetry upload API :
http://<Server IP/FQDN>:8080/telemetry-collector/rdkb-collector
RDKV telemetry upload API
http://<Server IP/FQDN>:8080/telemetry-collector/rdkv-collector
We have two servers which are used for microservice deployment and Elasticsearch-Kibana setup
Telemetry collector VM -
IP - 52.0.158.162
FQDN : telemetrycollector.rdkcentral.com
RAM : 16GB
Disk space : 128 GB
Ubuntu : 20.04.6 LTS
Telemetry Dashboard VM -
IP- 52.71.72.93
FQDN : telemetrydashboard.rdkcentral.com
RAM : 16GB
Disk space : 300 GB +
Ubuntu : 20.04.6 LTS
These are the telemetry upload URLs configured for RDKB and RDKV
RDKB : https://telemetrycollector.rdkcentral.com/telemetry-collector/rdkb-collector
RDKV: https://telemetrycollector.rdkcentral.com/telemetry-collector/rdkv-collector
We have currently setup RDKB and RDKV dashboards in the Kibana instance. Given below are the details
Device Performance - https://telemetrydashboard.rdkcentral.com:5601/app/dashboards#/view/714270f0-a933-11ee-9d36-e37c9e3d744b?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2w,to:now))