Versions Compared

Old Version 7

changes.mady.by.user Abhija Jayan

Saved on

compared with

New Version Current

changes.mady.by.user Abhija Jayan

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Architecture and working principle

 Image Removed

Advantages of the reference telemetry setup

1.Real-Time Monitoring: Telemetry dashboards provide real-time monitoring of systems and processes. This allows for immediate detection and response to any issues or anomalies, enhancing operational efficiency and reducing downtime.

2.Data Visualization: They convert complex data sets into easy-to-understand visual formats like graphs, charts, and maps. This aids in quicker comprehension and better decision-making by presenting data in a user-friendly manner.

3.Performance Analysis: By tracking various metrics and KPIs, telemetry dashboards enable detailed performance analysis. This can lead to insights into trends, patterns, and potential areas for improvement.

How to setup the telemetry upload setup

Elastic Search -  Kibana Setup

Server Prerequisites

VM :  Ubuntu 20.04 +

You can either deploy the EK setup in the same VM or use another VM. For reference setup, we have used two separate  ubuntu server instances

Given below is the VM Configuration for the RDKM reference setup,  the configs needs to be modified as per the requirement

 VM Configuration

  • RAM : 16GB
  • Disk space :  300 GB
  • Ubuntu : 20.04.6 LTS 

Install Java

  1. Run the following commands from the command line
Code Block
$ sudo apt update
$ sudo apt install openjdk-8-jdk

      2. Check Java installation by running the below command

Code Block
$ java -version

      3. The output should look something like this:

Code Block
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1ubuntu1-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

Install NGINX

  1. Install Nginx by running the following command from command line:
Code Block
sudo apt-get install nginx

     2. If prompted, type y and hit Enter for the process to finish.

Install ElasticSearch

Elastic repositories enable access to all the open-source software in the ELK stack. To add them, start by importing the GPG key.

1 . Enter the following into terminal window to import the PGP key for Elastic:

Code Block
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

2.The system should respond with OK,

3. Next, install the apt-transport-https package:

Code Block
sudo apt-get install apt-transport-https

4. Add the Elastic repository to your system’s repository list:

Code Block
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee –a /etc/apt/sources.list.d/elastic-7.x.list

...


Why we need telemetry upload system


In RDK Telemetry, the process of ingestion refers to the collection of telemetry data from various sources within the RDK ecosystem, such as applications, subsystems, and network interfaces. This data is then brought into a central system for processing and analysis. The transformation aspect involves converting this raw telemetry data into a format that is useful for analysis. This may include structuring, filtering, and enriching the data to make it more meaningful for specific uses, like performance monitoring or issue diagnosis. The transformed data can then be used for various purposes, including troubleshooting, network optimization, and improving user experiences. This process ensures that the vast amounts of data generated by RDK devices are effectively utilized to enhance service delivery and operational efficiency.


This reference setup is a simpler solution to upload and view telemetry data from RDK devices based on the Spring boot  based Microservice – Elasticsearch DB – Kibana architecture which are used in data processing, data storage and data visualization, respectively.



Architecture and a brief overview of the work flow


 Image Added



  • Telemetry is configured in the RDKB /RDKV devices with on xconf(Single profile telemetry) based telemetry or telemetry2 multiprofile based telemetry.
  • The upload URL will be configured as the Telemetry-collector microservice URL , So data will be uploaded to the microservice from the boxes.
  • Telemetry microservice is a springboot based microservice that will receive the telemetry data, parses it and sends the processed data to Elastic Search
  • Elastic search DB is  where the data is stored, It  is deployed along with Kibana.
  • Kibana is a data visualization tool and  user can analyze and filter out the data stored in Elasticsearch DB , with Kibana
  • Kibana Admin user will create dashboards based on the requirements from the end user.



Advantages of the reference telemetry setup


1.Real-Time Monitoring: Telemetry dashboards provide real-time monitoring of systems and processes. This allows for immediate detection and response to any issues or anomalies, enhancing operational efficiency and reducing downtime.

2.Data Visualization: They convert complex data sets into easy-to-understand visual formats like graphs, charts, and maps. This aids in quicker comprehension and better decision-making by presenting data in a user-friendly manner.

3.Performance Analysis: By tracking various metrics and KPIs, telemetry dashboards enable detailed performance analysis. This can lead to insights into trends, patterns, and potential areas for improvement.


How to setup the telemetry upload system


Elastic Search -  Kibana Setup


Server Prerequisites

VM :  Ubuntu 20.04 +

You can either deploy the EK setup in the same VM or use another VM. For reference setup, we have used two separate  ubuntu server instances

Given below is the VM Configuration for the RDKM reference setup,  the configs needs to be modified as per the requirement

 VM Configuration

  • RAM : 16GB
  • Disk space :  300 GB
  • Ubuntu : 20.04.6 LTS 


Install Java

  1. Run the following commands from the command line
Code Block
$ sudo apt update
$ sudo apt install openjdk-8-jdk

      2. Check Java installation by running the below command

Code Block
$ java -version

      3. The output should look something like this:

Code Block
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1ubuntu1-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)



Install NGINX


  1. Install Nginx by running the following command from command line
Code Block
sudo apt-get update

...

  1. :
Code Block
sudo apt-get install elasticsearchnginx

...

Code Block
sudo nano /etc/elasticsearch/elasticsearch.yml

 

8. You should see a configuration file with several different entries and descriptions. Scroll down to find the following entries:

#network.host: 192.168.0.1
#http.port: 9200

...

Code Block
network.host: localhost

http.port: 9200

10 . Just below, find the Discovery section. We are adding one more line, as we are configuring a single node cluster:

Code Block
discovery.type: single-node

11. By default, JVM heap size is set at 1GB. It is recommended setting it to no more than half the size of your total memory. Open the following file for editing:

Code Block
sudo nano /etc/elasticsearch/jvm.options

12. Find the lines starting with -Xms and -Xmx  and set the value. In reference setup we set it as 4gb

13. Start the Elasticsearch service by running a systemctl command: 

It may take some time for the system to start the service. There will be no output if successful.

Code Block
sudo systemctl start elasticsearch.service

14.  To enable Elasticsearch to start on boot:

Code Block
sudo systemctl enable elasticsearch.service

15.  Use the curl command to test your configuration. Enter the following:

Code Block
curl -X GET "localhost:9200"

16. The name of your system should display, and elasticsearch for the cluster name. This indicates that Elasticsearch is functional and is listening on port 9200.

Install Kibana

1.  Run the following command to install Kibana:

Code Block
sudo apt-get install kibana

2. Allow the process to finish. Once finished, it’s time to configure Kibana. Next, open the kibana.yml configuration file for editing:

Code Block
sudo nano /etc/kibana/kibana.yml

3.  Delete the # sign at the beginning of the following lines to activate them:

...


     2. If prompted, type y and hit Enter for the process to finish.


Install ElasticSearch


Elastic repositories enable access to all the open-source software in the ELK stack. To add them, start by importing the GPG key.

1 . Enter the following into terminal window to import the PGP key for Elastic:

Code Block
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

2.The system should respond with OK,

3. Next, install the apt-transport-https package:

Code Block
sudo apt-get install apt-transport-https


4. Add the Elastic repository to your system’s repository list:

Code Block
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee –a /etc/apt/sources.list.d/elastic-7.x.list



5. Prior to installing Elasticsearch, update the repositories by entering:

Code Block
sudo apt-get update


6. Install Elasticsearch with the following command:

Code Block
sudo apt-get install elasticsearch


7. Elasticsearch uses a configuration file to control how it behaves. Edit the file.


Code Block
sudo nano /etc/elasticsearch/elasticsearch.yml

 


8. You should see a configuration file with several different entries and descriptions. Scroll down to find the following entries:

#network.host: 192.168.0.1
#http.port: 9200



9.Uncomment the lines by deleting the hash (#) sign at the beginning of both lines and replace 192.168.0.1 with localhost.

Code Block
network.host: localhost

http.port: 9200


10 . Just below, find the Discovery section. We are adding one more line, as we are configuring a single node cluster:

Code Block
discovery.type: single-node


11. By default, JVM heap size is set at 1GB. It is recommended setting it to no more than half the size of your total memory. Open the following file for editing:

Code Block
sudo nano /etc/elasticsearch/jvm.options


12. Find the lines starting with -Xms and -Xmx  and set the value. In reference setup we set it as 4gb


13. Start the Elasticsearch service by running a systemctl command: 

It may take some time for the system to start the service. There will be no output if successful.


Code Block
sudo systemctl start elasticsearch.service

14.  To enable Elasticsearch to start on boot:

Code Block
server.port: 5601

server.host: "localhost"

elasticsearch.hosts: ["http://localhost:9200"]

...

6.  Start the Kibana service:

Code Block
sudo systemctl start kibana

7. Next, configure Kibana to launch at boot if you want


Code Block
sudo systemctl enable kibanaelasticsearch.service


815. If   Use the UFW firewall is enabled on your Ubuntu system, you need to allow traffic on port 5601 to access the Kibana dashboard. In a terminal window, run the following commandcurl command to test your configuration. Enter the following:

Code Block
sudocurl ufw-X allow 5601/tcpGET "localhost:9200"

9. To access Kibana, open a web browser and browse to the following address:

Code Block
http://<serverIP>:5601

10 .If you receive a “Kibana server not ready yet” error, Wait for some time ,check if the Elasticsearch and Kibana services are active. 

Configure Kibana for the dashboard

Create required index for RDKB and RDKV as per the requirement

  1. We need to create index for RDKV and RDKB as per the requirement
  2. To create an index from the command line, run the below given curl command for creating RDKB telemetry index in elastic search DB
Code Block
curl -X PUT "localhost:9200/rdkb-telemetry" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "Time": {
        "type": "date" ,
         "format": "yyyy-MM-dd HH:mm:ss"
      }
    }
  }
}


'

     To create RDKV index in elastic search DB, run the following command


16. The name of your system should display, and elasticsearch for the cluster name. This indicates that Elasticsearch is functional and is listening on port 9200.

Install Kibana


1.  Run the following command to install Kibana:

Code Block
sudo apt-get install kibana



2. Allow the process to finish. Once finished, it’s time to configure Kibana. Next, open the kibana.yml configuration file for editing:


Code Block
sudo nano /etc/kibana/kibana.yml

3.  Delete the # sign at the beginning of the following lines to activate them:


#server.port: 5601

#server.host: "your-hostname"

#elasticsearch.hosts: ["http://localhost:9200"]

4. The above-mentioned lines should look as follows:

Code Block
server.port: 5601

server.host: "localhost"

elasticsearch.hosts: ["http://localhost:9200"]



5. Save the file


6.  Start the Kibana service:

Code Block
sudo systemctl start kibana


7. Next, configure Kibana to launch at boot if you want


Code Block
sudo systemctl enable kibana


8. If the UFW firewall is enabled on your Ubuntu system, you need to allow traffic on port 5601 to access the Kibana dashboard. In a terminal window, run the following command:


Code Block
sudo ufw allow 5601/tcp


9. To access Kibana, open a web browser and browse to the following address:

Code Block
http://<serverIP>:5601


10 .If you receive a “Kibana server not ready yet” error, Wait for some time ,check if the Elasticsearch and Kibana services are active. 


Configure Kibana for the dashboard



Create required index for RDKB and RDKV as per the requirement


  1. We need to create index for RDKV and RDKB as per the requirement
  2. To create an index from the command line, run the below given curl command for creating RDKB telemetry index in elastic search DB
Code Block
curl -X PUT "localhost:9200/rdkvrdkb-telemetry" -H 'Content-Type: application/json' -d'
{
  "mappings": {
    "properties": {
      "Time": {
        "type": "date" ,
         "format": "yyyy-MM-dd HH:mm:ss"
      }
    }
  }
}


'

   

...

 To create

...

RDKV index in elastic search DB, run the following command


Code Block
curl -X PUT "localhost:9200/rdkv-telemetry" -H 'Content-Type: application/json' -d'

...

  • Go to 'Console' tab

Image Removed

  • Paste the below code and click on the execute button to create index for RDKB
Code Block
PUT /rdkb-telemetry
{
  "mappings": {
    "properties": {
      "Time": {
        "type": "date" ,
         "format": "yyyy-MM-dd HH:mm:ss"
      }
    }
  }
}
'


     3. To create an index from the  Kibana 

  •   Click on menu

    Image Added


  • Go to DevTools under 'Management tools'

    Image Added


  • Go to 'Console' tab

Image Added


  • Paste the below code and click on the Paste the below code and click on the execute button to create index for RDKB


Code Block
PUT /rdkvrdkb-telemetry
{
  "mappings": {
    "properties": {
      "Time": {
        "type": "date" ,
         "format": "yyyy-MM-dd HH:mm:ss"
      }
    }
  }
}

Image Removed

Configuring the index for use in the dashboard

  1. Go to the menu Stack management →Index patterns. Click on 'Create index pattern'

Image Removed

   2.  If you want to setup RDKB telemetry dashboard, add name as rdkb-telemetry, Select 'Time' from the drop down menu, You will get a message like this - 'Your index pattern matches 1 source.' Click on 'Create index pattern'


  • Paste the below code and click on the execute button to create index for RDKB


Code Block
PUT /rdkv-telemetry
{
  "mappings": {
    "properties": {
      "Time": {
        "type": "date" ,
         "format": "yyyy-MM-dd HH:mm:ss"
      }
    }
  }
}

Image Added


Configuring the index for use in the dashboard


  1. Go to the menu Stack management →Index patterns. Click on 'Create index pattern'

Image Added


   2.  If you want to setup RDKB telemetry dashboard, add name as rdkb-telemetry, Select 'Time' from the drop down menu, You will get a message like this - 'Your index pattern matches If you want to setup RDKV telemetry dashboard, add name as rdkv-telemetry, Select 'Time' from the drop down menu, You will get a message like this - 'Your index pattern matches 1 source.' Click on 'Create index pattern'.

Image Removed


If you want to setup RDKV telemetry dashboard, add name as rdkv-telemetry, Select 'Time' from the drop down menu, You will get a message like this - 'Your index pattern matches 1 source.' Click on 'Create index pattern'.


Image Added


Telemetry collector microservice Telemetry collector microservice application

Server Prerequisites

...

There are  upload APIS for RDKB and RDKV devices separately in the microservice. So we need to  setup elastic search URL and indexes corresponding to rdkb and rdkv in  telemetry-collector\src\main\resources\application.properties. Sample application.properties with reference setup Elastic search given. Change the elastic search URL corresponding to your elastic search-Kibana setup

Code Block
rdkv.index=rdkv-telemetry
rdkb.index=rdkb-telemetry
elasticsearch.url=http://52.71.72.93:9200/  <Change this accordingly>


server.port=8080

      2.   Go to telemetry-collector and run the below maven command

Code Block
mvn clean install

     3. War file- telemetry-collector.war will be created in the folder - telemetry-collector\target 

Install Tomcat 9

  1. Download Tomcat 9 from the Apache Tomcat Archive with the below given step
Code Block
wget -c https://downloads.apache.org/tomcat/tomcat-9/v9.0.85/bin/apache-tomcat-9.0.85.tar.gz

       2. Extract the Tomcat archive to the directory in which you want to deploy the Automatics Tools

Code Block
sudo tar -xvf apache-tomcat-9.0.85.tar.gz -C /opt/automatics/

 Deployment of the war  

  1. Copy the telemetry-collector.war file to the apache-tomcat/webapps folder.
  2. Navigate to apache-tomcat/bin directory and start the server by executing command.
Code Block
./startup.sh

     3. Sample output is given below

Image Removed

     4. Logs can be found in the file -  apache-tomcat<>/logs/catalina.out

 telemetry-collector\src\main\resources\application.properties. Sample application.properties with reference setup Elastic search given. Change the elastic search URL corresponding to your elastic search-Kibana setup

Code Block
rdkv.index=rdkv-telemetry
rdkb.index=rdkb-telemetry
elasticsearch.url=http://52.71.72.93:9200/  <Change this accordingly>


server.port=8080


      2.   Go to telemetry-collector and run the below maven command


Code Block
mvn clean install


     3. War file- telemetry-collector.war will be created in the folder - telemetry-collector\target 


Install Tomcat 9


  1. Download Tomcat 9 from the Apache Tomcat Archive with the below given step
Code Block
wget -c https://downloads.apache.org/tomcat/tomcat-9/v9.0.85/bin/apache-tomcat-9.0.85.tar.gz

       2. Extract the Tomcat archive to the directory in which you want to deploy the Automatics Tools

Code Block
sudo tar -xvf apache-tomcat-9.0.85.tar.gz -C /opt/automatics/


 Deployment of the war  


  1. Copy the telemetry-collector.war file to the apache-tomcat/webapps folder.
  2. Navigate to apache-tomcat/bin directory and start the server by executing command.
Code Block
./startup.sh

     3. Sample output is given below

Image Added


     4. Logs can be found in the file -  apache-tomcat<>/logs/catalina.out


Telemetry upload APIS


The APis support upload of both telemetry (Single profile) and telemetry(Multi Profile) which are in different JSON schema/format


RDKB telemetry upload API : 


Code Block
http://<Server IP/FQDN>:8080/telemetry-collector/rdkb-collector



RDKV telemetry upload API


Code Block
http://<Server IP/FQDN>:8080/telemetry-collector/rdkv-collector


Reference setup details


Server details


We have two servers which are used for microservice deployment and Elasticsearch-Kibana setup


Telemetry collector VM - 

IP - 52.0.158.162

FQDN :  telemetrycollector.rdkcentral.com

RAM : 16GB

Disk space :  128 GB

Ubuntu : 20.04.6 LTS


Telemetry Dashboard VM - 

IP- 52.71.72.93

FQDN :  telemetrydashboard.rdkcentral.com

RAM : 16GB

Disk space : 300 GB +

Ubuntu : 20.04.6 LTS


Telemetry collector

These are the telemetry upload URLs configured for RDKB and RDKV


RDKB : https://telemetrycollector.rdkcentral.com/telemetry-collector/rdkb-collector


RDKV: https://telemetrycollector.rdkcentral.com/telemetry-collector/rdkv-collector

Telemetry dashboard


We have currently setup RDKB and RDKV dashboards in the Kibana instance. Given below are the details

RDKB dashboards

Device Info   - https://telemetrydashboard.rdkcentral.com:5601/app/dashboards#/view/d7462060-ab98-11ee-9d36-e37c9e3d744b?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2w,to:now))

Device Performance - https://telemetrydashboard.rdkcentral.com:5601/app/dashboards#/view/714270f0-a933-11ee-9d36-e37c9e3d744b?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2w,to:now))

Error Markers - https://telemetrydashboard.rdkcentral.com:5601/app/dashboards#/view/8953c6a0-a954-11ee-9d36-e37c9e3d744b?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2w,to:now))


RDKV dashboards


https://telemetrydashboard.rdkcentral.com:5601/app/dashboards#/view/30d28b50-95c3-11ee-818a-3bf22ddefffd?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-2w,to:now))



Reference


...