Overview

This page presents an brief overview about webPA 2.0 (xmidt) components required for a reference webPA cluster setup and explains how to setup the cluster & establish an end-to-end connection with CPE devices.

WebPA is a secure web protocol messaging system for bi-directional communication between cloud server and RDK devices. It was built from the ground up specifically with security and performance as priorities. WebPA 2.0 commonly known as Xmidt (pronounced "transmit") is a combination of a server cluster and client that provide a highly available data path to devices deployed all over the world.

• Unlike TR-69, which polls wide-and-deep across a device landscape, on a less frequent basis, WebPA can precision-poll for the most useful data, much more quickly. That’s mainly because it’s lightweight, and because the load can be redistributed into all the apps needing to access the data.
• Specifically, WebPA is a lightweight connectivity socket, with lower-level protocol connectivity between devices, and the cloud — much like a websocket. It works by each client registering with the cloud, so that it maintains a socket connection to the cloud. On the cloud side, APIs enable polls through which other cloud components could receive events, or translate them in to a TR-181 device data model for further analysis.
• And, because WebPA is an “always on” connection (Websocket), asynchronous notifications for a change in device data are a straightforward exercise, where the application “listens” for webhook events.

Setting up a single node webPA cluster

System Requirement

Software Dependencies

Supervisor

About Supervisor

Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.

Unlike other system initialization services, it is not meant to be run as a substitute for init. Instead it is meant to be used to control processes related to a project or a customer, and is meant to start like any other program at boot time.

How to install

a) Enable Extra Packages for Enterprise Linux (EPEL)
	$ wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
	$ rpm -Uvh epel-release-6-8.noarch.rpm
b) Install Python meld3
	$ yum install python-meld3
c) Install supervisor
	$ yum install supervisor

ZooKeeper

About ZooKeeper

ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs.

How to install

$ yum install zookeeper

Adding a startup script

# cat /etc/init.d/zookeeper 
#!bin/bash
#
# zookeeper Startup Script
#
# chkconfig: 345 90 14
# description: Zookeeper Application Startup Script

# Source function library
. /etc/rc.d/init.d/functions

#-------------------------------------------------------------------------------

start() {
	echo -n $"Starting Zookeeper: "
	/usr/lib/zookeeper/bin/zkServer.sh start
	RETVAL=$?
	echo
	[ $RETVAL -eq 0 ] && echo "[ OK ]"
}

stop() {
	echo -n $"Stopping Zookeeper: "
	/usr/lib/zookeeper/bin/zkServer.sh stop
	RETVAL=$?
	echo
	[ $RETVAL -eq 0 ] && echo "[ OK ]"
}

restart() {
	stop
	start
}

case "$1" in
  start)
	start
	;;
  stop) 
	stop
	;;
  restart|force-reload|reload)
	restart
	;;
  status)
	/usr/lib/zookeeper/bin/zkServer.sh status
	RETVAL=$?
	;;
  *)
	echo $"Usage: $0 {start|stop|status|restart|reload|force-reload}"
	exit 1
esac

exit $RETVAL

Enable the service at bootup

Launch system-config-services from a console and enable the zookeeper service from the services list.

Process flow diagram

WebPA Server components setup

Below is the list of components needed for a Xmidt (webPA 2.0) cluster setup. For a single node reference setup, few of the services are not mandatory hence not used.

Using the pre-built packages from GitHub

a) Import the GPG Key (Required once, common for all the packages)
	$ rpm --import https://github.com/Comcast/tr1d1um/releases/download/0.0.1-65/RPM-GPG-KEY-comcast-webpa
b) Install the packages
	$ rpm -Uvh https://github.com/Comcast/scytale/releases/download/0.1.1-83/scytale-0.1.1-83.el6.x86_64.rpm
	$ rpm -Uvh https://github.com/Comcast/talaria/releases/download/0.1.1-153/talaria-0.1.1-153.el6.x86_64.rpm
	$ rpm -Uvh https://github.com/Comcast/tr1d1um/releases/download/0.1.1-228/tr1d1um-0.1.1-228.el6.x86_64.rpm
Note: Change version number for downloading the required package.

Building from the Source

If pre-built packages are already installed as explained in previous section & we want to use the same, skip to configuration section

Dependencies of Build system

Install golang

Required for compiling server components written in go language.

$ sudo yum install golang

install glide

Glide is a package manager for Go that is conceptually similar to package managers for other languages. Glide provides the following functionality:

• • Records dependency information in a glide.yaml file. This includes a name, version or version range, version control information for private repos or when the type cannot be detected, and more.
  • Tracks the specific revision each package is locked to in a glide.lock file. This enables reproducibly fetching the dependency tree.
  • Utilizes vendor/ directories, known as the Vendor Experiment, so that different projects can have differing versions of the same dependencies.

$ wget -c https://github.com/Masterminds/glide/releases/download/v0.13.1/glide-v0.13.1-linux-amd64.tar.gz
$ tar -xzf glide-v0.13.1-linux-amd64.tar.gz -C /opt
$ echo "export PATH=$PATH:/opt/linux-amd64/" >> $HOME/.bash_profile

Downloading the source code

1. create a directory in $HOME say webpa_modules
$ mkdir $HOME/webpa_modules && cd $HOME/webpa_modules

2. Checkout the components from GitHub repository.
$ git clone https://github.com/Comcast/talaria.git
$ git clone https://github.com/Comcast/scytale.git
$ git clone https://github.com/Comcast/tr1d1um.git

Building the components

1. Set the GOPATH & change to the source directory, e.g.
	$ export GOPATH=$HOME/webpa_modules/petasos/
	$ cd $HOME/webpa_modules/petasos/src/petasos

2. Resolve package dependencies using glide.
	$ glide install --strip-vendor

3. Build the component from source
	$ go build petasos

4. Create the package
	$ mkdir $HOME/rpmbuild
	$ ./build_rpm.sh --no-sign

5. Install the locally built webPA component package
e.g. $ cd /root/rpmbuild/RPMS/x86_64/
     $ rpm -Uvh petasos-0.1.1-87.el6.x86_64.rpm

	If the script terminates with "error: Bad owner/group: /root/webpa_modules/petasos/petasos.spec"

	change the ownership to match current user name

	$ chown root.root petasos.spec

Configuring the server components 

Prerequisite

Generating a auto token

WebPA server components as well as requesting application has to use a autorization token for bearer authentication. We can either use a basic authorization token or make use of a key server for obtaining a bearer token.

For example, a UI application needs to invoke some Preference setting or to obtain some diagnostics information on behalf of a MSO partner, deviceId, serviceAccountId or combination of the three. It will first obtain or use a pre-defined auth token, set it as a HTTP header and then invoke the GET/SET operation.

However in the standalone setup, we have used basic base64 encoded autherization token because SAT requires access to operator specific key servers. This auth token will be used when configuring different webPA components as well while performing GET/SET requests to the CPE from a 3rd party application.

We can use either of the below 2 methods to generate a basic authorization string. 

1. Using openssl command to generate the base64 encoded token.
[root@webpa-node1 ~]# openssl enc -base64 <<< "webpa@1234567890"
[OUTPUT] : d2VicGFAMTIzNDU2Nzg5MAo=
2. Using Linux coreutils tools to generate the base64 encoded token
[root@webpa-node1 ~]# echo "webpa@1234567890"|base64
[OUTPUT] : d2VicGFAMTIzNDU2Nzg5MAo=

Talaria configuration

Edit the configuration file & modify port number for running talaria service in a different port (default value is 8080).

{
        "port": 8080,
        "hcport": 8888,
        "pprofport": 9999,
        "discoveryClient": {
                "staticNodes": ["https://localhost:8585" ]
        },

        "log" : {
                "file"      : "talariaLog.log",
                "level"     : "DEBUG",
                "maxSize"   : 5242880,
                "maxBackup" : 3
        }
}

Scytale configuration

Edit the configuration file under /etc/scytale and modify following values

"fqdn"             : Fully qualified domain name of the server
"server"          : Listening IP address (using "localhost" will allow connections only from the current machine.)

"endpoints"    : Under "fanout" section, change the IP / Port value to match to the one where Talaria service is listening.

"authHeader" : Auth token Use the auth token which was generated in previous section

"file"                : Under "log" section, change the value from "stdout" to a file name if we need to redirect debug messages to a separate log file.

Add the "aws" section with following values for supressing few error messages 

   "aws": {
          "accessKey": "fake",
           "secretKey": "fake",
           "env": "fake",
           "sns": {
               "region": "us-east-1",
               "topicArn": "arn:aws:sns:us-east-1:999999999999:fake",
               "urlPath" : "/api/v2/aws/sns"
           }
    },

This will set AWS & SNS parameters with fake ones since we don't use actual keys and SNS (amazon simple notification service) in the current setup.

{
	"fqdn": "192.168.30.105",
	"server": "192.168.30.105",

	"primary": {
		"address": ":6000"
	},

	"health": {
		"address": ":6001"
	},

	"pprof": {
		"address": ":6002"
	},

	"fanout": {
		"method": "POST",
		"endpoints": ["http://192.168.30.105:8080/api/v2/device/send"],
		"authorization": "QWxhZGRpbjpPcGVuU2VzYW1l"
	},
	
	
	"log" : {
		"file"      : "stdout",
		"level"     : "DEBUG",
		"json": true
	},

	"aws": {
		"accessKey": "fake",
		"secretKey": "fake",
		"env": "fake",
		"sns": {
	                  "region": "us-east-1",
	                  "topicArn": "arn:aws:sns:us-east-1:999999999999:fake",
	                  "urlPath" : "/api/v2/aws/sns"
	        }
	},
	"authHeader": "d2VicGFAMTIzNDU2Nzg5MA=="
}

Tr1d1um configuration

Edit the configuration file from /etc/tr1d1um to set following parameters

"fqdn"             : Fully qualified domain name of server

"server"          : IP Address to which the service has to listen

"version"        : Current version of the service

"region"          : Region of deployment

"flavor"           : Development, Production etc.

"address"       : Under "primary" section, change the value to point to the port where tr1d1um service will listen for incoming requests.

"targetURL"   : Change to IP-Address:Port value where SCYTALE service is running.

"authHeader" : Auth token Use the auth token which was generated in previous section.

"aws"              : Add fake values as described previously.

{
	"fqdn": "192.168.30.105",

	"server": "192.168.30.105",
	"version": "0.1.1-228",
	"region": "india",
	"flavor": "lab", 
	
	"primary": {
		"address": ":6003"
	},

	"health": {
		"address": ":6004",
		"logInterval": "60s",
		"options": [
			"PayloadsOverZero",
			"PayloadsOverHundred",
			"PayloadsOverThousand",
			"PayloadsOverTenThousand"
		]
	},

	"pprof": {
		"address": ":6005"
	},
	
	"metrics": {
		"address": ":8082"
	},
	
	"log": {
		"file"      : "tr1d1um.log",
		"level"     : "DEBUG",
		"maxSize"   : 52428800,
		"maxBackup" : 10,
		"json"      : true
	},

	"aws": {
		"accessKey": "fake-accessKey",
		"secretKey": "fake-secretKey",
		"env": "fake-env",
		"sns": {
			"region": "fake-region",
			"topicArn": "fake-sns-topic",
			"urlPath" : "/api/v2/aws/sns"
		}
	},

	"targetURL": "http://192.168.30.105:6000",
	"supportedServices": ["config"],
	"authHeader": "d2VicGFAMTIzNDU2Nzg5MA=="
}

Enable the services at boot-up

Use system-config-services from Centos menu or console for enabling the talaria, scytale & tr1d1um services at system boot-up.

WebPA Client Setup

Parodus is the client-end service running on the RDK-V CPE devices which establishes a connection with webPA service on device boot-up and delivers request-response between the webPA server & CPE device services. Parodus provides following functionalities in a CPE device.

Websocket client: Nopoll library used as Websocket Client.  It allows building pure WebSocket solutions or to provide WebSocket support to existing TCP oriented applications. Nopoll handles all the messages coming from or to the server asynchronously.

Nanomsg Server: Parodus acts as Nanomsg server to distribute messages upstream and downstream.

Configuring Parodus

Edit parodus startup script for enabling the CPE device to use local webPA server

--webpa-url       :  Set with IP Address and Port of talaria service

--force-ipv4       : Force use of IPv4 for communication.

vi /lib/rdk/startParodus.sh
/bin/systemctl set-environment PARODUS_CMD=" --hw-mac=$HwMac --webpa-ping-time=$PingWaitTime --webpa-interface-used=$NwInterface --webpa-url=http://192.168.30.105 --partner-id=comcast --webpa-backoff-max=9 --force-ipv4 --ssl-cert-path=$SSL_CERT_FILE"

Restart the service after changes are done.

# systemctl restart parodus

Debugging & log files

Log files

server logs

• WebPA server logs are distributed among following locations.
  • /var/log/<webpa-service>/ : keeps debug log files.
    • • supervisord.log : Log messages related to service boot-up & initialization
      • console.out       : console logs (debug message will appear here if "file": "stdout" is configured in <webpa-service>.json file
  • /var/run/<webpa-service>/ : keeps service specific debug messages
    • <service-name>Log.log : component specific debug messages will appear here if "file" : "fileName.log" is configured in <webpa-service>.json file

Client logs

Parodus service log file is located as /opt/logs/parodus.log, provides debug information such as connection details, service initialization, which protocols are enabled/disabled etc.

Common Errors

Service fails to start

When we see an error similar to the below, it is related to zookeeper service failed to load or not running currently. restarting zookeeper & subsequent restart of other services solves the issue.

$ cat talariaLog.log
ts=2018-01-16T13:14:55.143587713Z caller=talaria.go:133 level=error msg="Unable to obtain service discovery instancer" error="zk: could not connect to a server"

Use cases

Downstream request

[External Application --> CPE Device] (e.g. Query from a dashboard application to request for how long has a specific router been up):

1. 1. 1. External service request goes to tr1d1um service
      2. tr1d1um forwards the request to Scytale API server cluster
      3. Scytale sends request to talaria server cluster
      4. Talaria then sends request to appropriate WebPA device

Upstream request

[CPE Device --> external service] (e.g. WebPA client notifications to external services, for instance, at boot time):

1. 1. 1. Request is sent through the same web socket connection that is established to Talaria
      2. Talaria sends request upstream to Scytale
      3. Scytale sends request to external service.

Testing the connection

Using Postman GUI application

TODO

Using console command

AUTH_TOKEN       : Basic base64 encoded auth token or SAT (if enabled).

WEBPA-URL          : URL of Tr1d1um service in IP:PORT format.

DEVICE_MAC        :  MAC address of the CPE device.

PARAMETER         : GET/SET Parameter that need to be requested.

$curl -H ''Authorization:Basic <AUTH_TOKEN>' -i http://<WEBPA-URL>/api/v2/device/mac:<DEVICE_MAC>/config?names=<PARAMETER>
e.g.
$curl -H 'Authorization:Basic d2VicGFAMTIzNDU2Nzg5MA==' -i http://192.168.30.105:6003/api/v2/device/mac:84e058575831/config?names=Device.DeviceInfo.FirmwareName

Common TR181 parameters