WebKit 2.38.1 results :
| Player – Asset Tests | Observation on BCM-R4 &wpe-2.38.1(04Dec-23) | BCM-feature-webkit 2.38.1-thunder42 (18-DEC-2023) | BCM-feature-webkit 2.38.1-thunder42 (19-DEC-2023) | | Shaka player |
|
|
| | #1:Big Buck Bunny [DASH, MP4 and WEBM) | Played | Played | Played | | #2:Big Buck Bunny [HLS & MP4 ) | Played | Played | Played | | #3: Angel One(multicodec, multilingual) | Played | Played | Played | | #4: Angel One( Widevine, multicodec, multilingual) | Played | Played | Played | | #110: Dash - Clear (2s segments) | Played | Played | Played | | #111: Hls - Clear (Fragmented MP4 & 2s segments) | Not Played | Not played | Not played | | #112: Dash Playready | Not Played | Not played | Played | | #113: HLS - Multi-Track (HLS, Clear & Multi-Track) | Played | Played | Played | | #114: MP4 - Clear (MP4 only) | Played | Played | Played | | DashJS player |
|
|
| | #1:Big Buck Bunny [DASH, MP4 and WEBM) | Played | Played | Played | | #3: Angel One(multicodec, multilingual) | Played | Played | Played | | #4: Angel One( Widevine, multicodec, multilingual) | Played | Played | Played | | #110: Dash - Clear (2s segments) | Played | Played | Played | | #112: Dash Playready | Played | Not played (Blank screen) | Played | | HLSJS player |
|
|
| | #2:Big Buck Bunny [HLS & MP4 ) | Not Played | Not played | Not played | | #111: Hls - Clear (Fragmented MP4 & 2s segments) | Not Played | Not played | Not played | | #113: HLS - Multi-Track (HLS, Clear & Multi-Track) | Not Played | Not played | Not played | | AAMP |
|
|
| | #1:Big Buck Bunny [DASH, MP4 and WEBM) | Played | Played | Played | | #2:Big Buck Bunny [HLS & MP4 ) | Played | Played | Played | | #3: Angel One(multicodec, multilingual) | Played | Played | Played | | #110: Dash - Clear (2s segments) | Played | Played | Played | | #111: Hls - Clear (Fragmented MP4 & 2s segments) | Played | Played | Played | | #113: HLS - Multi-Track (HLS, Clear & Multi-Track) | Played | Played | Played | | #114: MP4 - Clear (MP4 only) | Played | Not played | Not played | | HTML5 |
|
|
| | #1:Big Buck Bunny [DASH, MP4 and WEBM) | Played | Played | Played | | #2:Big Buck Bunny [HLS & MP4 ) | Played | Played | Played | | #3: Angel One(multicodec, multilingual) | Played | Played | Played | | #110: Dash - Clear (2s segments) | Played | Played | Played | | #111: Hls - Clear (Fragmented MP4 & 2s segments) | Played | Played | Played | | #113: HLS - Multi-Track (HLS, Clear & Multi-Track) | Played | Played | Played | | #114: MP4 - Clear (MP4 only) | Not Played | Not played | Not played |
|
Note : > Tested the Features video row on the media player
> Progress bar is not moving & Player Stats is blank for HTML 5 media player videos.
> Trick-mode doesn't works in HTML 5
Apps Testing:
| Apps | BCM-R4-Wpe-2.38.1(04Dec-23) | BCM-feature-webkit 2.38.1-thunder42 (18-DEC-2023) | SGM-feature-webkit 2.38.1-thunder42 (19DEC-2023) |
| Amazon Prime Video | N/A | N/A | N/A |
| YouTube | Working | Working | Working |
| Peacock | N/A | N/A | N/A |
| Xumo | Working | Working | Working |
| CNN | Working | Working | Working |
| VimeoRelease | N/A | N/A | N/A |
| WeatherNetwork | Working | Blank | Blank |
| EuroNews | Working | Working | Working |
| AccuWeather | Working | Working | Working |
| Aljazeera | Working | Working | Working |
| GuessThatCity | Working | Working | Working |
| Radioline | Working | Working | Working |
| WallStreetJournal | Working | Not working (error) | Not working (error) |
| FRacer | Working | Working | Working |
| Aquarium | Working | Working | Working |
| Fireplace | Working | Working | Working |
| Deutsche Welle | Working | Working | Working |
| MyTuner Radio | Working | Working | Working |
| Sudoku | Working | Working | Working |
| Tastemade | Working | Working | Working |
| Bloomberg | Working | Working | Working |
| Playworks | Working | Working | Working |
| Sunrise | Working | Working | Working |
Featured VOD content apps playback is working Fine (BCM-R4-2.38.1)- Tested on Dec 04.
WEBAUDIO TEST RESULTS
https://webbrowsertools.com/audio-test/
Verified on webkit Browser plug-in. Followed the steps in 
Observation on BCM-feature-webkit 2.38.1-thunder42 - Not Working
Observation on SGM-feature-webkit 2.38.1-thunder42 - Not Working
TDK Test scripts for Validating WPE Browser
- RDKService WebKitBrowser Plugin APIs
The purpose of this test suite is to validate RDKService APIs and events. The validation process involves using a custom XML-based test case and a Python framework that is integrated into the TDK (Test Development Kit) tool. The main validation criteria include checking the correctness of the API and event responses, as well as analyzing logs generated by the device.
Script Used: RDKV_CERT_AVS_WebKit_Browser
WebKitBrowser test suite performs set and get operations to validate features like browser visibility, browser FPS, cookie accept policy, preferred language etc, in order to validate this test cases user needs to configure the webkit_test_url,webkit_test_url_1 and webkitbrowser playback url in the corresponding configuration file.
- Browser Performance Tests
The Browser performance test suite is used for getting the browser score using different test tools like CSS3, Octane etc. The browser test URL will be loaded in to device using the RDKService apis and fetch the score with the help of web inspect page. The device must provide the web inspect page for the device browser, to perform this suite. The framework is internally using the selenium with Python for parsing the data from browser.
Scripts used:
RDKV_CERT_PVS_Browser_HTML5
The HTML5 test score is an indication of how well your browser supports the HTML5 standard and related specifications. It also gives individual scores for tests like video/audio/streaming etc.
RDKV_CERT_PVS_Browser_CSS3
CSS3 test offers a quick and easy way to test and compare browser support for CSS3. The test checks for almost everything which includes Backgrounds and Borders/Image Values and Replaced Content/Selectors/Media Queries/Basic User Interface,Transitions/Animations etc.
RDKV_CERT_PVS_Browser_Octane
Octane 2.0 is a benchmark that measures a JavaScript engine’s performance by running a suite of tests representative of certain use cases in JavaScript applications. Octane 2.0 consists of 17 tests which includes OS kernel simulation benchmark/DeltaBlue/Raytrace etc.
RDKV_CERT_PVS_Browser_SunSpider
SunSpider is a benchmark created by the webkit team that aims to measure JavaScript performance on tasks relevant to the current and near future use of JavaScript in the real world, such as encryption and text manipulation. The suite further attempts to be balanced and statistically sound. The sunspider Url is not deprecated. So marked this script as NA.
RDKV_CERT_PVS_Browser_Strike_720_resolution
RDKV_CERT_PVS_Browser_Strike_1080_resolution
Strike is a benchmarking tool that runs on a web browser and performs the following tasks:- Measure to what extent your software stack is compatible with running Lightning applications.
- Measure how well your software and hardware stack can perform Lightning applications.
- Test the graphical capabilities of your software and hardware configuration.
- Measure the capabilities of the web browser in which Strike is executed.
- Provide a total score to your software and hardware configuration.
RDKV_CERT_PVS_Browser_Animation_FPS
Animation benchmark test will load bubbles animation to browser and it will display the FPS value based on browser capability, The script will take the FPS value displayed.
RDKV_CERT_PVS_Browser_Speedometer
Speedometer is a browser benchmark that measures the responsiveness of Web applications. It uses demo web applications to simulate user actions such as adding to-do items.
RDKV_CERT_PVS_Browser_MotionMark
MotionMark is a graphics benchmark that measures a browser’s capability to animate complex scenes at a target frame rate.
RDKV_CERT_PVS_Browser_SmashCat
Smash Cat is Canvas Performance Test. No external libs used. This does NOT use WebGL
RDKV_CERT_PVS_Browser_Kraken
Kraken is a JavaScript performance benchmark created by Mozilla that measures the speed of several different test cases extracted from real-world applications and libraries.
- Robustness Certification Tests
Ensuring device stability and consistently reliable user experiences is crucial. To verify this, it is essential to prevent any crashes or stability issues.
To accomplish this, the test suite includes various stress test scenarios like multiple reboots, stress testing of native/web applications, graphical plugins, and life cycle management of different RDK services. The primary validation mechanisms employed in the test suite involve the RDK Service API and event responses, wpeframework logs, and Lightning app logs. The test also monitors device CPU and memory usage during testing.
Scripts used:
RDKV_CERT_RVS_LightningApp_LifeCycleManagement
RDKV_CERT_RVS_ResidentApp_LifeCycleManagement
RDKV_CERT_RVS_WebKitBrowser_LifeCycleManagement
RDKV_CERT_RVS_HtmlApp_LifeCycleManagement
These scripts will execute life cycle methods like launch, set URL, get URL, suspend, resume, moveToBack, moveToFront for LightningApp/ResidentApp/WebKitBrowser/HtmlApp plugin for a minimum of 1000 times.
RDKV_CERT_RVS_LightningApp_LoadURL_OnStressLaunch
RDKV_CERT_RVS_WebKitBrowser_LoadURL_OnStressLaunch
RDKV_CERT_RVS_HtmlApp_LoadURL_OnStressLaunch
These scripts will Launch and exit LightningApp/webKitBrowser/HtmlApp for 99 times. On 100th time it will load LightningApp/WebKitBrowser/HtmlApp and will set a URL and verify whether the URL is set properly.
RDKV_CERT_RVS_LightningApp_SetURLAndExit
RDKV_CERT_RVS_WebKitBrowser_SetURLAndExit
RDKV_CERT_RVS_HtmlApp_SetURLAndExit
These scripts will launch LightningApp/WebKitBrowser/HtmlApp set a URL and will destroy the plugin for a minimum of 1000 times. In each iteration system resource usage will be validated.
RDKV_CERT_RVS_LightningApp_SuspendResume
RDKV_CERT_RVS_ResidentApp_SuspendResume
RDKV_CERT_RVS_WebKitBrowser_SuspendResume
RDKV_CERT_RVS_HtmlApp_SuspendResume
These scripts will suspend and resume ResidentApp/LighningApp/WebKitBrowser/HtmlApp plugin for a minimum of 1000 times.
RDKV_CERT_RVS_ResidentApp_ActivateDeactivate
The script will activate and deactivate Resident App using Resident plugin for a minimum or 1000 times. This is to test the robustness of the plugin and DUT as well.
RDKV_CERT_RVS_ResidentApp_Navigation
The script will continuously navigate through resident app UI and validate the stability of device.
RDKV_CERT_RVS_WebKitBrowser_SetURL
The script will set 2 different URLs back to back for minimum 100 times. This will ensure the stability of device browser.
RDKV_CERT_RVS_WebKitBrowser_KeyStress
The script will do key stress test on WebKitBrowser by loading a Lightning app for detecting key press. Script will pass 10 keys to the app for a minimum of 100 times.
RDKV_CERT_RVS_WebKitBrowser_Load_GraphicsApp
The script will set a graphics/animation application in WebKitBrowser and will monitor the resource usage for 6 hours.
RDKV_CERT_RVS_WebKitBrowser_LaunchAndDestroy
The script will continuously launch and destroy WebKitbrowser for a minimum of 1000 iterations and it will validate the stability and resource usage in each iteration.
- Functionality Performance Tests
This suite mainly covers the performance of the device. The script utilizes the TDK frameworks for performing different operations in device. One of the main validation mechanism used is logs from the device.
Scripts used:
RDKV_CERT_PVS_Functional_ResidentApp_LifeCycleManagement
RDKV_CERT_PVS_Functional_WebKitBrowser_LifeCycleManagement
RDKV_CERT_PVS_Functional_LightningApp_LifeCycleManagement
RDKV_CERT_PVS_Functional_HtmlApp_LifeCycleManagement
These scripts will execute life cycle methods like launch, get URL, suspend, resume, moveToBack, moveToFront for ResidentApp/WebKitBrowser/LightningApp/HtmlApp plugin for a minumum of 1000 times.
RDKV_CERT_PVS_Functional_WebKitBrowser_TimeTo_LoadURL
The script will calculate the time taken to load a given URL in WebKitBrowser. This is to get the performance data of the plugin. The script makes use of event logs to fetch the timestamp.
RDKV_CERT_PVS_Functional_ResidentApp_TimeTo_Launch
RDKV_CERT_PVS_Functional_LightningApp_TimeTo_Launch
RDKV_CERT_PVS_Functional_HtmlApp_TimeTo_Launch
These scripts will validate the time taken to launch ResidentApp/LightningApp/HtmlApp plugin using the logs from wpeframework log.
RDKV_CERT_PVS_Functional_WebKitBrowser_ResourceUsage_OnLoadURL
RDKV_CERT_PVS_Functional_LightningApp_ResourceUsage_onLoadURL
RDKV_CERT_PVS_Functional_HtmlApp_ResourceUsage_onLoadURL
These scripts will launch a URL in WebKitBrowser/LightningApp/HtmlApp and it will validate the resource usage after setting the URL.
RDKV_CERT_PVS_Functional_WebKitBrowser_TimeTo_ActivateDeactivate
The script will calculate the time taken to activate the plugin as well as time taken to deactivate the plugin. This is to get the performance data of the plugin. The script makes use of event logs to fetch the timestamp.
RDKV_CERT_PVS_Functional_WebKitBrowser_TimeTo_SuspendResume
RDKV_CERT_PVS_Functional_LightningApp_TimeTo_SuspendResume
RDKV_CERT_PVS_Functional_HtmlApp_TimeTo_SuspendResume
These scripts will calculate the time taken to suspend the plugin as well as time taken to resume the plugin. This is to get the performance data of the plugin. The script makes use of event logs to fetch the timestamp.
RDKV_CERT_PVS_Functional_TimeTo_LaunchWebKitBrowser_AfterReboot
The script will reboot the DUT. Once the device is online it will launch WebKitBrowser and will validate the time taken to launch WebKitBrowser.
RDKV_CERT_PVS_Functional_WebKitBrowser_Reboot_OnLoadURL
RDKV_CERT_PVS_Functional_LightningApp_Reboot_OnLoadURL
RDKV_CERT_PVS_Functional_HtmlApp_Reboot_OnLoadURL
These scripts will load a URL in WebKitBrowser/LightningApp/HtmlApp then it will immediately reboot the device and it will try to load the URL again.
RDKV_CERT_PVS_Functional_WebKitBrowser_Destroy_OnLoadURL
RDKV_CERT_PVS_Functional_LightningApp_Destroy_OnLoadURL
RDKV_CERT_PVS_Functional_HtmlApp_Destroy_OnLoadURL
These scripts will load a URL in WebKitBrowser/LightningApp/HtmlApp then it will immediately destroy the plugin and it will try to load the URL again
RDKV_CERT_PVS_Functional_LightningApp_ResourceUsage_Launch
RDKV_CERT_PVS_Functional_HtmlApp_ResourceUsage_Launch
These scripts will validate the system resource usage while launching LightningApp/HtmlApp plugin.
RDKV_CERT_PVS_Functional_LightningApp_TimeTo_Destroy
RDKV_CERT_PVS_Functional_HtmlApp_TimeTo_Destroy
These scripts will validate the time taken to destroy LightningApp/HtmlApp plugin based on threshold value.
References:
UVE/QAM feature in RPi Validation tests for WPE browser on Sagemcom and Broadcom
HTML Spec Compliance: (23-Nov-23)
https://html5test.com/
WebKit 2.38.1 result : Url Not loaded
HTML Spec Compliance: (18-DEC-23)
https://html5test.com/
WebKit 2.38.1 result : Obtained score as 409 out of 555 when loaded in HtmlApp - BCM-feature-webkit 2.38.1-thunder42
WebKit 2.38.1 result : Url Not Loaded - SGM-feature-webkit 2.38.1-thunder42
- Functional Tests
- MSE
- EME
- Format Support Tests
WebKit 2.38.1 results :
| Tests | Observation on BCM-R4 &wpe-2.38.1(04 DEC-23) | BCM-feature-webkit 2.38.1-thunder42 (18-DEC-2023) | SGM-feature-webkit 2.38.1-thunder42 (19-DEC-2023) | | Functional Tests | 1-22 Failed
47-48 Failed
64-65 Failed
68-92 Failed
94 Failed
96 Failed
98-99 Failed | 1-22 Failed
47-48 Failed
64-66 Failed
68-93 Failed
95 Failed
97 Failed
99-100 Failed
104 Failed | 1-22 Failed
47-48 Failed
64-66 Failed
68-93 Failed
95 Failed
97 Failed
99-100 Failed
104 Failed | | MSE Codec tests | 1-2 Failed
52- Failed
76- Failed
100-Failed | 1-2 Failed
52 Failed
76 Failed
100 Failed | 1-2 Failed
52 Failed
76 Failed
100 Failed | | MSE Conformance Tests | 30-37 Failed
42-43 Failed | 30-38 Failed
41 Failed | 30-38 Failed | | EME Conformance Tests | 13-16 Failed
24-25 Failed
34-35 Failed
39-40 Failed
41-84 Failed | 1 Failed
3 - 10 Failed
13 - 18 Failed
20 - 22 Failed
24 - 25 Failed
29 Failed
31 - 32 Failed
34 - 84 Failed
| 1 Failed
3 - 10 Failed
13 - 18 Failed
20 - 22 Failed
24 - 25 Failed
29 Failed
31 - 32 Failed
34 - 84 Failed | | Format Support Tests | 3 Failed
7-13 Failed
40 Failed
72 Failed
99 Failed | 3 Failed
7-13 Failed
40 Failed
56 Failed
72 Failed
99 Failed
111 Failed | 3 Failed
7-13 Failed
40 Failed
56 Failed
72 Failed
99 Failed
111 Failed |
|
Observations: