WebKit 2.38.1 results :

   Procedure :Enable WebkitBrowser Plugin from the controller UI <deviceIP:9998> and enter the above URL and tested the videos

Player – Asset TestsObservation on SGM-R2 &wpe-2.38.1(18Dec23-CMF NG Build)Observation on BCM-R2 &wpe-2.38.1(16Dec23- CMF NG Build))Observation on BCM-R4 &wpe-2.38.1(14Dec23-Local Build)Observation on SGM-R4 &wpe-2.38.1(15Dec23-Local Build)
Shaka player



#1:Big Buck Bunny [DASH, MP4 and WEBM)

PlayedPlayedPlayedPlayed
#2:Big Buck Bunny [HLS & MP4 ) PlayedPlayedPlayedPlayed
#3: Angel One(multicodec, multilingual) PlayedPlayedPlayedPlayed
#4: Angel One( Widevine, multicodec, multilingual) PlayedDid Not PlayPlayedPlayed
#110: Dash - Clear (2s segments)PlayedPlayedPlayedPlayed
#111: Hls - Clear (Fragmented MP4 & 2s segments)Did Not PlayDid Not PlayDid Not PlayDid Not Play
#112: Dash PlayreadyDid Not PlayDid Not PlayDid Not PlayDid Not Play
#113: HLS - Multi-Track (HLS, Clear & Multi-Track)PlayedPlayedPlayedPlayed
#114: MP4 - Clear (MP4 only)PlayedPlayedPlayedPlayed
DashJS player



#1:Big Buck Bunny [DASH, MP4 and WEBM)PlayedPlayedPlayedPlayed
#3: Angel One(multicodec, multilingual) PlayedPlayedPlayedPlayed
#4: Angel One( Widevine, multicodec, multilingual) Did Not PlayDid Not PlayPlayedDid Not Play
#110: Dash - Clear (2s segments)PlayedPlayedPlayedPlayed
#112: Dash PlayreadyDid Not PlayDid Not PlayDid Not Play Did Not Play
HLSJS player



#2:Big Buck Bunny [HLS & MP4 ) Did Not PlayDid Not PlayDid Not PlayDid Not Play
#111: Hls - Clear (Fragmented MP4 & 2s segments)Did Not PlayDid Not PlayDid Not PlayDid Not Play
#113: HLS - Multi-Track (HLS, Clear & Multi-Track)Did Not PlayDid Not PlayDid Not PlayDid Not Play
AAMP



#1:Big Buck Bunny [DASH, MP4 and WEBM)PlayedPlayedPlayedPlayed
#2:Big Buck Bunny [HLS & MP4 ) PlayedPlayedPlayedPlayed
#3: Angel One(multicodec, multilingual) PlayedPlayedPlayedPlayed
#110: Dash - Clear (2s segments)PlayedPlayedPlayedPlayed
#111: Hls - Clear (Fragmented MP4 & 2s segments)PlayedPlayedPlayedPlayed
#113: HLS - Multi-Track (HLS, Clear & Multi-Track)PlayedPlayedPlayedPlayed
#114: MP4 - Clear (MP4 only)Played (video zoomed)PlayedPlayed

Played

HTML5



#1:Big Buck Bunny [DASH, MP4 and WEBM)PlayedPlayedPlayedPlayed
#2:Big Buck Bunny [HLS & MP4 ) PlayedPlayedPlayedPlayed
#3: Angel One(multicodec, multilingual) PlayedPlayedPlayedPlayed
#110: Dash - Clear (2s segments)PlayedPlayedPlayedPlayed
#111: Hls - Clear (Fragmented MP4 & 2s segments)Did not playPlayedPlayedPlayed
#113: HLS - Multi-Track (HLS, Clear & Multi-Track)PlayedPlayedPlayedPlayed
#114: MP4 - Clear (MP4 only)PlayedPlayedPlayedPlayed

Note : > Tested the Features video row on the media player

> Progress bar is not moving & Player Stats is blank for HTML5 media player videos.

> Trickmode doesn't works in HTML5


Apps Testing:

AppsSGM-R2-Wpe-2.38.1(18Dec23-CMF NG Build)Observation on BCM-R2 &wpe-2.38.1(16Dec23-CMF NG Build)Observation on BCM-R4 &wpe-2.38.1(14Dec23-Local Build)Observation on SGM-R4 &wpe-2.38.1(15Dec23-Local build)
Amazon Prime VideoNAN/AN/AN/A
YouTubeWorkingWorkingWorkingWorking
PeacockNAN/AN/AN/A
XumoWorkingWorkingWorkingWorking
CNNWorkingWorkingWorkingWorking
WeatherNetworkBlankBlankBlankBlank
EuroNewsWorkingWorkingWorkingWorking
AccuWeatherWorkingWorkingWorkingWorking
AljazeeraWorkingWorkingWorkingWorking
GuessThatCityWorkingWorkingWorkingWorking
RadiolineWorkingWorkingWorkingWorking
WallStreetJournalWorkingWorkingWorkingWorking
FRacerWorkingWorkingWorkingWorking
AquariumWorkingWorkingWorkingWorking
FireplaceWorkingWorkingWorkingWorking
Deutsche WelleWorkingWorkingWorkingWorking
MyTuner RadioWorkingWorkingWorkingWorking
SudokuWorkingWorkingWorkingWorking
TastemadeWorkingWorkingWorkingWorking
BloombergWorkingWorkingWorkingWorking
PlayworksWorkingWorkingWorkingWorking
SunriseWorkingWorkingWorkingWorking

WEBAUDIO TEST RESULTS

https://webbrowsertools.com/audio-test/

Verified on webkit Browser plug-in. Followed the steps in BCM72180-821 - Getting issue details... STATUS

Observation on BCM-R4 &wpe-2.38.1 - Not working

Observation on SGM-R4 &wpe-2.38.1 - Not working

TDK Test scripts for Validating WPE Browser

  • RDKService WebKitBrowser Plugin APIs
    The purpose of this test suite is to validate RDKService APIs and events. The validation process involves using a custom XML-based test case and a Python framework that is integrated into the TDK (Test Development Kit) tool. The main validation criteria include checking the correctness of the API and event responses, as well as analyzing logs generated by the device.
    Script Used: RDKV_CERT_AVS_WebKit_Browser
    WebKitBrowser test suite performs set and get operations to validate features like browser visibility, browser FPS, cookie accept policy, preferred language etc, in order to validate this test cases user needs to configure the webkit_test_url,webkit_test_url_1 and webkitbrowser playback url in the corresponding configuration file.

  • Browser Performance Tests
    The Browser performance test suite is used for getting the browser score using different test tools like CSS3, Octane etc. The browser test URL will be loaded in to device using the RDKService apis and fetch the score with the help of web inspect page. The device must provide the web inspect page for the device browser, to perform this suite. The framework is internally using the selenium with Python for parsing the data from browser.
    Scripts used:
    RDKV_CERT_PVS_Browser_HTML5
    The HTML5 test score is an indication of how well your browser supports the HTML5 standard and related specifications. It also gives individual scores for tests like video/audio/streaming etc.

    RDKV_CERT_PVS_Browser_CSS3
    CSS3 test offers a quick and easy way to test and compare browser support for CSS3. The test checks for almost everything which includes Backgrounds and Borders/Image Values and Replaced Content/Selectors/Media Queries/Basic User Interface,Transitions/Animations etc.

    RDKV_CERT_PVS_Browser_Octane
    Octane 2.0 is a benchmark that measures a JavaScript engine’s performance by running a suite of tests representative of certain use cases in JavaScript applications. Octane 2.0 consists of 17 tests which includes OS kernel simulation benchmark/DeltaBlue/Raytrace etc.

    RDKV_CERT_PVS_Browser_SunSpider
    SunSpider is a benchmark created by the webkit team that aims to measure JavaScript performance on tasks relevant to the current and near future use of JavaScript in the real world, such as encryption and text manipulation. The suite further attempts to be balanced and statistically sound. The sunspider Url is not deprecated. So marked this script as NA.

    RDKV_CERT_PVS_Browser_Strike_720_resolution
    RDKV_CERT_PVS_Browser_Strike_1080_resolution 
    Strike is a benchmarking tool that runs on a web browser and performs the following tasks:
    • Measure to what extent your software stack is compatible with running Lightning applications.
    • Measure how well your software and hardware stack can perform Lightning applications.
    • Test the graphical capabilities of your software and hardware configuration.
    • Measure the capabilities of the web browser in which Strike is executed.
    • Provide a total score to your software and hardware configuration.

    RDKV_CERT_PVS_Browser_Animation_FPS
    Animation benchmark test will load bubbles animation to browser and it will display the FPS value based on browser capability, The script will take the FPS value displayed.

    RDKV_CERT_PVS_Browser_Speedometer
    Speedometer is a browser benchmark that measures the responsiveness of Web applications. It uses demo web applications to simulate user actions such as adding to-do items.

    RDKV_CERT_PVS_Browser_MotionMark
    MotionMark is a graphics benchmark that measures a browser’s capability to animate complex scenes at a target frame rate.

    RDKV_CERT_PVS_Browser_SmashCat
    Smash Cat is Canvas Performance Test. No external libs used. This does NOT use WebGL

    RDKV_CERT_PVS_Browser_Kraken
    Kraken is a JavaScript performance benchmark created by Mozilla that measures the speed of several different test cases extracted from real-world applications and libraries.

  • Robustness Certification Tests
    Ensuring device stability and consistently reliable user experiences is crucial. To verify this, it is essential to prevent any crashes or stability issues. 
    To accomplish this, the test suite includes various stress test scenarios like multiple reboots, stress testing of native/web applications, graphical plugins, and life cycle management of different RDK services. The primary validation mechanisms employed in the test suite involve the RDK Service API and event responses, wpeframework logs, and Lightning app logs. The test also monitors device CPU and memory usage during testing.

    Scripts used:

    RDKV_CERT_RVS_LightningApp_LifeCycleManagement
    RDKV_CERT_RVS_ResidentApp_LifeCycleManagement
    RDKV_CERT_RVS_WebKitBrowser_LifeCycleManagement
    RDKV_CERT_RVS_HtmlApp_LifeCycleManagement

    These scripts will execute life cycle methods like launch, set URL, get URL, suspend, resume, moveToBack, moveToFront for LightningApp/ResidentApp/WebKitBrowser/HtmlApp plugin for a minimum of 1000 times.

    RDKV_CERT_RVS_LightningApp_LoadURL_OnStressLaunch
    RDKV_CERT_RVS_WebKitBrowser_LoadURL_OnStressLaunch
    RDKV_CERT_RVS_HtmlApp_LoadURL_OnStressLaunch
    These scripts will Launch and exit LightningApp/webKitBrowser/HtmlApp for 99 times. On 100th  time it will load LightningApp/WebKitBrowser/HtmlApp and will set a URL and verify whether the URL is set properly.

    RDKV_CERT_RVS_LightningApp_SetURLAndExit
    RDKV_CERT_RVS_WebKitBrowser_SetURLAndExit
    RDKV_CERT_RVS_HtmlApp_SetURLAndExit

    These scripts will launch LightningApp/WebKitBrowser/HtmlApp set a URL and will destroy the plugin for a minimum of 1000 times. In each iteration system resource usage will be validated.

    RDKV_CERT_RVS_LightningApp_SuspendResume
    RDKV_CERT_RVS_ResidentApp_SuspendResume
    RDKV_CERT_RVS_WebKitBrowser_SuspendResume
    RDKV_CERT_RVS_HtmlApp_SuspendResume
    These scripts will suspend and resume ResidentApp/LighningApp/WebKitBrowser/HtmlApp plugin for a minimum of 1000 times.

    RDKV_CERT_RVS_ResidentApp_ActivateDeactivate
    The script will activate and deactivate  Resident App using Resident plugin for a minimum or 1000 times. This is to test the robustness of the plugin and DUT as well.

    RDKV_CERT_RVS_ResidentApp_Navigation
    The script will continuously navigate through resident app UI and validate the stability of device.

    RDKV_CERT_RVS_WebKitBrowser_SetURL
    The script will set 2 different URLs back to back for minimum 100 times. This will ensure the stability of device browser.

    RDKV_CERT_RVS_WebKitBrowser_KeyStress
    The script will do key stress test on WebKitBrowser by loading a Lightning app for detecting key press. Script will pass 10 keys to the app for a minimum of 100 times.

    RDKV_CERT_RVS_WebKitBrowser_Load_GraphicsApp
    The script will set a graphics/animation application in WebKitBrowser and will monitor the resource usage for 6 hours. 

    RDKV_CERT_RVS_WebKitBrowser_LaunchAndDestroy
    The script will continuously launch and destroy WebKitbrowser for a minimum of 1000 iterations and it will validate the stability and resource usage in each iteration.


  • Functionality Performance Tests
    This suite mainly covers the performance of the device. The script utilizes the TDK frameworks for performing different operations in device. One of the main validation mechanism used is logs from the device.
    Scripts used:
    RDKV_CERT_PVS_Functional_ResidentApp_LifeCycleManagement
    RDKV_CERT_PVS_Functional_WebKitBrowser_LifeCycleManagement
    RDKV_CERT_PVS_Functional_LightningApp_LifeCycleManagement
    RDKV_CERT_PVS_Functional_HtmlApp_LifeCycleManagement
    These scripts will execute life cycle methods like launch, get URL, suspend, resume, moveToBack, moveToFront for ResidentApp/WebKitBrowser/LightningApp/HtmlApp plugin for a minumum of 1000 times.

    RDKV_CERT_PVS_Functional_WebKitBrowser_TimeTo_LoadURL
    The script will calculate the time taken to load a given URL in WebKitBrowser. This is to get the performance data of the plugin. The script makes use of event logs to fetch the timestamp.

    RDKV_CERT_PVS_Functional_ResidentApp_TimeTo_Launch
    RDKV_CERT_PVS_Functional_LightningApp_TimeTo_Launch
    RDKV_CERT_PVS_Functional_HtmlApp_TimeTo_Launch
    These scripts will validate the time taken to launch ResidentApp/LightningApp/HtmlApp plugin using the logs from wpeframework log.

    RDKV_CERT_PVS_Functional_WebKitBrowser_ResourceUsage_OnLoadURL
    RDKV_CERT_PVS_Functional_LightningApp_ResourceUsage_onLoadURL
    RDKV_CERT_PVS_Functional_HtmlApp_ResourceUsage_onLoadURL
    These scripts will launch a URL in WebKitBrowser/LightningApp/HtmlApp and it will validate the resource usage after setting the URL. 

    RDKV_CERT_PVS_Functional_WebKitBrowser_TimeTo_ActivateDeactivate
    The script will calculate the time taken to activate the plugin as well as time taken to deactivate the plugin. This is to get the performance data of the plugin. The script makes use of event logs to fetch the timestamp.

    RDKV_CERT_PVS_Functional_WebKitBrowser_TimeTo_SuspendResume
    RDKV_CERT_PVS_Functional_LightningApp_TimeTo_SuspendResume
    RDKV_CERT_PVS_Functional_HtmlApp_TimeTo_SuspendResume
    These scripts will calculate the time taken to suspend the plugin as well as time taken to resume the plugin. This is to get the performance data of the plugin. The script makes use of event logs to fetch the timestamp.

    RDKV_CERT_PVS_Functional_TimeTo_LaunchWebKitBrowser_AfterReboot
    The script will reboot the DUT. Once the device is online it will launch WebKitBrowser and will validate the time taken to launch WebKitBrowser. 

    RDKV_CERT_PVS_Functional_WebKitBrowser_Reboot_OnLoadURL
    RDKV_CERT_PVS_Functional_LightningApp_Reboot_OnLoadURL
    RDKV_CERT_PVS_Functional_HtmlApp_Reboot_OnLoadURL
    These scripts will load a URL in WebKitBrowser/LightningApp/HtmlApp then it will immediately reboot the device and it will try to load the URL again.

    RDKV_CERT_PVS_Functional_WebKitBrowser_Destroy_OnLoadURL
    RDKV_CERT_PVS_Functional_LightningApp_Destroy_OnLoadURL
    RDKV_CERT_PVS_Functional_HtmlApp_Destroy_OnLoadURL
    These scripts will load a URL in WebKitBrowser/LightningApp/HtmlApp then it will immediately destroy the plugin and it will try to load the URL again

    RDKV_CERT_PVS_Functional_LightningApp_ResourceUsage_Launch
    RDKV_CERT_PVS_Functional_HtmlApp_ResourceUsage_Launch
    These scripts will validate the system resource usage while launching LightningApp/HtmlApp plugin.

    RDKV_CERT_PVS_Functional_LightningApp_TimeTo_Destroy
    RDKV_CERT_PVS_Functional_HtmlApp_TimeTo_Destroy
    These scripts will validate the time taken to destroy LightningApp/HtmlApp plugin based on threshold value.

References:


  • UVE/QAM feature in RPi Validation tests for WPE browser on Sagemcom and Broadcom


    HTML Spec Compliance:
    https://html5test.com/ 

    WebKit 2.38 result : URL is loading but Results are not Displayed - BCM R2_WPE-2.38.1-16 DEC 2023

    WebKit 2.38 result :  Obtained score as 409 out of 555 when loaded in HtmlApp - SGM_R2_WPE-2.38.1- 18 DEC 2023


           HTML Spec Compliance:
           https://html5test.com/

           WebKit 2.38 result :  Obtained score as 409 out of 555 when loaded in HtmlApp - BCM-R4_WPE-2.38.1- 14 DEC 2023

            WebKit 2.38 result :  Obtained score as 409 out of 555 when loaded in HtmlApp - SGM-R4_WPE-2.38.1- 15 DEC 2023

          

YT Automated test suite:  https://ytlr-cert.appspot.com/latest/main.html#1652453525576

Specifically below categories.

  1. Functional Tests
  2. MSE
  3. EME
  4. Format Support Tests


WebKit 2.38 results :

Oops, it seems that you need to place a table or a macro generating a table within the Table Filter macro.

TestsObservation on SGM-R2 &wpe-2.38.1(18DEC23-CMF NG build)Observation on BCM-R2 &wpe-2.38.1(16Dec23- CMF NG Build)Observation on BCM-R4 &wpe-2.38.1(14Dec23 - Local Build)Observation on SGM-R4 &wpe-2.38.1(15Dec23-Local Build)
Functional Tests1-22 failed
47-48 failed
64-66 failed
68-88 failed
90 failed
94, 96 failed
98-99 failed
1-22 failed
47-48 failed
64-66 failed
68-93 failed
95 failed
97 failed
99-100 failed
104 failed
1-22 failed
47-48 failed
64-66 failed
68-93 failed
95 failed
97 failed
99-100 failed
104 failed
1-22 failed
47-48 failed
64-66 failed
68-93 failed
95 failed
97 failed
99-100 failed
MSE Codec tests1-2 failed
18-20 failed
38 failed
52 failed
76 failed
100 failed
1-2 failed
18-20 failed
38 failed
52 failed
76 failed
100 failed
1-2 failed
18-20 failed
38 failed
52 failed
76 failed
100 failed
1-2 failed
18-20 failed
38 failed
52 failed
76 failed
100 failed
MSE Conformance Tests30-38 Failed30-38 Failed
30-38 Failed
41 Failed
30-38 Failed
41 Failed
EME Conformance Tests
13-16 failed
24, 25 failed
27-29 failed
31-32 failed
34,35 failed
39-84 Failed
1 failed
3-10 failed
13-18 failed
20-22 failed
24, 25 failed
27-29 failed
31-32 failed
34-84 failed
13-16 failed
24-25 failed
27-29 failed
31-32 failed
34-35 failed
39-84 failed
13-16 failed
24, 25 failed
27-29 failed
31-32 failed
34,35 failed
39-84 Failed
Format Support Tests  

3 failed
7-13 failed

40, 56, 72, 99 Failed

3 failed
7-13 failed
40 failed
72 failed
99 failed
3 failed
7-13 failed
40 failed
72 failed
99 failed
3 failed
7-13 failed
40 failed
72 failed
99 failed

Issue Observed on Thunder4.2 WPE2.38 Builds

  1. BCM72180-821
  2. BCM72180-867
  3. BCM72180-856
  4. BCM72180-857 https://code.rdkcentral.com/r/c/components/generic/rdk-oe/meta-cmf-video/+/96623(need to check Alexa regression with the CL or not)
  5. BCM72180-858
  6. SGMM393-974
  7. SGMM393-975




  • No labels