This page created for architecture overview and roadmap discussions.

Additional Resources:

https://www.adobe.com/content/dam/acom/en/devnet/video/pdfs/introduction_to_closed_captions.pdf

Text Track Capability Support/Test Matrix

Status/Workstreams

4 Dec 2020

• Finished font style support (not includes italics and underline)
• Have option to use dedicated font if needed, but for now using sheering transformation
• Review shared, +1 from Steve Waddel; need merged (Phil & Shripad can help with review)
• Even though we are using patch, be nice to get a normal gerrit review with changes in context against subtec component
• Started looking into UVE integration; reviewed existing bindings (with mappings to RDK CC manager); this is right place; Vinish KB can help with review / guidance
• Note: App team looking to get something to start coding against “ASAP”
  • Expecting same UVE/JS APIs to work without changes as documented
  • What do current APIs do today if used on PlatCo (w/o additional RDK changes merged); CMAKE flags are off, so API use should be NOP
• Functionally, should be short runway to introduce working 708 changes
  • But also will need code to create Wayland surface – pulled in Mike Fiess? <- biggest risk
    • Currently using manual curl commands from command line to create surface – using rdkshell
      • This is moved to top of all layers (for now)
      • Consistent with X1, where today there are three layers: Video, “Graphics” (Guide) and CC plane
    • Looking for guidance on best practices for final solution
      • Lifecycle? Only needed with ATSC and AAMP playback
      • Good to have option to change Z-order: over/under UI Graphics plane; Highsense may have strong feelings here
      • Currently abstracted as implementation detail; PLUI uses UVE bridge to select/enable caption tracks
      • C/C++ code can manage use of RDK shell directly avoiding need to use command line; there’s C/C++ thunder interface
      • Should we create/destroy surface dynamically? Or leave there hidden (SetVisible) (to avoid blit/composition overhead) when not needed?  Worried about wasting memory when IP Video not being presented.
      • Could create lazily only when caption plane needed, to avoid any possible impact to tune times
      • Mike notes that the create is NOP if called more than once with same surface name
      • Note: if another app becomes visible later, could start at higher z-order; could use an “alwaysOnTop” api to allow caption plane to avoid being obscured during composition
        • Action: Mike to create and share ticket for this feature request
    • Q: what about scaled down apps?
      • Probably should remain either full screen (and readable)
      • Or suppressed/hidden
    • And testing to ensure App can discover caption availability at runtime
    • AML DVB & HAL changes will need to be pushed too – risk: could clash with AmLogic platform changes (merged to sprint yesterday)

Will need to decide whether tracks not published in PAT/PMT need to be handled

• Seen in test streams
• Need to check production ATSC content
• And check behavior with reference TV, here

13 Nov 2020

• Llama (AmLogic) badly needs out of band VTT caption support; tested/working with Subtec on Xione (Realtek), not yet integrated/tested for Llama (AmLogic)
  • Recommend a dedicated resource for this – Steve to check for support from Fred

• Plan: focus like laser beam on subtec integration (over RDK CC manger), starting with AmLogic (PlatCo)
• Premyslaw Status & Open Issues for CEA708 POC
  • Started with code in aamp, but moved to CC HAL – intercepting inband caption data and dispatching to subtec
  • For video playback used mediarite test page / mediarite player – playback from file option
  • CEA708 Placement issues – under investigation
  • Synchronization? Subtec renders as they arrive – not using precise timing information - may be good enough as-is.
• Worksteams:
  • CEA708 subtec caption support integrated with AAMP IP Video playback
  • CEA708 subtec caption support integrated/working with Mediarite playback
  • CEA608 support in subtec (James) - will surely spill into December, but should be doable/straightforward
  • Support for user/app-managed CC styling options (as mentioned in RDK-28824)
  • This is a regulatory issue; user needs to be able to specify styles, font size, etc. and override presentation. Today, we have a set of JSON data that can be used for configuration (from UVE), with well-defined set of attribute/value pairs.  Subtec does not yet support this type of override - currently driven only by control codes embedded in user data - can/should change be done with source /dev/amstream_userdata?

Open Architecture Notes/Questions

• In POC, created a Wayland display for Subtec and took care of z-order manually using RDKShell curl commands
• What surface should Subtec use in real deployment? Should it have its surface or use some existing one? 
• How to pass the name of Wayland display to Subtec? Currently, the Wayland display to use is fixed on Subtec startup - is this okay?
• If it should have its own surface, who should create it and take care of the z-order?
• For "apps" (AAMP), do we need to consider/test with in-band (HLS?) captions, or is it all webvtt/ttml? Yes – aamp needs to. work with both OOB web vtt/ttml captions (currently only Sky requirement for XiOne/Llama), but also 708/608 captions for US feeds (PlatCo, and eventually replacing use of RDK CC manager)
• Don’t expect to have to make changes in tsprocessor – HAL can extract CC data from currently streaming video

1. What component(s) do we want to pass up userdata and control subtec? Presumably this should be a common solution for apps and mediarite playback (so not mediarite).
2. subtec gstreamer elements exist that could be common, but they rely on tsdemux which is not present with appsrc we think.
   https://wiki.rdkcentral.com/pages/viewpage.action?spaceKey=RDK&title=Subtec+filter+GStreamer+SOC+level
3. should AAMP be running in parallel to mediarite gstreamer pipeline, reading userdata from the HAL and passing it up to subtec app? So it owns this interface for all use cases (apps/mediarite).
4. Are we aware of any re-timestamping in the gstreamer pipeline? i.e. Will HAL timestamps match PTS going to amlhalasink and westeros sinks? Meaning we don't need to be feeding from the gstreamer pipeline.

Next steps

• System diagrams to help with review/understanding:
  • RDK CC manager as used in RDK today (Deepa)
  • Subtec as used in POC for 708 caption rendering (Premyslaw)
  • Subtec as used by AAMP for VTT/TTML caption rendering (Steve)
• Capture in new wiki page for easier collaboration
• Can/should we use CC manager approach (APIs)?
• Review by RDK CC manager SMEs (including QAM RMF player) for eventual replacement with subtec

19 Nov 2020

• Discussions with existing subtec team; looking to understand architecture
• Agreement:
  • where to extract user data for 708 (and eventually 608) captions
    • use AmLogic HAL for all inband CC data harvesting - should work with both AAMP and OTA playback
• Where are control points?
• track discovery
• expectation: don't need to filter 708/608 captions - send it all to subtec, and inform subtec which to select for display
• style configuration - where to do?
• subtec does not support overrides today - from user app settings; need place to support this
  • extension to subtec
  • there is internal styling part of 708 raw signals
• gap: subtec support for 608
  • focused on single test feed
  • some have mix of 608/708
  • some have only 608
• Jan's sharepoint - referenced in IPTV Glue Squad
  • set of test streams
• Q: where can we find definitive documentation for CC formats?
• How to expose path to signal subtec which inband subtitles to render?
  • could extend existing socket communication if needed
  • there is an existing "set active type" command that should work for this purpose
  • need new component between AmLogic HAL to forward data to subtec
  • need to ensure path from UVE API to signal which track to actually render
    • subtec supports raw 708 data; does parsing/rendering
    • some more complexity with subtitles - can be on different pid?
      • what does "subtitle" mean in UK context?  inband dvb? teletext?
      • HAL handles user data from Video, not on separate pid
      • not requirement for PlatCo US

• Wayland surface management?
  • separate, important discussion

• For Llama/MR native integration, not currently our problem; MR could render directly

• OOB vs IB caption flow
  • AAMP→VTT/TTML over socket→subtec
    • LGI - direct socket vs. gstreamer plugins - pros/cons?
    • using gst pipeline could help with timestamp handling; not good fit
  • AAMP->gstreamer->HAL->subtec
• Live vs. VOD
• PlatCo: UVE → (AAMP, Thunderer Mediarite) captions
• Llama w/ AS → (JSPP->AAMP, Native Mediarite) captions

• proposal:
  • UVE could talk directly to subtec over direct different socket
  • UVE could talk to the component that marshals data from HAL to subtec, then have commands injected
• Q: when will Mediarite be able to advertise available tracks?

• Jan: needs to be in follow-up discussions; especially regarding compositor for Sky use cases along w/ Mike Fiess
• Want to avoid overlap in workstreams where possible
• we have lots of test streams where it's signaled in PMT as expected
  • but also found some streams that only CC data in video, without advertising in PMT(!)
  • what does this signify?  is this meant to be a hidden track?
  • Or must we peek inside video to find and advertise?
  • What do competitor TVs do with such feeds?

Architecture Diagrams

RDK CC Manager (Deepa)

rdk_cc_manager.svg

Inband Caption Support using Subtec (Premyslaw) - for use with Mediarite ATSC and AAMP HLS/DASH 708/608 captions.

• Diagram of the POC. Subtec has it's own Wayland Display created using RDKShell. The proper z-order is configured manually using RDKShell.
  In this POC, CCDataForwarder is a standalone app that registers its hooks in CC HAL, packages user-data according to Subtec's protocol, and sends datagram packets to Subtec's Unix socket.
  
  

Subtec PoC (Steve) - as used with OOB Web VTT