2023-10-17 Meeting notes

RDK6

• Dac-Sec in rdk-next not turned on , cherrypicked to rdk6_main (LGI ticket) - not looking for certs. People / PIF can turn on themselves with their certs enabled in build.
• BRCM could take the RC candidate and fork - their changes not to be merged into RDK6 as it could adversely impact RDK6. 
• Cobalt 24 being worked by App team for RDK6.  Will need RDK6 updates, new recipe, DAB - independent components/changes. Missed RC.3
• RC.3 recut with revert of Rpi change (Pradeep/Deepthi agree basic sanity test gap) - will be available tomorrow. (builds in progress.)
• Need gap of a few days between rdk-next and rdk-main merge.

Q3

• RDKV Aamp issue - offending change identified. To be reverted and tried (was a specific change for Foxtel). James to provide Nightly with change reverted for platform teams to try.
• Gaps in testing - need dedicated manual testers and reviews form engineering leads.
• Hotfix for RDKV (sleep fix) required. TDK testing started.
• RDKB testing coming to end - should be able to release this week.
• Test results / documentation with Rama to update for RDKB.
• Might be an idea to drop camera next year.

Scanning

• • Hackathon branches are an issue - not good, one is 320 MB into RDK_Apps.
  • Retries issue on Github is quite rare (once a quarter) - Martin to open ticket for better error handling
  • Alan investigating GitSecrets client side hooks with Steve.
  • Needed to enable https to get UI to work - but introduces scan timeout issue- workaround is to do port forwarding to access the UI.
  • Reading up on Coverity. Bitbake build for component in place. Options to run binary with files as parameters. will need to install coverity on elastic slaves and add coverity builds. 
  • Coverity exe used to generate json file - may be sufficient. Otherwise will need to look at soap/rest api. Need mapping to know which recipe is used for building a component. 
  • XGuardCert Comcast tool to be investigated
  • Rialto coverity report created - follow up call with Luke of Sky needed.
  • Supporting Alan on Coverity - connected with Tony Colclough also.
  • Created Approval request for Jeremy's icons.
  • Martin to address KB updates on return from vacation.
  • .
  • ermgr scanned.
  • KB upgrades to be discussed with JAmes - Thurs/Fri. 
  • .
  • 3 components opensourced
  • Scanned repos in Comcast in preparation for hosting.
  • Coverity upgrade to be done this week
  • Target Friday for Protex KB upgrade.
  • ermgr opensourced but recipe not moved / updated yet.
  • .
  • KB upgrade failed - synopsys ticket opened. Need dev license
  • Working on dac_seec and LISA.

Lab

• • Status
    • Everything setup manually first - Plan to work with Cedric to add lab slaves to Cedric's Puppet.
    • Rack 2 - switch is fubar - not currently used so won't spend too much time on that. 
    • Rack 4 has intermittent IOLAN issues - low priority
    • Rack 5 -  12 slots out of 16
    • 5GHZ WiFi connection failures (40%) - needs ticket and investigation. 
  • Priority now: Stormtest->CATS→Puppet
  • Providing access to CMFLab to RDKM team would expose .netrc passwords etc. and give access to soc code. So problematic.
  • Python 2 special on old stormtest server is fine as is - won't update
  • PFSense has different base OS - need to figure out how to handle that - reading documentation
  • Rpi3 Broadband sanity test issue (REFPLTB-2296) - being worked. (Not seen on Rpi-4) rpi-4 default now, rpi-3 nightly.
  • CATS Power requirements worked out by Alan - 33KW (PIF team may also be able to provide input). Mapping out circuitry of racks
  • Waiting on electrician for transformer
  • .
  • One transformer arrived.  NSC rack removed. Next step is for 2nd transformer to arrive, electrician to lift floor tiles to provide power circuit. Will do PFSense at same time
  • Need to request CATs support person in advance.  Alan to get electrician timelines from Sue.
  • .
  • 1 transformer in place - electrician trying to source 3 phase one.
  • .
  • Rack moved to make space for CATS rack - power moved by electrician
  • 3-Phase transformer - bespoke being soured by electrician.
  • .
  • Completed NW and power change connections to make way for CATs rack. VM updates in progress. Alan to add checklist for James for when he is out.
  • Metric report being generated
  • Griffin has potential HD issue - repowered up and okay for now.
  • .
  • Hybrid networking changes complete (avahi support)

Test

• DAC tests to be added  or refapp2 - once 23.0.1 migration is complete.
• BRCM RefApp2 broken
• Rpi-4 migration in progress.
• Next up supporting Aamp team CI.
• .
• Rpi-4 CI done - adding RPi4s to racks this week.
• Supporting BRCM team on serial port issues
• Need to reflash realtek boards due to OTA issue
• Complete WLPL investigation.
• Next - support AAMP CI.
• .
• Imports / Roundtripping taking a lot of time
• Lost day to realtek issue.
• Rpi-4 migration in progress
• .
• Rpi-3 to be removed from contribution testing (will still do nightly). Some Rpi3s will continue to be used for Aamp CI. 

• Metrics / Tooling :

• • TruffleHog now reporting to slack for Gerrit and GitHub - if issues detected - DSL changes to be pushed. Martin and Stephen to be added to channel.
  • Jenkins DSL → pipeline triggering -  put on hold for now
  • Migrated secrets scanning to python3
  • Need table on TruffleHog/GitSecreta/Coverity on Gerrit , on Github, automated/manual, contributions.
  • Coverity:
    • Challenge with Coverity will be mapping of recipes to components. 
    • Refocus on coverity contribution scanning for next month.
    • Analysing rdk coverity builds, Charlies web pages, Simon's scripts. Got new slave to work on Coverity.
    • Lots of work on Coverity - documented on wiki. (Builds, Streams / Projects). Static analysis on all components - not just C (Javascript/Python etc.)
    • Steve to create metatdata tuple recipe - component.
    • CMF-13655 coverity contribution - get POC on internal jenkins hardcoded with Utopia to begin with.
    • Coverity scanning - reading up on synopsys documentation. Did build all and produced utopia report.
    • Next step build Utopia in isolation. Need to have meeting with Simon/Martin to clarify how existing tools handle changes.
    • Martin to reach out to Tony Colclough to request meeting on on what they do in Comcast.
    • Coverity pipeline put in place by Simon - for contributions. alan looking at data - analysing existing versus new - generate report that identifies changes.
    • Update from Tony Colcough - provided link to what they do, webportal available for developers. Nightly builds - can isolate newly occurring issues only.
  • Contribution metrics pages updated for August
  • .
  • Metrics report released today.
  • Coverity - going through Martins report. Next step contribution identification - need to identify lines changed - Steve working on providing that
  • This week focus on isolating changes to files.
  • Alan to provide table of tools - Gerrit & GitHub status
  • .
  • Script to identify line numbers of change set in Gerrit completed - jason format to be agreed.
  • Next step is to marry that with work done to generate an over all report. 
  • Next step get working for GitHub.
  • Investigating static analysis on non C repos (WebUI)
  • Project: Repo, Stream : Branch. vs Project : Profile, Stream component.
  • Monthly metrics reports (Parked SLOC)
  • Need to plan migration to main from python3-master. 
  • .
  • Coverity static analysis working for non C code on Gerrit and GitHub
  • Writing python script to marry Steve's line number data with Coverity reports. 
  • Steve to work on providing line numbers from GitHub PRs - non trivial.  GraphQL
  • Alan also updated SLOC data - will sync on that. 
  • .
  • Working on GitHub PRs which can contain hundreds of commits - complex but possible to get lines. (Plan B - clone and do diff)
  • Static analysis of contributions working on Gerrit - received first issue reported to slack today.
  • Next step C code - how to get the list of changes to pipe into Steve's script?  Take offline  - chat with Simon. 
  • .
  • Metric maintenance completed.
  • Coverity removed hardcoding form scripts. Investigated robot comments - back to gerrit/github. Utilities avaialble (Simon's scripts etc)
  • C code integration not progressed - to be discussed with Steve/Simon/Alan today.
  • MArtin / Alan / Steve t review ticket before sharing with Comcast security team.
  • GitHub use of graphQL - too complex. Will try git diff instead - hopefully works by end of this week (sed / awk to pick out line numbers.)

Sysadmin: 

• • Jenkins Pipeline- deferred. Gerrit upgraded to 3.7.4
  • Gerrit issue where UI reverts didn't get mirrored to replica. Ticket raised. Alarm script complete for reverts not getting to replica. 
  • Moving ubuntu entities to 20.0.4. Kirkstone requires 20.0.4
  • Final jenkins upgraded to ubuntu 20.04. Build slaves remain on 18.04 for now - will migrate to 20.04 when we move to kirkstone.
  • BOT account still to be reassigned to Cedric - will follow up with Rama next week.
  • Lambda job in pace - SQS plugin still there but not required. Cedric to raise a ticket outlining jobs that need to move.
  • Build IMI images - removed python2 stuff. Shining Panda uses 2.7. Yocto analysis.  Ticket to update to python 3.Build Slave - 20.04 in progress. Needs tests
  • Crowdstrike update - Cedric to email for update.
  • Upgraded Ubuntu build AMI to be tested by Simon when time permits.
  • Tracked emails sent to junk folders
  • Gerrit upgraded to 3.7.4 - issue with labels mostly sorted.
  • Will upgrade jenkins next weekend. 
  • Progressing SQS removal from build jobs
  • Still chasing Crowdstrike issue and Junk email issue.
  • .
  • SQS removed from internal jenkins.  1 to do on RDM jenkiins (handle Simon's blacklist filter - cant access meta data on public jenkins. Steve to check if anything proprietary there.
  • Working on persistent build slaves for wider RDKM team.
  • Ticket closed on junk email issue - seems sorted
  • Jenkins upgrade postponed for now.
  • .
  • Jenkins internal upgrade complete
  • Additional Alerts added
  • 2 AM Gerrit load very high - analysing.
  • 1 Tata person - stopped cloning, 1 still cloning with multiple threads - will contact again.
  • .
  • Cedric working to provide RDKM access to builds - phase 1 provide access, phase 2 optimise ss state cache. 
  • Setting up Autobhan for external access (superseed ssm access) = subnet created, AWS setup implemented, AMIs in place - next security groups and Autobhan.
  • Should be able to test with friend user in a few days.
  • Crowdstrike issue persists on ephemeral build slaves.
  • Vantiva issue resolved, Tata person needs follow up. 
  • .
  • Crowdstrike upgrades completed. Still an issue on build servers - no response to Cedrics suggested solution.
  • Autbahn not a good option as woul dneed to publish IPs. Wil go with AWS SSM instead. End of month should be ready with phase 1.
  • Phase 2 might take a bit longer - ssstate cache - need to be clear on what can be shared, how to limit access.
  • Have 2 flavours of EFS - Rpi and Platform. Will need 2 more for platform variants. This week meet to determine how its going to be done.
  • No outstanding issues with community high volume clones.

Build

• • IPClient Rpi4 build broken - with platform team.
  • DVBDTVKIT - new branch in GitHub. Platform team to try out. BRCM only. Rpi use binaries in build. Hosted code not merged. Martin okay to scan code on Gerrit - not related to SDK23.
  • Clean up of branch names on cmflib etc will be done in August by James..
  • QT source code needs update for new ubuntu AMI builds.
  • .
  • WLPL change to be brought to rdk6_main
  • rc.2 build jobs / scripts created to handle hybrid of rdk-main and BSP update. Testing.  rc.2 has new bsps. 
  • Next
    • Slash / Dash problem with branching under investigation - goes back to Morty.
    • Kirkstone support - a lot of the changes are very hackey (if it doesn't work - remove it!). Depends on BRoadband team having a build off of rdk_next.
    • Feature branch CI support (need to consider DEPS notation)
    • QT code change for focal upgrade
  • Imports very busy - breaking builds
  • RC.2 cut
  • RDK6_Main CCB, management
  • Assisting those with basic cherry picking problems - dealing with conflicts.
  • .
  • Updating branching scripts for kirkstone. Q3 will use feature branch.  Lots of changes yet to roundtrip to rdk_next.
  • Rpi-4 roundtrips stil not completed after 3 months! Follow up with Pradeetpa re prioritising strategy.
  • Fixed coverity build script to handle corner case.
  • A lot of refactoring ongoing - breaking imports,
  • CI support for branches up next.
  • .
  • Lots of build breakages - not getting to important stuff 
  • Another raft of AAMP changes came in this week. 
  • CI for feature branches started (changed refapp build to use our manifest)
  • James has slave that Simon can use.
  • Way too many patches (particularly in SOC layers)- lots of work for us.
  • .
  • Added recipe for ermgr (not rc3.1)
  • Import fixing and patch updates taking up most of time
  • Not much progress on branch CI
  • Removed Turris extender and Rpi3 broadband builds.
  • Lots of work on releases - Q3 and RC.3
  • RDK6-Main taking a lot of time.

• General Admin

• Working on repo creation tickets - CMFSupport
• Developing scripts for Coverity contribution isolation
• Working on changelog stuff. (cherrypicks, hyperlinks, Gerrit/Github - determine remote.
• Branching convention for OE layers to be agreed. 
• James to update changelogs associated with the quarterly release.
• Slides yet to be done.
• Hal and Halif repos to be renamed today.
• halif repos renamed - metadata to be updated.
• Branch protection on GitHub
• Working with Sky (Luke) on CI questions.
• .
• Discussion ongoing on amazon launcher access - Steve to provide conversation trail

Other 

• • Requested intern. 
  • Client side webhooks pseudo code to be provided to Alan to progress. Git client-side hooks working in principle - need to document workflow. Will create repo to host. Release naming rules to be defined. 
  • Irdeto VA being worked to enable better security, to satisfy premium apps requirements. (Anand). Nagra CAS also being progressed. 
  • Kirkstone on Broadband for Q2 will be dev version only. Q3 will be timeframe for formal Kirkstone release.
  • Completed scripts for file extension detection - deployed on Gerrit. Github needs table - not deployed yet. Need DSL base builder!
  • Glee to host monthly call with team.
  • One CODEMGMT ticket to be created for all HAL / HALTest repos.
  • Regular calls with Z and Bart to be setup.
  • Progressed scripting for importing GitHub repos (snapshots, with history, with branched etc) all defined in meta data)..
  • Global summit 19th Sep - Follow up with Brendan re IBC meetings that would require technical backup.
  • Trufflehog / Git Secrets  client side pre-commits  being worked by Steve - will expand to rest of team and look to make available globally to dev community - plan next week.
  • Steve to email 2 tata resources with very slow connections.
  • RDK7/RDK-E hosting still under discussion - Khem producing something for Labeeb
  • Steve to provide slides on Dev Sec Ops key drivers.
  • Need to connect with Jose re LAB support.
  • Martin on vacation next week
  • Kieran to check is alternative office can be used for meeting room.
  • .
  • RDKM senior leadership team planning Ireland trip in October / November (and Rotterdam)
  • Trip to Philadelphia - sync with Brendan / Jose, in context of travel restrictions. Early October.  (SCTE 16-19 Oct Network X 24-26 Oct)
  • Lightning 2.0 maintenance. Lightning 3.0 Wouter to manage his own space on GitHub
  • Glee setup monthly call with team.
  • Monthly status report to senior RDKM team to be created / presented
  • Broadband to remain as quarterly release. No VA planned.
  • RDK7 - not fully understood yet.
  • Change logs updated to include GitHub. Federated components are not in manifests - so need to use gitlog command to get changes between 2 sha-1s (wpe framework, rdkservices, rbus etc.)
  • .
  • IBC Comcast focus on EntOS + Sky Glass.
  • Summit focused on Apps, and certification suite. Irdeto support for operators.
  • Lab support for Broadcom initiatives - Jose looking to Rotterdam lab. 
  • Network issue since Friday - requires DNS updates to config files. Imports from Comcast only handled mediaclient case not hybrid.
  • RDKB release candidate cut today.

Mastership - Stalled

• STBT-33275. Resolved Reopened. Kieran to organise call to get latest update. 
• ccspPandM POC -  Glee to follow up with Sanjay Dwivedi on ccspPandM - no response from Sanjay. On hold