...

• First Level
  
  • Frequency - start with 5 minutes and then expand.
  • If log upload frequency is 5 minutes, that is separate from the log collection frequency, which will be based on various parameters. 
    
    • Parameters could be counters. 
    • Parameters may not be available in logs but may need real-time probing using snmp/tr-69 or other methods.
    • We will need collect records based on frequency at which collection needs to occur.
  • Log some events back to the cloud. Could be to S3 or another destination.
  • Records that it upload should be records that could be fed into Splunk without having to do much processing, for example, a name/value pair or another that is easy to expose on Splunk.
  • There could be multiple records since logging period. 
  • Data collection frequency and upload frequency can be configured differently.
  • The level will be enabled for every device.
• Second Level
  
  • Medium number of logs. Full logs will need to be pushed. 
  • Second and Third level are similar. 
• Third Level
  
  • Similar to current method of full log uploads.

Architectural diagram

Process Flow

• DCMscript.sh communicates with Xconf server and fetches the predefined markers
• Using the markers, DCM Script will prepare a sorted map file for the log lookup, creates a CRON job for DCA Agent.
• CRON job retrieves data from the device using DCA which filters search patterns by looking into logs.
• From the retrieved data, it will create a JSON formatted message.
• JSON format data is uploaded to server

...