First Level
- Frequency - start with 5 minutes and then expand.
- If log upload frequency is 5 minutes, that is separate from the log collection frequency, which will be based on various parameters.
  - Parameters could be counters.
  - Parameters may not be available in logs but may need real-time probing using snmp/tr-69 or other methods.
  - We will need collect records based on frequency at which collection needs to occur.
- Log some events back to the cloud. Could be to S3 or another destination.
- Records that it upload should be records that could be fed into Splunk without having to do much processing, for example, a name/value pair or another that is easy to expose on Splunk.
- There could be multiple records since logging period.
- Data collection frequency and upload frequency can be configured differently.
- The level will be enabled for every device.
Second Level
- Medium number of logs. Full logs will need to be pushed.
- Second and Third level are similar.
Third Level
- Similar to current method of full log uploads.

Architectural diagram

...

Image Added

Process Flow

Image Added

DCMscript.sh communicates with Xconf server and fetches the predefined markers
Using the markers, DCM Script will prepare a sorted map file for the log lookup, creates a CRON job for DCA Agent.
CRON job retrieves data from the device using DCA which filters search patterns by looking into logs.
From the retrieved data, it will create a JSON formatted message.
JSON format data is uploaded to server

...