ParticipantSyslog and Trap passthrough in MC
Hallo,
I was advised by the support that this is "the way it should be" as Message collector are not log aggregator.
in Version 12.1.x, the message collector will no longer record the syslog or snmp trap that is received by the MC in any of the capacity. These messages are being proceed on a memory buffer directly into the Event Engine into the Database.
1 - devices are sending syslog + trap to MC - we are able to confirm in tcpdump
2 - MC takes these packets and memory buffer it to somewhere and it get proceed by the mc event engine as ignore or proceed
3 - if proceed - we will see it on the DB
4 - if not proceed - we will see nothing on DB
So now, we have no ability to know on the format or the actual message received by the message collector. There is currently no "known way" to redact this into a file or forward out to another syslog server as they are passthrough over the memory, at least we tried and no success. Docs indicated that the rsyslog function will also work with local message, which excludes the device syslog/trap.
I can't tell if this is good or bad but I know it's hard to troubleshoot when we are not receiving on the DB and we are trying to decipher the message on the MC. Anyone, have any suggestion, or should this be a EM number :)
Hello Ken,
By design, the data and message collectors are designed to not store data, instead the data gets pushed to the CDB. To clarify how SL1 collects syslogs:
The end device sends the syslog message to either the data collector or the message collector. The rsyslog process will then verify that the syslog came from a known device (meaning the device is in the same CUG as the data or message collector receiving the syslog).
The rsyslog process then writes the syslog message to the MariaDB database on the data or message collector, specifically into the in_syslog.messages table. The event engine process on the collector will then consume the database record and see if it matches any event policies. If it does, then the event will be sent to the CDB as a high frequency data pull object, while the syslog message itself is sent to the CDB as a low frequency data pull object. The syslog message is sent regardless of whether or not there is a matching event policy. Once the storage object is written into the database on the CDB, you will see the syslog message and the event, if there is one, in the SL1 UI.
If you are having issues with syslogs, we do have ways to stop the event engine from processing the database records. To do this, you can run the command visilo in the CLI of the data or message collector, then remove the value syslog from the eventmanager variable and save the change. Then run the command sudo systemctl restart em7_event. This will stop syslogs from being processed and they will remain in the in_syslog.messages table. Once you are done, you can re-add the syslog value back to the eventmanager variable after running the visilo command and after saving the change, re-running the command sudo systemctl restart em7_event.
This of course assumes that rsyslog writes the syslog to the database. If the syslog is not getting past rsyslog, then it usually means the information in the syslog packet is not matching to a device in the CUG the data or message collector are aligned to. To troubleshoot this, you will need to do a tcpdump to look at the packets and investigate further. Please let me know if this helps.
Antonio Andres
Senior Technical Support Engineer | ScienceLogic
