Forum Discussion

kenwu's avatar
kenwu
Icon for Participant rankParticipant
14 days ago

Syslog and Trap passthrough in MC

Hallo,  I was advised by the support that this is "the way it should be" as Message collector are not log aggregator.  in Version 12.1.x, the message collector will no longer record the syslog or s...
sl1
  • TonyAndres's avatar
    TonyAndres
    13 days ago

    Hello Ken,

    By design, the data and message collectors are designed to not store data, instead the data gets pushed to the CDB. To clarify how SL1 collects syslogs:

    The end device sends the syslog message to either the data collector or the message collector. The rsyslog process will then verify that the syslog came from a known device (meaning the device is in the same CUG as the data or message collector receiving the syslog). 

    The rsyslog process then writes the syslog message to the MariaDB database on the data or message collector, specifically into the in_syslog.messages table. The event engine process on the collector will then consume the database record and see if it matches any event policies. If it does, then the event will be sent to the CDB as a high frequency data pull object, while the syslog message itself is sent to the CDB as a low frequency data pull object. The syslog message is sent regardless of whether or not there is a matching event policy. Once the storage object is written into the database on the CDB, you will see the syslog message and the event, if there is one, in the SL1 UI.

    If you are having issues with syslogs, we do have ways to stop the event engine from processing the database records. To do this, you can run the command visilo in the CLI of the data or message collector, then remove the value syslog from the eventmanager variable and save the change. Then run the command sudo systemctl restart em7_event. This will stop syslogs from being processed and they will remain in the in_syslog.messages table. Once you are done, you can re-add the syslog value back to the eventmanager variable after running the visilo command and after saving the change, re-running the command sudo systemctl restart em7_event.

    This of course assumes that rsyslog writes the syslog to the database. If the syslog is not getting past rsyslog, then it usually means the information in the syslog packet is not matching to a device in the CUG the data or message collector are aligned to. To troubleshoot this, you will need to do a tcpdump to look at the packets and investigate further. Please let me know if this helps.

    Antonio Andres

    Senior Technical Support Engineer | ScienceLogic

TonyAndres's avatar
TonyAndres
Icon for Moderator rankModerator
13 days ago

Hello Ken,

For SNMP traps, the process is very similar. The end device sends the SNMP trap to the data or message collector where it is received by the snmptrapd process. If the trap aligns to a known device, the snmptrapd process writes the trap into the net_snmp.notifications table in the MariaDB database on the data or message collector that received the trap. The event engine then consumes the record in the database, like it does with the syslog records, and proceses it. If the trap makes it past the SNMP trap filter and it matches an event policy, then the event will be sent to the CDB in the same way, as a high frequency data pull object. If the trap gets past the trap filter, then the trap itself is sent as a low frequency data pull object as well. 

Troubleshooting SNMP traps is very similar to how you troubleshoot syslogs. Run the command visilo in the CLI of the data or message collector, then remove the value trap from the eventmanager variable and save the change. Then run the command sudo systemctl restart em7_event. This will stop the SNMP traps from being processed and they will remain in the net_snmp.notifications table. Once you are done, you can re-add the trap value back to the eventmanager variable after running the visilo command and after saving the change, re-running the command sudo systemctl restart em7_event.

If the SNMP trap is not being written to the MariaDB database table, then just like with syslogs, you will need to run a tcpdump to get the packets to investigate any discrepancies in the trap packet with the SL1 device.

Antonio Andres

Senior Technical Support Engineer | ScienceLogic