Skip to main content
Solved

Syslog and Trap passthrough in MC

  • January 8, 2025
  • 2 replies
  • 0 views
K
Forum|alt.badge.img

Hallo, 

I was advised by the support that this is "the way it should be" as Message collector are not log aggregator. 

in Version 12.1.x, the message collector will no longer record the syslog or snmp trap that is received by the MC in any of the capacity. These messages are being proceed on a memory buffer directly into the Event Engine into the Database. 

1 - devices are sending syslog + trap to MC - we are able to confirm in tcpdump 
2 - MC takes these packets and memory buffer it to somewhere and it get proceed by the mc event engine as ignore or proceed 
3 - if proceed - we will see it on the DB  
4 - if not proceed - we will see nothing on DB 

So now, we have no ability to know on the format or the actual message received by the message collector. There is currently no "known way" to redact this into a file or forward out to another syslog server as they are passthrough over the memory, at least we tried and no success. Docs indicated that the rsyslog function will also work with local message, which excludes the device syslog/trap. 

I can't tell if this is good or bad but I know it's hard to troubleshoot when we are not receiving on the DB and we are trying to decipher the message on the MC.  Anyone, have any suggestion, or should this be a EM number :) 

Best answer by TonyAndres

Hello Ken,

By design, the data and message collectors are designed to not store data, instead the data gets pushed to the CDB. To clarify how SL1 collects syslogs:

The end device sends the syslog message to either the data collector or the message collector. The rsyslog process will then verify that the syslog came from a known device (meaning the device is in the same CUG as the data or message collector receiving the syslog). 

The rsyslog process then writes the syslog message to the MariaDB database on the data or message collector, specifically into the in_syslog.messages table. The event engine process on the collector will then consume the database record and see if it matches any event policies. If it does, then the event will be sent to the CDB as a high frequency data pull object, while the syslog message itself is sent to the CDB as a low frequency data pull object. The syslog message is sent regardless of whether or not there is a matching event policy. Once the storage object is written into the database on the CDB, you will see the syslog message and the event, if there is one, in the SL1 UI.

If you are having issues with syslogs, we do have ways to stop the event engine from processing the database records. To do this, you can run the command visilo in the CLI of the data or message collector, then remove the value syslog from the eventmanager variable and save the change. Then run the command sudo systemctl restart em7_event. This will stop syslogs from being processed and they will remain in the in_syslog.messages table. Once you are done, you can re-add the syslog value back to the eventmanager variable after running the visilo command and after saving the change, re-running the command sudo systemctl restart em7_event.

This of course assumes that rsyslog writes the syslog to the database. If the syslog is not getting past rsyslog, then it usually means the information in the syslog packet is not matching to a device in the CUG the data or message collector are aligned to. To troubleshoot this, you will need to do a tcpdump to look at the packets and investigate further. Please let me know if this helps.

Antonio Andres

Senior Technical Support Engineer | ScienceLogic

2 replies

T
  • TonyAndres
  • Community Manager
  • Answer
  • January 9, 2025

Hello Ken,

By design, the data and message collectors are designed to not store data, instead the data gets pushed to the CDB. To clarify how SL1 collects syslogs:

The end device sends the syslog message to either the data collector or the message collector. The rsyslog process will then verify that the syslog came from a known device (meaning the device is in the same CUG as the data or message collector receiving the syslog). 

The rsyslog process then writes the syslog message to the MariaDB database on the data or message collector, specifically into the in_syslog.messages table. The event engine process on the collector will then consume the database record and see if it matches any event policies. If it does, then the event will be sent to the CDB as a high frequency data pull object, while the syslog message itself is sent to the CDB as a low frequency data pull object. The syslog message is sent regardless of whether or not there is a matching event policy. Once the storage object is written into the database on the CDB, you will see the syslog message and the event, if there is one, in the SL1 UI.

If you are having issues with syslogs, we do have ways to stop the event engine from processing the database records. To do this, you can run the command visilo in the CLI of the data or message collector, then remove the value syslog from the eventmanager variable and save the change. Then run the command sudo systemctl restart em7_event. This will stop syslogs from being processed and they will remain in the in_syslog.messages table. Once you are done, you can re-add the syslog value back to the eventmanager variable after running the visilo command and after saving the change, re-running the command sudo systemctl restart em7_event.

This of course assumes that rsyslog writes the syslog to the database. If the syslog is not getting past rsyslog, then it usually means the information in the syslog packet is not matching to a device in the CUG the data or message collector are aligned to. To troubleshoot this, you will need to do a tcpdump to look at the packets and investigate further. Please let me know if this helps.

Antonio Andres

Senior Technical Support Engineer | ScienceLogic

    T
    • TonyAndres
    • Community Manager
    • January 9, 2025

    Hello Ken,

    For SNMP traps, the process is very similar. The end device sends the SNMP trap to the data or message collector where it is received by the snmptrapd process. If the trap aligns to a known device, the snmptrapd process writes the trap into the net_snmp.notifications table in the MariaDB database on the data or message collector that received the trap. The event engine then consumes the record in the database, like it does with the syslog records, and proceses it. If the trap makes it past the SNMP trap filter and it matches an event policy, then the event will be sent to the CDB in the same way, as a high frequency data pull object. If the trap gets past the trap filter, then the trap itself is sent as a low frequency data pull object as well. 

    Troubleshooting SNMP traps is very similar to how you troubleshoot syslogs. Run the command visilo in the CLI of the data or message collector, then remove the value trap from the eventmanager variable and save the change. Then run the command sudo systemctl restart em7_event. This will stop the SNMP traps from being processed and they will remain in the net_snmp.notifications table. Once you are done, you can re-add the trap value back to the eventmanager variable after running the visilo command and after saving the change, re-running the command sudo systemctl restart em7_event.

    If the SNMP trap is not being written to the MariaDB database table, then just like with syslogs, you will need to run a tcpdump to get the packets to investigate any discrepancies in the trap packet with the SL1 device.

    Antonio Andres

    Senior Technical Support Engineer | ScienceLogic

    Terms & ConditionsAccessibility statement