Problem Statement:
When monitoring cloud environments (such as Microsoft Azure or Aruba Central) via API, transient network issues outside of ScienceLogic's control—such as connection resets, HTTP timeouts, or truncated/malformed JSON payloads—can occasionally cause a single poll of a caching Dynamic Application (DA) to fail.
Currently, when a caching DA experiences a transient, single-poll failure, it triggers a cascading "ripple effect" across the entire Dynamic Component Mapping and Relationships (DCM+R) tree. Because component devices rely on that parent cache for their state, a single missed poll immediately marks all dependent component devices as unavailable. This results in:
-
Alert Storms & False Positives: Dozens or hundreds of "Device Failed Availability" events are generated simultaneously, automatically creating unnecessary tickets and waking up on-call engineers for issues that self-resolve on the very next 5-minute poll.
-
Erosion of Platform Trust: When SL1 generates high volumes of transient tickets that clear themselves 5 minutes later, end-users and stakeholders begin to perceive the monitoring platform as noisy or unreliable.
Proposed Enhancement:
Introduce a configurable resilience or "self-healing" retry mechanism specifically for parent/caching Dynamic Applications before triggering downstream component availability failures:
-
Immediate/Short-Delay Retry on Collection Failure: If a caching DA encounters an API timeout, connection reset, or payload parsing error (e.g.,
JSONDecodeError), allow the collector to execute an immediate retry (or a configurable short backoff retry, such as 15–30 seconds) rather than waiting for the next standard 5-minute polling cycle. -
Grace Period / Cache Retention for Component Availability: Allow a configurable threshold (e.g., requiring 2 consecutive failed cache polls) before marking dependent DCM+R component devices as unavailable, while still logging a minor/notice event on the parent root device that the initial poll experienced a transient API error.
Business Value:
-
Reduces Alert Fatigue: Dramatically cuts down on false-positive P1/P2 tickets caused by momentary cloud API hiccups.
-
Improves Operational Efficiency: Prevents unnecessary on-call escalations and manual ticket triage for transient cloud provider blips.
-
Enhances Cloud Monitoring Reliability: Makes SL1's cloud PowerPacks significantly more resilient to the realities of public cloud API behavior.