Elastic Data Lake - Decision to move away from Tivoli LFA

AIX logs from hundreds of servers, were being ingested to Elastic Datalake; to Logstash using Tivoli LogFile Agent (LFA).

Problem statement:?

• Consistent high utilization of CPU and memory leading to many incident tickets on LFA nodes.
• Issues of node hung, requiring forced reboot, involvement of other support team members.?
• Frequent clean-up of temp storage due to high ingestion rate.
• Recurring issues leading to escalations and wastage of time and effort of resources involved.
• Unavailability of data during the time of outage.

What was done:

Used rsync to ingest data directly to Logstash. rsync pods running in the K8 cluster provides built-in tolerance.

Benefits:

• Synchronizes incremental data from last successful sync. So no more missing data.
• Released all LFA nodes and no more issues stated in problem statement.

Initiative and ownership:?Rajaraman Sathyamurthy

LFA migration lead and executed by: Prashanth Jonna