Agile SSIS series (12/12): Future of FLOGS

So project FLOGS is over. The task was to design comprehensive and high performing logging platform for SSIS. Also I wanted to improve my knowledge of agile (SCRUM). I believe that I have achieved both goals. However, during the design I had couple more ideas that I did not have time to implement.

How to improve FLOGS?

While testing final version of FLOGS, I suggests following areas to be developed further:

• Improved error logging in data flow – ability to redirect rows to a generic data structure and achieve non interruptible loads. This would mean that the SSIS load would carry on and not fail even though error in data flow would occur (for example wrong data type got into the pipeline).
• Sophisticated reporting solution on top of FLOGS – develop OLAP cubes, reports and dashboard based on data collected from FLOGS. These could further support goals G1 – G3.
• Improve performance of logging – as suggested in previous text the performance of the logging can be significantly improved by storing the log data temporarily to in-memory structure and flush the data to the database once in a while. Performance impact of such logging would be close to nothing.
• Align FLOGS data with Windows Management Instrumentation data (CPU or memory utilization etc.) - analysis can be taken even further and determine hardware bottlenecks when the SSIS packages execute.

Few last words...

Thanks for reading this series, I am looking forward to your comments as well as improvements to FLOGS. Please download FLOGS here (from Dropbox).

Previous articles in the series

• Agile SSIS series (1/12): High performance SSIS logging platform – introduction to the project FLOGS; what do I want to achieve and who is the target audicence?
• Agile SSIS series (2/12): Top 4 features that SSIS logging platform must have – definition of 4 basic requirements and justification why I see these as the most important criteria.
• Agile SSIS series (3/12): Why do we REALLY use agile approach? - justification of agile; one needs to know what are the reasons to use agile.
• Agile SSIS series (4/12): Different ways how to do SSIS logging – review of existing solutions or approaches how to log execution of DTSX packages; this review also gives a hint which way to go.
• Agile SSIS series (5/12): What is the smartest way to log SSIS execution? - data flow diagram defining the architecture of the solution.
• Agile SSIS series (6/12): Well designed data structures for SSIS logging – entity relationship diagram and snow flake schema which was developed in the first design iteration.
• Agile SSIS series (7/12): Prototyping the agile way! - first prototype for testing; implemented the dataflow as well as the entity relationship diagram and build test DTSX packages.
• Agile SSIS series (8/12): Performance testing... - the prototype is tested in order to determine the impact on the performance of SSIS execution; it proved that the design is flawed and can impact performance of rapid executing packages.
• Agile SSIS series (9/12): Back to the drawing board – based on the performance testing, the architecture of the solutio needs to be changed.
• Agile SSIS series (10/12): Performance testing of the final prototype – after the architecture change, has the impact on performance been mitigated?
• Agile SSIS series (11/12): Log data analysis – once the data has been logged how do I interrogate it? How can one improve performance of SSIS packages using extensive logging?

Want to know more about Agile BI? Please read my agile BI series...