Optimizing Memory Usage in LINQ: A Case Study with IEnumerable

Optimizing Memory Usage in LINQ: A Case Study with IEnumerable

Today I was dealing with an IEnumerable which would iterate over 4 millions predictions to group by Fraud (a boolean predicted value) and count number of records. As a side note, each prediction (output) was an object with 300+ properties (it is a lot, but on purpose).

Below is an snippet of the code:

var metrics = predictions.GroupBy(p => p.Fraud).Select(g => $"{g.Key};{g.Count()}")        

Half way through, I received an "Out of memory exception". After a few minutes thinking about, that was how I solved the problem (pay attention to the Select clause before GroupBy):

var metrics = predictions.Select(p => p.Fraud).GroupBy(fraud => fraud).Select(g => $"{g.Key};{g.Count()}")        

By putting a Select before GroupBy I was able to abstract and send forth only the info I needed (out of the 300+ I have), speeding up, sparing memory and getting out of the exception.

Please comment and let me know what do you think about the problem I faced and the solution/path I took.


Luciano Evaristo Guerche

Senior Systems Analyst at Banco Daycoval

9 个月
回复

要查看或添加评论,请登录

Luciano Evaristo Guerche的更多文章

社区洞察

其他会员也浏览了