Rebuttal: Need More IoT Data? Try Stream Processing
While the value (or at least promise) of stream processing appears valuable indeed, I did notice a couple of things in Brandon's (well written) article that I feel the need to rebut (well, maybe rebut is too strong but....).
- Lack of available storage capacity: with each successive NAND flash technology advancement, solid state storage capacities are increasing within the same (even smaller) footprint and at lower $/GB. As next-gen 100+ layer 3D TLC/QLC NAND is released it won't be too long before 1TB single NAND chips are available. These devices can be packaged small enough and at low enough power to satisfy many IoT/edge applications, which brings me to the second point.
- Intel? Core? and Intel? Xeon? processors are ideally suited for stream processing: many IoT/edge applications require low operating power (some are even battery-powered) so high-powered processors aren't necessarily the best option. An alternative is a lower powered edge computer running an intelligent storage platform that can filter, process, and analyze data at the edge, on-the-fly and provide ability to react to this data in real-time (similar in concept to stream processing, I think). In addition, there are added benefits of local storage in the event that the system loses power or a network connectivity, ability to structure the filtered data into formats easily digestible by the data center, and lower network congestion and usage costs.