How can you collect IoT device data effectively?

Defining clear data goals is a fundamental step before collecting IoT device data. These goals serve as your compass, helping you determine what questions you aim to answer, problems to solve, or value to create through the data. Your data goals provide critical guidance for decisions on what specific data to collect, the methods of collection, storage, and analysis. For instance, if your objective is to optimize the energy consumption of smart home devices, your data goals would lead you to collect data on power usage, environmental conditions, user behavior, and relevant external factors.

In addition to comprehending end-user challenges and gathering relevant data, it's essential to emphasize the critical nature of collecting data from devices, as this is pivotal for future diagnostics and early-stage understanding of field issues. Sometimes, adjustments on the device side become imperative to ensure the timely and accurate acquisition of necessary data.

In my case I am the leader of a SaaS distributor, we perform marketing, sales, onboarding and support for our customers. My goals are typical of most businesses: nurture high company culture, retain existing customers, achieve new sales and deliver customer excellence.

Selecting the appropriate data sources is a critical step once you've defined your data goals. IoT devices generate various types of data, including numeric, categorical, text, image, audio, video, and geospatial. Choosing relevant, reliable, and available data sources that align with your data goals is essential. For example, if your objective is to detect anomalies in industrial equipment, collecting data from sensors measuring temperature, pressure, vibration, or sound is relevant. Besides, considering factors like the frequency, volume, and quality of data from each source is vital for ensuring that your data collection efforts are both effective and efficient.

For our business goals it is necessary to measure: ? Team member performance and job satisfaction ? Customer solution utilisation and satisfaction ? Website visits, forms completed, marketing and sales emails response rate, and meetings booked ? Overall growth of customer base both in average use (i.e. renewal value), and number of retained customers per annum.

We have implemented a packaged solution across our organisation that provides CRM, CMS, advertising, social media, helpdesk, feedback forms and more. All of these modules include extensive reporting and dashboard display of real time data. I consider this "internal" data. We also have a range of other business intelligence solutions that track generalised customer interest in our business segment, growth in potential marketplaces and more. Again we can report extensively on this BI. I consider this "external" data. By aggregating these two forms of data I aim to provide more predictive and ultimately prescriptive reporting. For this an external databasics such as Spark may be used.

Designing a robust data architecture is indeed essential when working with IoT data. This framework shapes how data is collected, processed, stored, and accessed. Several key considerations come into play during this process

After designing your data architecture, the next step is implementing your data pipeline. This pipeline executes your architecture and delivers data from IoT devices to consumers. Choose appropriate technologies (e.g., AWS, Azure, Google Cloud), test thoroughly, monitor continuously, and optimize for performance, reliability, and scalability.

Consider an IoT use case pipeline developed using Microsoft Azure where information from devices is ingested through Azure IoT Hub and processed in real-time using Azure Stream Analytics or Databricks. Processed data is stored in Azure Data Lake Storage, Table storage or Cosmos DB, then analyzed and visualized using Azure Machine Learning, Azure Synapse Analytics, and Power BI. The pipeline is monitored through Azure Monitor and secured by Azure Security Center, ensuring compliance with organizational standards via Azure Policy and Azure Blueprint.

For the two BI platforms that we have implemented, see above, Internal and External; both these platforms are delivered on their own cloud based platforms. For the aggregation of these two data sources we will use AWS based platform. For our predictive and prescriptive data analysis and reporting, this information needs to be available in near time, e.g. updated weekly.

To evaluate data outcomes effectively, use relevant metrics like KPIs and create clear dashboards and reports. Compare results with data goals to spot discrepancies and identify areas for improvement in your data collection and analysis processes.

Our Objective Planning Procedure uses ? Gap Analysis to determine what needs to be measured and what metrics are required to measure ? PCDA process to continually perform planning, implementation and improvement of our processes. This applies across all our business systems and in particular would be used to implement and evaluate the effectiveness of the final analytics, and delivery of indicative business information.