What data collection method is best for your machine learning project?

As you know, the data have to be stored somewhere in a reliable order which is called databae. For example, SQL Server, Oracle and etc. are some kinds of famous databases. One important point here is that it is not logical to give all the people the capability to reach our database directly. So, We could use some Python libraries for creation a connection string to our data source. For example, we could use PYODBC library in Python to connect a designed form to a SQL server data base and implement the CRUD to our data base by Python programming.

I will go with Surveys and questionnaires. - This method is a primary source of data that seems to be accurate and present things in the original forms. - Making surveys and questionnaires anonymous (truly anonymous) can increase the accuracy of the data and also lead to more responses from individuals as people need not worry about revealing too much information. - Potential problems might that affect data quality from this source are: question quality/clarity and question designs that sometimes don't totally capture real-life scenarios as the designer might have a particular design in mind to achieve the project objectives.

Data collection methods include surveys/questionnaires for structured responses, observational studies for natural behavior observation, and interviews for qualitative insights. Experiments manipulate variables under controlled conditions, while case studies offer in-depth analysis of specific instances. Field trials implement interventions in real-world settings, and observational coding categorizes observed behaviors systematically. Social media scraping and web scraping extract insights from online platforms, while sensor data collection uses devices like IoT for monitoring. Choose methods based on research goals, available resources, and ethical considerations.

There are varius tools in the market which is good for data collection puropse but out of these various tools I like Azure Data Factory the most as it has many connectors available which is code free with the help of which we can connect to multiple sources and collect the data in any storage location like Azue Data Lake Gen2.This ADF is a low code ETL tool of Microsoft whose popularity is increasing day by day.One more good thing about this tool is that it is present in cloud paltfom.

The data collection method mostly depends on the type of ML project that one is undertaking, if it is sentiment analysis, you have to collect data using web scraping tools. If the suppose the data is in a repository or in a data base then SQLlite or PyMongo are some of the tools. If suppose you want to build a custom algo trading strategy then you have to collect data from APIs. However all of these methods presuppose that their is data available for the project. What if we are unsure about data availability? Then again we have to fall back on statistical methods for collecting and curating the data.

I’ve used BeautifulSoup for my project, but it falls short when faced with even simple anti-scraping mechanism. Selenium solves this problem by imitating a “user” in a browser session. However, user intervention is need when the website implements captcha-based measures. Automation software such as UiPath are also a great if the user intends to add robotic process automation (RPA) into their skill set.

Scrapy is useful when data is hide within some CSS and HTML protocols in a website. So, we could use it to extract and manipulate the data. Hopefully, Python enables us to perform such tasks so easily by using a wide variety of libraries like BeautifulSoap, Selenium and etc. After scarping the data, it is also possible to store and modify the data in data dictionary format.

Web scraping has some quite evident advantages, such as time effectiveness, cost effective, requires less effort, more accurate and fact-based, more consistent, real-time without lags. Some of the disadvantages comprise of legal and ethical concerns, technical challenges, website changes, sublinks in the websites, and scraped data might have data quality issues.

In my experience scrapy is the best for middle sized projects (about 50-100 k rows) .It has less boiler-plate and async backend.Sometimes it can be hard to learning but trust me when you gain small experience in this framework you never want to look back (BeautifulSoup,Selenium).

For industrial grade scraper, use Scrapy. It's no debate to adhere to ethical guidelines. If you're doing large scale scraping, always 1. implement Proxy/ IP rotation 2. Set a real user-agent 3. Set headers 4. Set referrer (Where this traffic is coming from - Google/ Other search engine etc) 5. Most importantly, time your requests at random interval But it's super important to - 1. Respect privacy of the website you're scraping and scrape what's allowed 2. Not overflood the traffic with your requests

I think that another important challenge for these kinds of devices is their requirements to a very enormous and powerful hardware to detect, store and process the data. For example, huge amounts of data in our company are provided in print hard-copy version which is not useful for any processing. Using convolutional neural networks, they could be processed and transformed to suitable digital versions and after that be classified based on those contents. Such projects are interesting but as I mentioned, they are really expensive and require high amounts of resources.

Using sensors and devices to gather data is like having digital eyes and ears in the world. They can pick up all sorts of details—how hot it is, how much it's raining, if there's movement, or even what sounds are around. This tech is super handy for smart homes, making machines see, or letting computers understand speech. But it's not all smooth sailing; these gadgets can be pricey, need looking after, and you have to keep them secure. Plus, you've got to figure out where to keep all the data they collect and how to make sense of it.

It all depends on the type of project, different types of projects require different methods. Well, In my case data collected from sensors and physical devices is useful. Because the project is related to Heart Attack prediction and a pre-alert.

Once, a manufacturing company was struggling with inefficiencies in their production line. They implemented SAP MII (Manufacturing Integration and Intelligence) to integrate their shop floor with their enterprise systems. However, they lacked real-time data for decision-making. To solve this, they connected sensors to their machinery and used a historian database to capture the data. The historian provided a rich, time-stamped record of machine performance. With SAP MII and the historian working together, they could analyze trends, predict maintenance, and optimize operations. The outcome was a remarkable improvement in efficiency and a reduction in downtime, showcasing the power of integrating real-time sensor data with SAP MII.

- Sensor data is critical for preventive maintenance of large machines (large printers in printing presses, boilers, large coolers in plants etc.), self driven cars etc. Data volume of sensors are high, so can’t build POCs with google colab free subscription etc. For minimising computational cost, we can aggregate data at slightly higher levels. For example if sensor captures 10 data points per minute, value of these 10 points can be aggregated to reduce data by 10 times.

APIs (Application Programming Interfaces) and databases provide structured access to digital information. APIs are like bridges that allow one software application to communicate with another, while databases are organized repositories of data. Both are essential for retrieving data from online services or databases, such as social media platforms, e-commerce sites, or government records. They are crucial for applications like data analysis, software integration, or research, providing a reliable and organized source of information.

Understanding the scope and intricate details of a project is crucial and indeed precedes the selection of data collection methods. As a Data Analyst, one often navigates through diverse data sources ranging from databases and APIs provided by technical teams, to various company-specific Excel sheets or cloud-based spreadsheets. The choice of data collection method for your machine learning project should be strategically informed by several key factors, including the available resources and the overarching objectives of the project. It's not uncommon to combine multiple data collection techniques to compile a robust and valid dataset that aligns perfectly with your analytical requirements

Leveraging an API that creates a formulaic means of accessing data at scale for your machine-learning application can be one of the best methods of gathering data because of the beauty of automation. The issue with most APIs however, is finding one that is frequently updated and active. Due to the nature of open-source technology, you can find an API that does just about anything for your use case, enabling you to focus on building innovative new technologies. This also has the caveat that not all APIs are built equally and some may employ code that could bring ruin to your applications. Take the time to research an API in depth before committing to using one in production-scale apps.

Python offers a rich ecosystem of libraries for data visualization, each with its unique strengths and use cases. Here are some of the most popular Python libraries for data visualization: Matplotlib: One of the most popular Python libraries for creating static, animated, and interactive visualizations. It is highly customizable and serves as the foundation for many other libraries. Seaborn: Built on top of Matplotlib, Seaborn provides a high-level interface for drawing attractive and informative statistical graphics. It's particularly good for making complex plots more accessible. Plotly: This library is known for its interactive plots

Another method to collect data is from a storage space named databases where the data can be readily available or one have to do certain operations on the db to prepare data like joining multiple table(SQL) / collection (mongo) to create the data in some format . There are few api's available where we can connect to some specific DB to pull data in

Data augmentation is a process of expanding an existing dataset by creating variations of the original data. It involves techniques like rotating, flipping, or adjusting images to generate additional examples for training machine learning models. Data generation, on the other hand, involves creating entirely new data instances. This can be done using techniques like Generative Adversarial Networks (GANs) or by simulating data based on specific rules or patterns. Both data augmentation and generation are helpful when a machine learning model requires more diverse and abundant data for training, enhancing the model's accuracy and robustness.

Data augmentation is one of the method we use to increase the quantity of the data . let's say you are working on a text classification problem where there are 5 labels but assume 4 labels are almost of equal data points but one doesn't so in that case we use some synthetic data generation to increase the data point for the less label category , these techniques works for some competition or hackathon but for a real world problem it does not work as expected . There is a famous library for image data augmentation named albumentations which works really well for image based data augmentation .

Data augmentation is a technique to increase the diversity of your data. It involves making slight alterations to your existing dataset, such as rotating images, adjusting brightness, or cropping, to mimic the variance found in real-world data. It helps prevent overfitting and improves model robustness. Data generation, particularly with generative models like GANs creates new data samples from scratch that are similar to your original dataset. This is useful when data collection is challenging or privacy concerns limit the use of real data. Both methods are crucial for improving model performance, especially in domains like computer vision and natural language processing, where large and varied datasets are vital.

- I am careful in doing data augmentation and data generation, haven’t worked for me. However, one situation where I use data generation is when there are multiple classes in data, and data of one class or couple of classes is very low. Thats when I would synthetically generate data to increase volume of data of classes with low data volume.

Augmented or synthetically generated data can be useful when real data is limited in number, expensive, time-consuming or legally/ethically challenging to collect. While it may be suitable for prototyping, the data generated through augmentation may not be representative of the real data for various reasons. It takes domain knowledge to synthetically create or augment data from the existing ones. As an example, I can think of a Convolutional Neural Network, which requires thousands of images to train on. Data scientists often augment the images by rotating, flipping, changing color or contrast, and so on. The same goes for an audio classification project where you can add high/low noise, shift pitch, add high/low pass filters, etc.

When collecting data for a machine learning project, it's crucial to consider several factors: Data Relevance Data Quality Bias and Fairness Data Privacy Sample Size Data Diversity Data Collection Methods Data Versioning Data Storage and Management Ethical Considerations Data Labeling Data Annotation Data Acquisition Costs Data Licensing Data Validation Data Updates Data Documentation Feedback Loop Data Security Data Cleaning and Preprocessing In summary, successful data collection is a multi-faceted process that involves careful planning, ethical considerations, and attention to data quality. It's a critical foundation for building effective machine learning models.

The Data Collection technique depends on the domain of the project and it's specific tasks. For instance if your machine learning project is IoT based, like real time environment/weather monitoring data is collected using environmental sensors. For applications related to neuroimaging, data is often collected using imaging techniques that can capture brain activity, for instance MRI, fMRI or PET scans. This data is collected from willing participants & patients, often involves ethical & safety considerations. For a content moderation or recommender system, you need to acquire crowdsourced data using platforms like Amazon mechanical turk. More importantly you should ensure the data is clean, labelled & relevant to the problem at hand.

More important than where to extract the data from, is to analyze the data, prepare, and clean it. First of all, machine learning needs a big dataset, leaving data out to test later, if the quantity is not enough the model may be deficient. The data needs to be well-distributed. For instance, if you are training a model to detect failures, and 99% of the data is correct (when in reality the error is closer to 5% or 10%), or on the contrary, the error on the sample is 25%, then the model won't be very accurate. It requires much thinking to process the data to feed the algorithm, but simultaneously you prefer to avoid excessive modifications because the same alterations will have to be applied to every data we process in the model.

Data collection in the real world needs to be a dynamic process, because data tends to drift over time. This requires, ideally, a properly formulated plan to refresh data periodically, ensure that it captures the current essence of the underlying natural phenomenon, and continually modifying the machine learning model to reflect the new reality.

In real-world situations, there isn't a one-size-fits-all approach or method to data collection. instead, you encounter diverse sources. These sources can vary, with some being structured, such as databases, JSON, Excel, and CSV files, while others may be unstructured, like PDFs and logs