Machine Learning Algorithms & Implementations
Philip Armstrong
Senior Project Manager @ Pioneer Business Solutions | Risk Management, Business Proposals
Types of machine learning is divided into mainly four types, which are:
1. Supervised Machine Learning
As its name suggests, Supervised machine learning is based on supervision. It means in the supervised learning technique, we train the machines using the "labelled" dataset, and based on the training, the machine predicts the output. Here, the labelled data specifies that some of the inputs are already mapped to the output. More preciously, we can say; first, we train the machine with the input and corresponding output, and then we ask the machine to predict the output using the test dataset.
Let's understand supervised learning with an example. Suppose we have an input dataset of cats and dog images. So, first, we will provide the training to the machine to understand the images, such as the shape & size of the tail of cat and dog, Shape of eyes, colour, height (dogs are taller, cats are smaller), etc. After completion of training, we input the picture of a cat and ask the machine to identify the object and predict the output. Now, the machine is well trained, so it will check all the features of the object, such as height, shape, colour, eyes, ears, tail, etc., and find that it's a cat. So, it will put it in the Cat category. This is the process of how the machine identifies the objects in Supervised Learning.
The main goal of the supervised learning technique is to map the input variable(x) with the output variable(y). Some real-world applications of supervised learning are Risk Assessment, Fraud Detection, Spam filtering, etc.
Categories of Supervised Machine Learning
Supervised machine learning can be classified into two types of problems, which are given below:
a) Classification
Classification algorithms are used to solve the classification problems in which the output variable is categorical, such as "Yes" or No, Male or Female, Red or Blue, etc. The classification algorithms predict the categories present in the dataset. Some real-world examples of classification algorithms are Spam Detection, Email filtering, etc.
Some popular classification algorithms are given below:
b) Regression
Regression algorithms are used to solve regression problems in which there is a linear relationship between input and output variables. These are used to predict continuous output variables, such as market trends, weather prediction, etc.
Some popular Regression algorithms are given below:
Advantages:
Since supervised learning work with the labelled dataset so we can have an exact idea about the classes of objects.
These algorithms are helpful in predicting the output on the basis of prior experience.
Disadvantages:
These algorithms are not able to solve complex tasks.
It may predict the wrong output if the test data is different from the training data.
It requires lots of computational time to train the algorithm.
Applications of Supervised Learning
Some common applications of Supervised Learning are given below:
Image Segmentation:
Supervised Learning algorithms are used in image segmentation. In this process, image classification is performed on different image data with pre-defined labels.
Medical Diagnosis:
Supervised algorithms are also used in the medical field for diagnosis purposes. It is done by using medical images and past labelled data with labels for disease conditions. With such a process, the machine can identify a disease for the new patients.
Fraud Detection:
Supervised Learning classification algorithms are used for identifying fraud transactions, fraud customers, etc. It is done by using historic data to identify the patterns that can lead to possible fraud.
Spam detection:
In spam detection & filtering, classification algorithms are used. These algorithms classify an email as spam or not spam. The spam emails are sent to the spam folder.
Speech Recognition:
Supervised learning algorithms are also used in speech recognition. The algorithm is trained with voice data, and various identifications can be done using the same, such as voice-activated passwords, voice commands, etc.
2. Unsupervised Machine Learning
Unsupervised learning is different from the Supervised learning technique; as its name suggests, there is no need for supervision. It means, in unsupervised machine learning, the machine is trained using the unlabeled dataset, and the machine predicts the output without any supervision.
In unsupervised learning, the models are trained with the data that is neither classified nor labelled, and the model acts on that data without any supervision.
The main aim of the unsupervised learning algorithm is to group or categories the unsorted dataset according to the similarities, patterns, and differences. Machines are instructed to find the hidden patterns from the input dataset.
Let's take an example to understand it more preciously; suppose there is a basket of fruit images, and we input it into the machine learning model. The images are totally unknown to the model, and the task of the machine is to find the patterns and categories of the objects.
So, now the machine will discover its patterns and differences, such as colour difference, shape difference, and predict the output when it is tested with the test dataset.
Categories of Unsupervised Machine Learning
Unsupervised Learning can be further classified into two types, which are given below:
1) Clustering
The clustering technique is used when we want to find the inherent groups from the data. It is a way to group the objects into a cluster such that the objects with the most similarities remain in one group and have fewer or no similarities with the objects of other groups. An example of the clustering algorithm is grouping the customers by their purchasing behaviour.
Some of the popular clustering algorithms are given below:
2) Association
Association rule learning is an unsupervised learning technique, which finds interesting relations among variables within a large dataset. The main aim of this learning algorithm is to find the dependency of one data item on another data item and map those variables accordingly so that it can generate maximum profit. This algorithm is mainly applied in Market Basket analysis, Web usage mining, continuous production, etc.
Some popular algorithms of Association rule learning are Apriori Algorithm, Eclat, FP-growth algorithm.
Advantages and Disadvantages of Unsupervised Learning Algorithm
Advantages:
These algorithms can be used for complicated tasks compared to the supervised ones because these algorithms work on the unlabeled dataset.
Unsupervised algorithms are preferable for various tasks as getting the unlabeled dataset is easier as compared to the labelled dataset.
Disadvantages:
The output of an unsupervised algorithm can be less accurate as the dataset is not labelled, and algorithms are not trained with the exact output in prior.
Working with Unsupervised learning is more difficult as it works with the unlabelled dataset that does not map with the output.
Applications of Unsupervised Learning
Network Analysis: Unsupervised learning is used for identifying plagiarism and copyright in document network analysis of text data for scholarly articles.
Recommendation Systems: Recommendation systems widely use unsupervised learning techniques for building recommendation applications for different web applications and e-commerce websites.
Anomaly Detection: Anomaly detection is a popular application of unsupervised learning, which can identify unusual data points within the dataset. It is used to discover fraudulent transactions.
Singular Value Decomposition: Singular Value Decomposition or SVD is used to extract particular information from the database. For example, extracting information of each user located at a particular location.
3. Semi-Supervised Learning
Semi-Supervised learning is a type of Machine Learning algorithm that lies between Supervised and Unsupervised machine learning. It represents the intermediate ground between Supervised (With Labelled training data) and Unsupervised learning (with no labelled training data) algorithms and uses the combination of labelled and unlabeled datasets during the training period.
Although Semi-supervised learning is the middle ground between supervised and unsupervised learning and operates on the data that consists of a few labels, it mostly consists of unlabeled data. As labels are costly, but for corporate purposes, they may have few labels. It is completely different from supervised and unsupervised learning as they are based on the presence & absence of labels.
To overcome the drawbacks of supervised learning and unsupervised learning algorithms, the concept of Semi-supervised learning is introduced. The main aim of semi-supervised learning is to effectively use all the available data, rather than only labelled data like in supervised learning. Initially, similar data is clustered along with an unsupervised learning algorithm, and further, it helps to label the unlabeled data into labelled data. It is because labelled data is a comparatively more expensive acquisition than unlabeled data.
We can imagine these algorithms with an example. Supervised learning is where a student is under the supervision of an instructor at home and college. Further, if that student is self-analysing the same concept without any help from the instructor, it comes under unsupervised learning. Under semi-supervised learning, the student has to revise himself after analyzing the same concept under the guidance of an instructor at college.
Advantages and disadvantages of Semi-supervised Learning
Advantages:
It is simple and easy to understand the algorithm.
It is highly efficient.
It is used to solve drawbacks of Supervised and Unsupervised Learning algorithms.
Disadvantages:
Iterations results may not be stable.
We cannot apply these algorithms to network-level data.
Accuracy is low.
4. Reinforcement Learning
Reinforcement learning works on a feedback-based process, in which an AI agent (A software component) automatically explore its surrounding by hitting & trail, taking action, learning from experiences, and improving its performance. Agent gets rewarded for each good action and get punished for each bad action; hence the goal of reinforcement learning agent is to maximize the rewards.
In reinforcement learning, there is no labelled data like supervised learning, and agents learn from their experiences only.
The reinforcement learning process is similar to a human being; for example, a child learns various things by experiences in his day-to-day life. An example of reinforcement learning is to play a game, where the Game is the environment, moves of an agent at each step define states, and the goal of the agent is to get a high score. Agent receives feedback in terms of punishment and rewards.
Due to its way of working, reinforcement learning is employed in different fields such as Game theory, Operation Research, Information theory, multi-agent systems.
A reinforcement learning problem can be formalized using Markov Decision Process(MDP). In MDP, the agent constantly interacts with the environment and performs actions; at each action, the environment responds and generates a new state.
Categories of Reinforcement Learning
Reinforcement learning is categorized mainly into two types of methods/algorithms:
Positive Reinforcement Learning: Positive reinforcement learning specifies increasing the tendency that the required behaviour would occur again by adding something. It enhances the strength of the behaviour of the agent and positively impacts it.
Negative Reinforcement Learning: Negative reinforcement learning works exactly opposite to the positive RL. It increases the tendency that the specific behaviour would occur again by avoiding the negative condition.
Real-world Use cases of Reinforcement Learning
Video Games:
RL algorithms are much popular in gaming applications. It is used to gain super-human performance. Some popular games that use RL algorithms are AlphaGO and AlphaGO Zero.
Resource Management:
The "Resource Management with Deep Reinforcement Learning" paper showed that how to use RL in computer to automatically learn and schedule resources to wait for different jobs in order to minimize average job slowdown.
Robotics:
RL is widely being used in Robotics applications. Robots are used in the industrial and manufacturing area, and these robots are made more powerful with reinforcement learning. There are different industries that have their vision of building intelligent robots using AI and Machine learning technology.
Text Mining
Text-mining, one of the great applications of NLP, is now being implemented with the help of Reinforcement Learning by Salesforce company.
Advantages and Disadvantages of Reinforcement Learning
Advantages
It helps in solving complex real-world problems which are difficult to be solved by general techniques.
The learning model of RL is similar to the learning of human beings; hence most accurate results can be found.
Helps in achieving long term results.
Disadvantage
RL algorithms are not preferred for simple problems.
RL algorithms require huge data and computations.
Too much reinforcement learning can lead to an overload of states which can weaken the results.
Feature Engineering for Machine Learning
Feature engineering is the pre-processing step of machine learning, which is used to transform raw data into features that can be used for creating a predictive model using Machine learning or statistical Modelling. Feature engineering in machine learning aims to improve the performance of models. In this topic, we will understand the details about feature engineering in Machine learning. But before going into details, let's first understand what features are? And What is the need for feature engineering?
Feature Engineering for Machine Learning
What is a feature?
Generally, all machine learning algorithms take input data to generate the output. The input data remains in a tabular form consisting of rows (instances or observations) and columns (variable or attributes), and these attributes are often known as features. For example, an image is an instance in computer vision, but a line in the image could be the feature. Similarly, in NLP, a document can be an observation, and the word count could be the feature. So, we can say a feature is an attribute that impacts a problem or is useful for the problem.
What is Feature Engineering?
Feature engineering is the pre-processing step of machine learning, which extracts features from raw data. It helps to represent an underlying problem to predictive models in a better way, which as a result, improve the accuracy of the model for unseen data. The predictive model contains predictor variables and an outcome variable, and while the feature engineering process selects the most useful predictor variables for the model.
Feature Engineering for Machine Learning
Since 2016, automated feature engineering is also used in different machine learning software that helps in automatically extracting features from raw data. Feature engineering in ML contains mainly four processes: Feature Creation, Transformations, Feature Extraction, and Feature Selection.
These processes are described as below:
Feature Creation: Feature creation is finding the most useful variables to be used in a predictive model. The process is subjective, and it requires human creativity and intervention. The new features are created by mixing existing features using addition, subtraction, and ration, and these new features have great flexibility.
领英推荐
Transformations: The transformation step of feature engineering involves adjusting the predictor variable to improve the accuracy and performance of the model. For example, it ensures that the model is flexible to take input of the variety of data; it ensures that all the variables are on the same scale, making the model easier to understand. It improves the model's accuracy and ensures that all the features are within the acceptable range to avoid any computational error.
Feature Extraction: Feature extraction is an automated feature engineering process that generates new variables by extracting them from the raw data. The main aim of this step is to reduce the volume of data so that it can be easily used and managed for data modelling. Feature extraction methods include cluster analysis, text analytics, edge detection algorithms, and principal components analysis (PCA).
Feature Selection: While developing the machine learning model, only a few variables in the dataset are useful for building the model, and the rest features are either redundant or irrelevant. If we input the dataset with all these redundant and irrelevant features, it may negatively impact and reduce the overall performance and accuracy of the model. Hence it is very important to identify and select the most appropriate features from the data and remove the irrelevant or less important features, which is done with the help of feature selection in machine learning. "Feature selection is a way of selecting the subset of the most relevant features from the original features set by removing the redundant, irrelevant, or noisy features."
Below are some benefits of using feature selection in machine learning:
It helps in avoiding the curse of dimensionality.
It helps in the simplification of the model so that the researchers can easily interpret it.
It reduces the training time.
It reduces overfitting hence enhancing the generalization.
Need for Feature Engineering in Machine Learning
In machine learning, the performance of the model depends on data pre-processing and data handling. But if we create a model without pre-processing or data handling, then it may not give good accuracy. Whereas, if we apply feature engineering on the same model, then the accuracy of the model is enhanced. Hence, feature engineering in machine learning improves the model's performance. Below are some points that explain the need for feature engineering:
Better features mean flexibility.
In machine learning, we always try to choose the optimal model to get good results. However, sometimes after choosing the wrong model, still, we can get better predictions, and this is because of better features. The flexibility in features will enable you to select the less complex models. Because less complex models are faster to run, easier to understand and maintain, which is always desirable.
Better features mean simpler models.
If we input the well-engineered features to our model, then even after selecting the wrong parameters (Not much optimal), we can have good outcomes. After feature engineering, it is not necessary to do hard for picking the right model with the most optimized parameters. If we have good features, we can better represent the complete data and use it to best characterize the given problem.
Better features mean better results.
As already discussed, in machine learning, as data we will provide will get the same output. So, to obtain better results, we must need to use better features.
Steps in Feature Engineering
The steps of feature engineering may vary as per different data scientists and ML engineers. However, there are some common steps that are involved in most machine learning algorithms, and these steps are as follows:
Data Preparation: The first step is data preparation. In this step, raw data acquired from different resources are prepared to make it in a suitable format so that it can be used in the ML model. The data preparation may contain cleaning of data, delivery, data augmentation, fusion, ingestion, or loading.
Exploratory Analysis: Exploratory analysis or Exploratory data analysis (EDA) is an important step of features engineering, which is mainly used by data scientists. This step involves analysis, investing data set, and summarization of the main characteristics of data. Different data visualization techniques are used to better understand the manipulation of data sources, to find the most appropriate statistical technique for data analysis, and to select the best features for the data.
Benchmark: Benchmarking is a process of setting a standard baseline for accuracy to compare all the variables from this baseline. The benchmarking process is used to improve the predictability of the model and reduce the error rate.
Feature Engineering Techniques
Some of the popular feature engineering techniques include:
1. Imputation
Feature engineering deals with inappropriate data, missing values, human interruption, general errors, insufficient data sources, etc. Missing values within the dataset highly affect the performance of the algorithm, and to deal with them "Imputation" technique is used. Imputation is responsible for handling irregularities within the dataset.
For example, removing the missing values from the complete row or complete column by a huge percentage of missing values. But at the same time, to maintain the data size, it is required to impute the missing data, which can be done as:
For numerical data imputation, a default value can be imputed in a column, and missing values can be filled with means or medians of the columns.
For categorical data imputation, missing values can be interchanged with the maximum occurred value in a column.
2. Handling Outliers
Outliers are the deviated values or data points that are observed too away from other data points in such a way that they badly affect the performance of the model. Outliers can be handled with this feature engineering technique. This technique first identifies the outliers and then remove them out.
Standard deviation can be used to identify the outliers. For example, each value within a space has a definite to an average distance, but if a value is greater distant than a certain value, it can be considered as an outlier. Z-score can also be used to detect outliers.
3. Log transform
Logarithm transformation or log transform is one of the commonly used mathematical techniques in machine learning. Log transform helps in handling the skewed data, and it makes the distribution more approximate to normal after transformation. It also reduces the effects of outliers on the data, as because of the normalization of magnitude differences, a model becomes much robust.
Note: Log transformation is only applicable for the positive values; else, it will give an error. To avoid this, we can add 1 to the data before transformation, which ensures transformation to be positive.
4. Binning
In machine learning, overfitting is one of the main issues that degrade the performance of the model and which occurs due to a greater number of parameters and noisy data. However, one of the popular techniques of feature engineering, "binning", can be used to normalize the noisy data. This process involves segmenting different features into bins.
5. Feature Split
As the name suggests, feature split is the process of splitting features intimately into two or more parts and performing to make new features. This technique helps the algorithms to better understand and learn the patterns in the dataset.
The feature splitting process enables the new features to be clustered and binned, which results in extracting useful information and improving the performance of the data models.
6. One hot encoding
One hot encoding is the popular encoding technique in machine learning. It is a technique that converts the categorical data in a form so that they can be easily understood by machine learning algorithms and hence can make a good prediction. It enables group the of categorical data without losing any information.
Here is a list of popular machine learning algorithms:
1. Linear Regression
Description: A regression algorithm used to predict a continuous target variable based on one or more input features. It assumes a linear relationship between the input variables and the output.
Use Case: Predicting house prices based on features like size, number of bedrooms, and location.
2. Logistic Regression
Description: A classification algorithm used to predict binary outcomes (0 or 1, true or false) based on one or more input features. Despite its name, it is used for classification, not regression.
Use Case: Determining whether an email is spam or not.
3. Decision Trees
Description: A versatile algorithm that can be used for both classification and regression tasks. It works by splitting the data into subsets based on the value of input features.
Use Case: Classifying whether a patient has a certain disease based on medical test results.
4. Random Forests
Description: An ensemble method that uses multiple decision trees to improve the robustness and accuracy of the prediction. It reduces overfitting by averaging the results of various trees.
Use Case: Predicting loan defaults based on customer data.
5. Support Vector Machines (SVM)
Description: A classification algorithm that finds the hyperplane which best separates different classes in the feature space. It can also be used for regression (SVR).
Use Case: Image classification tasks, such as handwriting recognition.
6. K-Nearest Neighbors (KNN)
Description: A simple, instance-based learning algorithm used for classification and regression. It assigns the output based on the majority vote of the nearest k instances in the feature space.
Use Case: Recommending products to users based on their past preferences.
7. Naive Bayes
Description: A probabilistic classifier based on Bayes' theorem, assuming independence between the features. It is simple and effective for many applications.
Use Case: Text classification, such as spam detection and sentiment analysis.
8. Gradient Boosting Machines (GBM)
Description: An ensemble technique that builds multiple decision trees sequentially, where each tree tries to correct the errors of the previous one. It is used for both classification and regression.
Use Case: Predicting customer churn rates.
9. AdaBoost
Description: An ensemble method that combines multiple weak classifiers to form a strong classifier. It focuses on the instances that were misclassified by previous classifiers.
Use Case: Improving the accuracy of classification tasks, such as facial recognition.
10. XGBoost
Description: An optimized implementation of gradient boosting, which is efficient, scalable, and provides high performance. It is widely used in machine learning competitions.
Use Case: Winning Kaggle competitions, such as predicting flight delays.
11. LightGBM
Description: A gradient boosting framework that uses tree-based learning algorithms. It is designed for efficiency and speed, especially with large datasets.
Use Case: Large-scale machine learning tasks, such as ranking and recommendation systems.
12. CatBoost
Description: A gradient boosting algorithm that handles categorical features automatically and efficiently. It is designed to be fast and accurate.
Use Case: Predicting customer behavior in e-commerce.
13. K-Means Clustering
Description: An unsupervised learning algorithm that partitions the data into k clusters, with each cluster having a centroid representing the average of its points.
Use Case: Customer segmentation based on purchasing behavior.
14. Hierarchical Clustering
Description: An unsupervised learning algorithm that builds a hierarchy of clusters either through a bottom-up approach (agglomerative) or a top-down approach (divisive).
Use Case: Gene expression data analysis.
15. Principal Component Analysis (PCA)
Description: A dimensionality reduction technique that transforms the data into a new coordinate system, with the greatest variances along the first few principal components.
Use Case: Reducing the dimensionality of image data for compression.
16. Neural Networks
Description: A set of algorithms modeled after the human brain, used for complex pattern recognition tasks. It consists of interconnected layers of nodes (neurons).
Use Case: Handwriting recognition, speech recognition.
17. Convolutional Neural Networks (CNN)
Description: A type of neural network particularly effective for image recognition tasks. It uses convolutional layers to automatically and adaptively learn spatial hierarchies of features.
Use Case: Object detection in images, image classification.
18. Recurrent Neural Networks (RNN)
Description: A type of neural network designed for sequential data. It maintains a memory of previous inputs, making it suitable for time series and natural language processing tasks.
Use Case: Language translation, speech recognition.
19. Long Short-Term Memory Networks (LSTM)
Description: An advanced type of RNN that can learn long-term dependencies and avoid the vanishing gradient problem. It is effective for tasks involving sequences.
Use Case: Predicting stock prices, language modeling.
20. Autoencoders
Description: A type of neural network used for unsupervised learning, primarily for the purpose of dimensionality reduction and feature learning. It consists of an encoder and a decoder.
Use Case: Anomaly detection, data denoising.
21. Gaussian Mixture Models (GMM)
Description: A probabilistic model for representing the presence of subpopulations within an overall population, using a mixture of multiple Gaussian distributions.
Use Case: Clustering, density estimation.
22. Hidden Markov Models (HMM)
Description: A statistical model used for modeling sequential data, where the system being modeled is assumed to be a Markov process with hidden states.
Use Case: Speech recognition, bioinformatics.
23. Reinforcement Learning Algorithms (Q-Learning, SARSA)
Description: A set of algorithms where an agent learns to make decisions by performing actions in an environment to maximize some notion of cumulative reward.
Use Case: Game playing, robotics.
24. Association Rule Learning (Apriori, Eclat)
Description: Algorithms are used to discover interesting relations between variables in large datasets. They are often used in market basket analysis.
Use Case: Identifying frequently bought together products.
25. Dimensionality Reduction Techniques (t-SNE, UMAP)
Description: Techniques used to reduce the number of random variables under consideration, by obtaining a set of principal variables.
Use Case: Visualizing high-dimensional data in 2D or 3D.
Founder of SmythOS.com | AI Multi-Agent Orchestration ??
6 个月Future tech creates opportunities, not panic. Human creativity remains irreplaceable.