Astor Perkins的动态

My most recent Medium article related to outlier detection is now live. This covers the use of subspaces: creating sets of small detectors, each covering a restricted number of features, which can often allow for more accurate, faster, and more interpretable tests for outliers. https://lnkd.in/g3YF3KSa This includes an excerpt from Outlier Detection in Python https://lnkd.in/gVq-ACgJ

My next installment of articles on Medium on interpretable outlier detection is now up, https://lnkd.in/gpSDgYTM This is another sample from Outlier Detection in Python (https://lnkd.in/gVq-ACgJ), in this case presenting another interpretable algorithm called Counts Outlier Detector, based on multi-dimensional histograms. #outlierdetection #anomalydetection #XAI #machinelearning #datascience #python

Using gganimate in R (one can also do this in Python) to show what causal inference methods actually do to data: Evidence from Differences-in-Differences: Differences-in-Differences (DiD) Method: A Quick Overview for the GIF below The Differences-in-Differences (DiD) method is a powerful statistical technique used to estimate the causal effect of a treatment or intervention. It compares the changes in outcomes over time between a group that is exposed to the treatment and a group that is not. Here’s a simple breakdown: Two Groups: Identify a treatment group (exposed to the intervention) and a control group (not exposed). Two Time Periods: Measure outcomes for both groups before and after the intervention. Difference Calculation: Calculate the difference in outcomes for each group over time. Comparison: The key insight comes from comparing these differences between the two groups. This helps isolate the effect of the intervention from other factors that may just correlate with but not actually cause the difference. That's basically why it's called "the difference-in-the-difference." (DiD). Why Use DiD? Causal Inference: Helps determine if changes in outcomes can be attributed to the intervention. Control for Trends: Accounts for trends that affect both groups equally, improving accuracy. Example: Imagine a nonprofit or university implements a new training program to boost donations. By comparing donations growth before and after the program for both trained and untrained employees, DiD can help assess the program’s true impact.

The use of KNN K-Nearest Neighbors (KNN) is a simple yet powerful supervised learning algorithm used for classification and regression tasks. The core idea behind KNN is to find the 'k' most similar instances (nearest neighbors) to a new instance, and then use their labels to predict the label of the new instance. This is done by calculating the distance between the new instance and each instance in the training dataset, and then selecting the 'k' instances with the smallest distances. In Python, the most commonly used libraries to implement KNN are Scikit-learn and SciPy. Scikit-learn provides a simple and efficient implementation of KNN through its?KNeighborsClassifier?and?KNeighborsRegressor?classes. To use KNN, you need to first import the necessary libraries, load your dataset, and then create a KNN model with a specified value of 'k'. You can then fit the model to your training data and use it to make predictions on new, unseen data. SciPy is used to calculate the distances between instances, which is a crucial step in the KNN algorithm. To test the accuracy of a KNN model, you can use various metrics such as accuracy score, precision, recall, and F1 score. These metrics can be calculated using Scikit-learn's?metrics?module. For example, you can use the?accuracy_score?function to calculate the accuracy of your model, which is the proportion of correctly classified instances. You can also use techniques such as cross-validation to evaluate the performance of your model on unseen data. By tuning the value of 'k' and using different distance metrics, you can improve the accuracy of your KNN model and make more accurate predictions. using the famous Iris Dataset : https://lnkd.in/eFHipesV we can evaluate the model accuracy

Chapters 12, 13, and 14 are now available for Outlier Detection in Python. These cover: - Collective outliers - Explainable outlier detection - Creating ensembles of outlier detectors Often in outlier detection, we're concerned not just with unusual events, but with unusual sequences of events. For example, a failed password attempt may not be unusual, but many within a minute for many different user accounts may be. With credit card purchases, a single large purchase may not be unusual for the cardholder, but many within an hour may be. With staff expenses, single claims for meals may be common, but many claims for meals within the same day for the same staff may be unusual. To detect these, we use tests for collective outliers, which are explained in chapter 12. It's also common in outlier detection for it to be necessary to know why items flagged as outliers were. For example, if a series of credit card purchases were flagged as anomalous, and therefore suspicious, for anyone investigating these, it's necessary to know what is anomalous to investigate them efficiently and accurately. We cover explainable AI techniques such as feature importances, proxy models, counterfactuals, and ALE plots and describe how to apply these to outlier detection. We also cover interpretable outlier detection techniques. When performing outlier detection, it's very common to use a ensemble of outlier detectors. This is probably much more common (and necessary) than with predictive models. Each detector uses a specific algorithm, which detects a certain set of outliers. To find a wide range of the outliers present in a dataset, it's usually necessary to use multiple detectors, but combining them has some nuances we cover and explain how to deal with. https://mng.bz/KZe4 #OutlierDetection #AnomalyDetection #python #DataScience #DataAnalytics #analyitcs #MachineLearning #FraudDetection #Finance? #GoodBook #anomalies #outliers

?? Shared Nearest Neighbors (SNN) distance metric is clearly described with focus on its application to outlier detection in the below article. Thanks to Brett Kennedy for sharing a detailed work on #SNN ?? ??The article also covers quickly its #application to #prediction and #clustering, but focus on #outlier #detection, and specifically on SNN’s application to the k #Nearest #Neighbors outlier detection #algorithm. ??If your work deals with above challenges you should go and spend some time on this article and tests presented usin #Python #libraries. https://lnkd.in/eWiNREkA

Perform outlier detection more effectively using subsets of features https://ift.tt/FhlgT9o Identify relevant subspaces: subsets of features that allow you to most effectively perform outlier detection on tabular?data This article is part of a series related to the challenges, and the techniques that may be used, to best identify outliers in data, including articles related to using PCA, Distance Metric Learning, Shared Nearest Neighbors, Frequent Patterns Outlier Factor, Counts Outlier Detector (a multi-dimensional histogram-based method), and doping. This article also contains an excerpt from my book, Outlier Detection in?Python. We look here at techniques to create, instead of a single outlier detector examining all features within a dataset, a series of smaller outlier detectors, each working with a subset of the features (referred to as subspaces). Challenges with outlier detection When performing outlier detection on tabular data, we’re looking for the records in the data that are the most unusual?—?either relative to the other records in the same dataset, or relative to previous?data. There are a number of challenges associated with finding the most meaningful outliers, particularly that there is no definition of statistically unusual that definitively specifies which anomalies in the data should be considered the strongest. As well, the outliers that are most relevant (and not necessarily the most statistically unusual) for your purposes will be specific to your project, and may evolve over?time. There are also a number of technical challenges that appear in outlier detection. Among these are the difficulties that occur where data has many features. As covered in previous articles related to Counts Outlier Detector and Shared Nearest Neighbors, where we have many features, we often face an issue known as the curse of dimensionality. This has a number of implications for outlier detection, including that it makes distance metrics unreliable. Many outlier detection algorithms rely on calculating the distances between records?—?in order to identify as outliers the records that are similar to unusually few other records, and that are unusually different from most other records?—?that is, records that are close to few other records and far from most other?records. For example, if we have a table with 40 features, each record in the data may be viewed as a point in 40-dimensional space, and its outlierness can be evaluated by the distances from it to the other points in this space. This, then, requires a way to measure the distance between records. A variety of measures are used, with Euclidean distances being quite common (assuming the data is numeric, or is converted to numeric values). So, the outlierness of each record is often measured based on the Euclidean distance between it and the other records in the?dataset. These distance calculations can, though, break down where we are working with many features and, in...

#Coding_Fridays : MACD time series analysis in Python In time series analysis, such as in stock market, detecting shifts in trends is crucial for efficient decision-making. One such example is the very popular Moving Average Convergence/Divergence (MACD) metric, which has been heavily used as a guideline, especially in short-selling strategies. MACD is based on Exponential Moving Average (EMA) or simply "exponential smoothing". In summary, EMA is just a way of aggregating a series of running values as they are generated, giving more weight to the more recent time steps instead of just taking a typical (arithmetic) mean value over a fixed time frame. In terms of Signal Processing, both EMA and MACD are auto-regressive processes of order 1, i.e., AR(1). These assume only limited association between successive time steps, the scale of which is determined by a linear factor (1-a). Furthermore, no exogenous factor is taken into account here, not even a generic "noise" factor in the EMA equation above. And coefficient (a) is assumed to be constant throughout the process. In the real world, none of these assumptions are valid. In fact, there is also a time shift or "phase difference" regarding the exact positions where MACD is supposed to mark volatilty shifts. This means that the MACD signal is always delayed, usually by 3–4 or more time points depending on the exact value of α, because that is the way all linear filters work, including EMA. Hence, the assumption is that, if MACD is applied to a time series of sufficienlty high resolution, then a proper and informed decision can be made at a much larger time scale, provided that the process does not "move" much faster within that time frame. If MACD can provide hints in the order of 5–10 minutes, then decisions can be assisted when made in the order of an hour or so. See the full video in Youtube channel central -- https://lnkd.in/d2-Sfy4t Note: This is intentionally "enriched" with several typing errors and logical bugs during coding, in order to demonstrate how debugging occurs during the first "alpha" tests. If not interested in this, just skip the portion 17'45" - 23':15" of the video. Enable captions for more details and walk-through. Source code available at the Github repository (see channel info). #ambient #coding #programming #notalking #terminal #console #python

Learn how to classify handwritten digits using KNN in Python's sklearn. Compare KNN, SVM, Naive Bayes, and Decision Tree for accuracy and performance. #AI #KNN #SVM #Bayes #Decision #Python #sklearn

I am learning Machine Learning with Python and created this synthetic data and to practice on. This is going to be my first project on creating a Fraud Detection System. I did not know that building such a simple system is so involved. ?? import pandas as pd import numpy as np import random import seaborn as sb import matplotlib.pyplot as plt from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler, LabelEncoder from sklearn.metrics import classification_report, confusion_matrix, roc_auc_score, roc_curve from sklearn.linear_model import LinearRegression from sklearn.tree import DecisionTreeClassifier np.random.seed(42) n_transactions = 10000 transaction_ids = np.arange(1, n_transactions + 1) amounts = np.random.uniform(1, 1000, n_transactions).round(2) times = np.random.randint(0, 1000000, n_transactions) customer_ids = np.random.randint(1, 501, n_transactions) merchant_ids = np.random.randint(1, 101, n_transactions) locations = np.random.choice(['New York', 'California', 'Tokyo', 'Paris', 'Berlin'], n_transactions) device_types = np.random.choice(['Mobile', 'Desktop', 'Tablet'], n_transactions) is_international = np.random.choice([0, 1], n_transactions, p=[0.8, 0.2]) is_fraud = np.random.choice([0, 1], n_transactions, p=[0.95, 0.05]) payments = pd.DataFrame({ ??'TransactionID': transaction_ids, ??'Amounts': amounts, ??'Times': times, ??'CustomerID': customer_ids, ??'MerchantID': merchant_ids, ??'Locations': locations, ??'DeviceType': device_types, ??'IsInternational': is_international, ??'IsFraud': is_fraud }) #machinelearning #python #dataanalyst #sklearn