I'm following this article about Unsupervised Anomaly Detection Algorithms. In this article, a threshold value is calculated using the scipy score percentile method to determine whether the point is an outlier or not. What is the connection between percentile value and threshold and how can we decide if a point is an outlier or not using a threshold value?
Topic unsupervised-learning anomaly-detection
Category Data Science
Usually outliers are considered to be normally distributed (although that may not be the case in real life). Therefore, the anomaly threshold is computed in terms of the standard deviation of the dataset. Common choices are 3$\sigma$ or 5$\sigma$, i.e., the data point is considered an outlier if it deviates from the mean by more than 3 or 5 $\sigma$. Under the assumption of normal distribution, those correspond to $p$-values of $3\times 10^{-3}$ or $3 \times 10^{-7}$. However, these are likely to be underestimates of the real $p$-values for a real (non-normal) distribution. So it's a good idea to check that your model is not producing too many false positive anomaly reports.
You have a distribution of data (multidimensional or not it does not matter). And anomalies will be scattered towards corners of this dataset. Hence low percentile will indicate that certain points are anomalies in the data---cornered in. Threshold is, in this context, just a cutt-off value when you claim something is anomaly.
Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.