In many companies, the main problem in cybersecurity is not that systems fail to detect attacks, but that they create too many useless alerts. A large part of these alerts are false positives. This means normal actions are marked as dangerous. Every alert usually needs to be checked by a human analyst. When most of them are false alarms, the security team loses a lot of time and energy, and real threats can be missed. Some reports say that organizations spend tens of thousands of working hours and more than a million dollars each year just to confirm that most alerts are actually harmless. My project starts from this problem and asks a simple question: Can a machine learning model reduce false positives and false negatives when it tries to detect phishing websites, compared to a traditional rule-based system?
I used a public dataset from Kaggle with about 800,000 URLs that already have labels. Each row contains a URL string and a status label of 0 or 1, where 0 means phishing and 1 means legitimate. I first cleaned the data by dropping duplicate URLs and rows with missing labels. Then I split the dataset into three parts. The training set is used to fit the model. The validation set is used to tune parameters. The test set is used at the end to evaluate performance fairly. I kept the ratio of phishing and normal URLs similar across all three sets so that the comparison would be meaningful.
To make the experiment closer to a real-world setting, I decided to use only information from the URL itself, as if the system had to judge the risk at the moment it first sees the link. For each URL, I extracted simple and explainable features. Examples include the total length of the URL, how many dots it has, how many hyphens it has, how many digits appear, and what fraction of all characters they are. I also checked whether there are special symbols, whether the domain is an IP address, how many subdirectories appear in the path, and whether the URL has a query part. In addition, I looked for typical phishing keywords inside the URL, such as “login”, “signin”, “verify”, “secure”, “bank”, “paypal”, “appleid”, and “account”. These features allow the system to guess risk based only on how the URL looks, and they are cheap to compute and easy for humans to understand.
On top of these features, I built two different detection methods. The first one is a rule-based system. It gives each URL a score based on handwritten rules. The gains more points if it fits more features mentioned before. At the end, if the score is above a chosen threshold, the URL is classified as phishing; otherwise, it is classified as legitimate. The threshold is not chosen randomly. I scanned several possible values on the validation set and selected the one that gave the best F1 score for the phishing class. The second method is a machine learning model. I used a Random Forest classifier that takes the same features as input, but learns patterns automatically from training examples.
The difference between the two methods on the same test set is very clear. The rule-based system reaches an accuracy of about 55 percent, and it performs especially badly on phishing URLs. Out of about 120,000 test URLs, it produces around 12,922 false positives, where normal URLs are treated as phishing, and about 41,890 false negatives, where phishing URLs are treated as normal. In practice, this means many useless alerts and many dangerous links that slip through the system. The Random Forest model, on the other hand, reaches an accuracy of about 88.8 percent. Its ROC AUC is about 0.95, which shows that it is very good at ranking truly dangerous URLs ahead of safe ones. On the same test set, it produces about only 4,815 false positives and about 8,745 false negatives. Both numbers are much lower than the rule-based system. If we think of every false positive as a wasted investigation and every false negative as a risk, this gap means very real savings in time and cost in a security operations center.
A rule-based system is basically a summary of what humans think looks suspicious. It usually covers only the patterns we already know and can describe. A machine learning model, trained on many real examples, can adjust and correct these human guesses and capture more subtle structures in the data. At the same time, this experiment shows that we do not always need very complex inputs. Even simple URL structure features, when combined with a reasonable model and enough data, can significantly reduce both false positives and false negatives.
Ponemon Institute. The Cost of Malware Containment. Damballa, 2015, www.ponemon.org/local/upload/file/Damballa%20Malware%20Containment%20FINAL%203.pdf. Accessed 10 Dec. 2025.
Harisudhan411. (n.d.). Phishing and Legitimate URLs [Dataset]. Kaggle. https://www.kaggle.com/datasets/harisudhan411/phishing-and-legitimate-urls
Blum, A., Wardman, B., Solorio, T., & Warner, G. (2010). Lexical feature based phishing URL detection using online learning (Publication information available via ResearchGate).
Hong, J., Kim, T., Liu, J., Park, N., & Kim, S.-W. (2020). Phishing URL detection with lexical features and blacklisted domains. In Adaptive and Intelligent Systems (conference proceedings / paper listing).
Lashkari, A. H., Mamun, M. S. I., & Ghorbani, A. A. (2016). Detecting malicious URLs using lexical analysis. In [Book/Proceedings chapter].
Le, A., Markopoulou, A., & Faloutsos, M. (2010). PhishDef: URL names say it all (arXiv:1009.2275). arXiv. https://arxiv.org/abs/1009.2275
Linh, D. M., et al. (2025). A feature-engineered dataset of benign and phishing URLs with lexical and structural features. [Journal/Repository entry].
Lipton, Z. C., Elkan, C., & Narayanaswamy, B. (2014). Thresholding classifiers to maximize F1 score (arXiv:1402.1892). arXiv. https://arxiv.org/abs/1402.1892
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324
Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. https://doi.org/10.1016/j.patrec.2005.10.010