Application of Machine Learning Classification to Detect Fraudulent E‐wallet Deposit Notification SMSes

Fraudulent e-wallet deposit notification SMSes designed to steal money and goods from m-banking users have become pervasive in Namibia. Motivated by an observed lack of mobile applications to protect users from such deceptions, this study evaluated the ability of machine learning to detect the fraudulent e-wallet deposit notification SMSes. The naïve Bayes (NB) and support vector machine (SVM) classifiers were trained to classify both ham (desired) SMSes and scam (fraudulent) e-wallet deposit notification SMSes. The performances of the two classifier models were then evaluated. The results revealed that the SVM classifier model could detect the fraudulent SMSes more efficiently than the NB classifier.


Introduction and research problem
In the past decade, there has been significant growth of spam (unwanted messages) in email and short message service messages (SMSes), and a contemporaneous growth in the capabilities of mobile banking (m-banking) (Almeida et al., 2011;Shaikh & Karjaluoto, 2015). Among its capabilities, m-banking allows users to utilise mobile phones to make payments from their bank accounts to other users' electronic wallet (e-wallet) accounts. It also allows payment recipients to receive notification SMSes that acknowledge payments into their e-wallets. The widespread use of SMS notifications to acknowledge e-wallet deposits has served to establish such notifications as a trusted means of proof-of-payment for m-banking users. This trust has tended to make e-wallet recipients complacent, to the extent that they may neglect to verify the legitimacy of a payment upon receipt of a deposit notification SMS. This habit of non-verification has fostered the emergence of fraudulent SMSes that use false e-wallet deposit notifications in an attempt to deceive and defraud m-banking users (Arde, 2012;Erongo, 2016). It has been reported that m-banking users in Namibia, its neighbour South Africa, and other developing countries have suffered substantial losses, of both money and goods, through falling victim to fraudulent e-wallet deposit notification SMSes (Arde, 2012;Christopher & Kar, 2018;Erongo, 2016;Nagel, 2015). This study was motivated by an observed lack of mobile applications for detecting fraudulent e-wallet deposit notification SMSes in order to safeguard m-banking users from the associated frauds. Our study, conducted in Namibia, evaluated the application of two machine learning classifiers to distinguish between ham (legitimate) SMSes and scam (fraudulent e-wallet deposit notification) SMSes. The classifiers tested were the naïve Bayes (NB) and support vector machine (SVM) models. The evaluation aimed to determine which of the two models could detect the fraudulent e-wallet deposit notification SMSes more efficiently. The ultimate aim was to establish which model would be a good candidate for implementation as an application for detecting fraudulent notifications on users' mobile devices.

E-wallet deposit notification SMS fraud
In the typical scenario, the fraudster first obtains the m-banking user's mobile number from a source such as a website, a Facebook notice, or an advertisement. The fraudster then forges an e-wallet deposit notification SMS that purports to acknowledge the deposit of a certain amount and sends it to the m-banking user. The fraudster follows up with a call or SMS, and claims to have mistakenly deposited the amount specified in the bogus notification SMS into the user's e-wallet. The fraudster then asks the targeted user to refund the money via an e-wallet payment. The fraudster may seek to make the refund more palatable to the user by asking for only a portion of the amount supposedly deposited, and may also use other social engineering tricks to lure the user into falling for the fraud.

Application of Machine Learning to Detect Fraudulent E-wallet Deposit SMSes
If the targeted user does not verify that the funds that the fraudster claims to have sent have actually been paid into their e-wallet, the user may fall for the fraud and make a payment from their account to the fraudster's e-wallet. In cases where the targeted user falls for the fraud, the fraudster simply withdraws the money and discards the SIM card used. Fraudsters use similar tricks in respect of goods, e.g., sending bogus e-wallet deposit notification SMSes to salespersons pretending to have paid for the goods, thus seeking to obtain goods without actually paying for them.
Since such frauds are relatively easy to detect and avoid under normal circumstances, fraudsters take advantage of specific situations in order to increase the target's vulnerability-e.g., communicating with someone who has had a death in the family, or with an online seller-as seen in Figure 1, which provides examples of fraud in Namibia and South Africa. Among the factors that create an enabling environment for e-wallet fraudsters are low SIM card costs and ready access to e-wallet services by anyone with a SIM.

SMS spam, smishing, and machine learning (ML) classif ication
Approaches to combatting spam email and SMSes generally involve the use of machine learning classification (see Akbari & Sajedi, 2015;Choudhary & Jain, 2017). Initial machine learning classification approaches to detecting SMS spam largely treated spam as a generalised set of undesired SMSes, without detection and delineation of specific types of spam (Abdulhamid et al., 2017). As malicious SMS phishing, a form of cyber-fraud known as "smishing", has become more prominent, approaches to spam detection have had to become more focused. The emergence of smishing has led to investigation of machine learning classifiers designed to specifically detect smishing SMSes (Goel & Jain, 2018;Jain & Gupta, 2018).
Further complicating the picture has been the emergence of legitimate SMSes with characteristics that overlap with those of spam and that could easily be erroneously treated as spam by existing spam detection or filtering systems (Reaves, Blue, Tian, Traynor, & Butler, 2016). These legitimate SMSes include those sent for purposes of advertising and promotion, and SMSes for verification codes and for passwordreset codes, both of which users can receive from sender numbers that they could not have known beforehand (Reaves et al., 2016). This makes targeted approaches to detecting specific types of SMS spam even more necessary, especially for the types of spam, such as the e-wallet deposit notification frauds, that have the potential to cause users significant losses. The problem of fraudulent e-wallet deposit notification SMSes is relatively new, despite its reported pervasive manifestations, and this tends to explain why our literature survey did not manage to identify any work done on the application of machine learning classification to specifically detect fraudulent e-wallet deposit notification SMSes.

SMS datasets, feature extraction, and machine learning classif ication
Studying or employing machine learning classification of SMSes requires access to appropriate datasets of ham SMSes and spam SMSes (Abdulhamid et al., 2017). Methods often used to obtain the required dataset include obtaining SMSes from public databases (Ahmed et al., 2014), extracting SMSes from public web-based sources (Almeida et al., 2011), and collecting SMSes directly from users (Shahi & Yadav, 2014).
A study by Cormack (2008) explains that prior to applying machine learning to classify textual content, the content must first be represented as a collection of features derived from the text or from extrinsic information related to the text. Feature extraction is often employed to capture textual features for classifying texts, and the feature extraction process frequently produces multi-dimensional feature sets. It then becomes necessary to employ feature selection, so as to eliminate the features that are less significant in the classification of the text. Various machine learning classifiers have been employed to classify SMSes for filtering or detecting spam. Some of the classifiers that are extensively used include naïve Bayes (NB), Random Forest (RF) and support vector machine (SVM) (Ahmed et al., 2014;Hedieh et al., 2016;Nagwani & Sharaff, 2017;Nuruzzaman et al., 2011). The NB classifier is widely used for SMS classification due to its simplicity and speed, while the common use of SVM and RF tends to be motivated by their high classification accuracy (CA)-often reported to be in ranges above 90% (Nagwani & Sharaff, 2017). For our study, we chose to test and compare the accuracy of the NB and SVM classifiers.

Methodology
The study employed an experimental research design, and used the Weka open source data mining software platform for the experiments.

The SMS dataset
We collected a dataset of 240 unique SMSes from Namibian m-banking users: 184 ham (i.e., normal and legitimate) SMSes and 56 scam (i.e., fraudulent) e-wallet deposit notification SMSes. The ham SMSes included legitimate e-wallet deposit notification SMSes. The ham SMSes, and some of the scam e-wallet deposit notification SMSes, were solicited from volunteers via invitations sent out on Facebook. The majority of the scam e-wallet deposit notification SMSes were extracted from user posts on public Facebook group (M-banking users habitually share examples of scam e-wallet deposit notification SMSes in online fora in order to warn other users). We then represented the 240 raw SMSes in terms of three attributes and a class specification, as outlined in Table 1.

content
The string of content in the body of the SMS.

contentLen
The length (number of characters) of the SMS body. Ham SMSes tend to have fewer characters than deposit notification SMSes (both legitimate and fraudulent).

smsClass
Whether the SMS is in the ham class or the scam e-wallet deposit notification class.

Feature extraction
We found that the three initial attributes used to define the raw SMSes-senderNumLen, content and contentLen-were insufficient to effectively classify the ham SMSes and the scam e-wallet deposit notification SMSes. We also found that the SMSes' contents contained numerous textual features that could be used to improve the classification. Hence, we employed Weka's StringToWordVector unsupervised filter in order to extract normalised word and term features from the SMSes' contents, for use, in addition to the senderNumLen and contentLen features, in classifying the SMSes.

Optimal classif ication features
A total of 1,223 features were extracted from the contents of the 240 SMSes in the dataset. With the addition of the senderNumLen and contentLen features, the set of features numbered 1,225. Weka's information gain (IG) feature selection algorithm was then used to select a subset of the 1,225 features that allowed optimal classification of the SMSes. The optimal classification features were determined by applying feature selection using different IG threshold (IG thshld ) values. Table 2 shows the number of features selected using different IG thshld in the [0.0, 1.0] range, and the two classifier models' respective average CA for 10-fold cross-validation.  Table 2 shows that a subset of 48 features, selected with IG thshld = 0.025, allowed the NB and SVM models to classify the SMSes with optimal CA. These 48 features allowed the NB classifier model to maintain CA = 0.9711, its highest observed CA; and the SVM model to achieve CA = 0.9917, its highest recorded CA.

Classif ier models' training and evaluation
The dataset of the 240 SMSes, having been defined using the 48 optimal features, was then used for training and evaluating the NB and SVM classifier models. Supervised learning was employed, following a 10-fold cross-validation approach. The evaluation gave most weight to the models' capability to detect fraudulent e-wallet deposit notification SMSes, and was based on the following metrics adopted from works by  Table 4 presents the two classifier models' performances in terms of TP, TN, FP, FN, FPR and FNR evaluation metrics. The results in Table 4 show that, on average, the SVM model correctly classified 18.2 scam e-wallet deposit notification SMSes (i.e., TP = 18.2) and misclassified 0.0 (i.e., FN = 0.0) compared to the NB's 17.7 and 0.5 respectively. This reveals that the SVM classifier model was more efficient than the NB model with respect to detecting scam e-wallet deposit notification SMSes. However, the results indicate the contrary about the models' capability to detect the ham SMSes, with the NB having correctly classified an average of 5.7 ham SMSes and misclassified 0.1, compared to the SVM's 5.6 and 0.2 respectively. The two models' FPR and FNR averages conform with the aforementioned results.

Comparative evaluation results
Because the emphasis of the study was on the models' capability to efficiently detect scam e-wallet deposit notification SMSes, we gave performance in terms of this criterion more weight than performance in respect of the detection of ham SMSes. Thus, the SVM classifier model was found to be superior to the NB model.
The graph in Figure 2 depicts the two classifier models' performance in terms of CA, precision, recall and F1-measure. The results shown in Figure 2 indicate that the SVM model correctly classified more SMSes than the NB model, producing the highest CA. The two classifier models demonstrated contrasting performances in terms of precision and recall, with SVM achieving the highest recall while NB produced the highest precision. The differences between precision and recall, in terms of their respective equations (4) and (5) provided in the previous section, is represented by FP and FN. The two classifier models had contrasting FP and FN values due to their dissimilar performances with respect to correctly classifying the ham SMSes and the scam e-wallet deposit notification SMSes, as highlighted in the previous section. This caused the observed models' contrasting precision and recall values. The harmonic mean of precision and recall (i.e., F1-measure) helped to remove any ambiguity regarding which of the two models made a better overall classifier for both ham SMSes and scam e-wallet deposit notification SMSes, with the SVM model's F1-measure = 0.995 being superior to the NB model's F1-measure = 0.983.

Conclusions
In an attempt to contribute to solution of the problem of fraudulent e-wallet deposit notification SMSes, we trained NB and SVM classifier models to classify ham SMSes and scam e-wallet deposit notification SMSes, following which their performances were evaluated. The evaluation results indicate that the SVM classifier model can detect fraudulent e-wallet deposit notification SMSes more efficiently than the NB model. The SVM model's strength was highlighted by its FNR = 0.000, CA = 0.992, recall = 1.000 and F1-measure = 0.995, making it the more efficient model. Our envisaged future work is to extend this study by developing a mobile application or applications making use of the SVM classifier model to detect fraudulent e-wallet deposit notification SMSes on a user's mobile device.