Financial fraud, particularly accounting fraud, is a major concern in manipulating financial statements (Campa et al. 2023). Wang et al. (2006) define fraud as “a deliberate act contrary to law, rule, or policy with intent to obtain unauthorized financial benefit.” Joyce and Biddle (1981) noted that fraud committed by higher-level management is especially hard to detect. Kieso et al. (2020) emphasized that financial fraud involves intentional misstatements or omissions of material information. Beasley et al. (2010) identified improper revenue recognition and asset overstatement as the most common methods of fraud. The Fraud Triangle, outlined by SAS No. 99, describes the three conditions under which fraud occurs: pressure, opportunity, and rationalization.

The readability of financial reports, particularly in the MD&A sections, can provide insights into fraudulent behaviour. The Fog Index, developed by Gunning (1952), measures readability by scoring the complexity of a text, where a higher Fog Index indicates a more difficult to read text. Li (2008) found that firms with lower earnings tend to have higher Fog Index scores, suggesting a link between complex, harder to read reports and poorer financial performance. Martinc et al. (2021) demonstrated that AI can effectively assess the readability of documents, making it a useful tool for identifying anomalies in financial reports.

In addition to readability, the sentiment expressed through positive and negative words in a text can signal underlying financial conditions. Positive words convey positive sentiment, while negative words reflect a negative tone. However, as Ghosh et al. (2015) noted, context can alter meaning, for example, sarcastic use of positive words can imply negativity, and phrases like “not bad” can turn negative meanings positive. Loughran and McDonald (2011) developed word lists to capture sentiment in financial disclosures, categorizing words into positive, negative, litigious, and uncertain. Examples include “gains” and “profitability” for positive words, “loss” and “impairment” for negative, “contracts” and “regulatory” for litigious, and “may” and “risk” for uncertain. These sentiment patterns are critical in analyzing the MD&A sections of 10-K filings.

Artificial Intelligence (AI) encompasses advancements like Machine Learning (ML) and Natural Language Processing (NLP), along with traditional statistical methods such as classification and clustering (Sutton et al. 2016). AI models can be trained using various methods, including supervised, semi-supervised, unsupervised, and reinforcement learning (Burkov 2019). In supervised learning, the dataset contains labeled examples, where each label represents a specific feature, such as identifying fraud in the MD&A sections of 10-K filings.

In this study, NLP is employed to automatically classify companies as fraudulent or non-fraudulent. The model used is based on BERT (Bidirectional Encoder Representations from Transformers), introduced by Devlin (2018). BERT is designed to pre-train deep bidirectional representations from unlabeled text, jointly conditioning on both left and right contexts in all layers. BERT is one of the earliest Large Language Models (LLMs) and can be applied to various tasks such as question answering and language inference without significant modification.

Huang et al. (2023) noted that LLMs, due to their vast number of parameters, can learn semantic and syntactic relationships between words. However, these models are often expensive and difficult to train. Moreover, once trained, LLMs become “black boxes,” making their decision-making processes challenging to interpret (Hassija et al. 2024).

AI has also been widely implemented in accounting, transforming traditional tasks. Studies such as Zhang et al. (2020) show that Big 4 firms have integrated AI technologies to enhance their services through automated data entry, voice analysis, intelligent search engines, and predictive analytics. These innovations allow accountants to focus on higher-value activities like strategic planning and advisory services. AI-based fraud detection methods, such as those used by Brown et al. (2020) and Ikhsan et al. (2022), have demonstrated high accuracy in detecting financial misreporting, thereby improving audit quality.

Ranta et al. (2023) identified several areas where AI has impacted accounting, including changes in the profession, textual analysis of accounting data, and improved prediction methods. Tang et al. (2018) developed an ontology-based fraud detection model using decision trees, achieving 86.67% accuracy in identifying fraudulent financial activity. AI-driven accounting systems, as described by Lee and Tajudeen (2020), have greatly improved productivity, efficiency, and risk management in financial reporting.

Bhattacharya and Mićković (2024) applied BERT to detect fraudulent reporting in MD&A sections of 10-K filings between 1994 and 2013. Their models outperformed others by around 15% in detecting fraud, providing significant economic benefits to regulators, investors, and analysts.

To understand the sample, we calculated the total number of words, sentences, and words per sentence in the MD&A sections. We found that the number of words per year in MD&A sections increased significantly over time. Before the year 2000, MD&A sections contained around 2,000 to 3,000 words per year, but after 2015, this range grew to 12,000 to 14,000 words; an increase of nearly 10,000 words over the last 15 years. This aligns with the findings of Brown and Tucker (2011), who suggest that the increase in word count may result from managers using boilerplate disclosures, which are standardized and provide little firm-specific information, reducing the overall usefulness of the MD&A.

On average, the MD&A sections in our dataset contain 8,854 words, 498 sentences, and 17 words per sentence. We observed that both the number of sentences and the average sentence length increased over time, with the rise in total sentences likely explained by the increase in total words. However, the increase in sentence length was an unexpected finding, potentially due to the inclusion of boilerplate information, as noted by Brown and Tucker (2011).

Further, Figure 2 provides a visual comparison of the word count between fraudulent and non-fraudulent MD&A sections. Interestingly, fraudulent filings have an average of 12,482 words, exceeding the 8,926 word average for non-fraudulent filings. This contradicts Nakashima et al. (2022), who found that MD&A disclosures were insignificantly shorter for Japanese fraudulent firms. One possible explanation is that fraudsters may attempt to bury fraudulent activities in longer texts, as readers are less likely to scrutinize lengthy reports. This is supported by Hancock (2007), who suggests that liars tend to use more words, which could explain our findings.

Figure 2.

Comparison of word count in fraudulent vs. non-fraudulent MD&A sections. Note: The figure compares the average word count in MD&A sections of fraudulent (red line) and non-fraudulent (blue line) 10-K filings, with the overall average word count (green line) included for reference.

To validate the findings of Bhattacharya and Mićković (2024), we use the BERT-Base-uncased model from Hugging Face, a pre-trained language model. Similar to Bhattacharya and Mićković (2024), we apply the WordPiece tokenizer, which splits text into tokens by removing punctuation and identifying word roots. Following Devlin (2018) and Bhattacharya and Mićković (2024), we use a classification token (CLS) at the start and a separation token (SEP) at the end of each input, ensuring sequences do not exceed the 512-token limit. We fine-tune the BERT models using Google Colab Pro’s Nvidia Tesla A100 GPU. The fine-tuning process is done with a learning rate of 2e-5, the AdamW optimizer, over 3 epochs, and with a batch size of 16.

Bhattacharya and Mićković (2024) approach fraud detection as a ranking task, using rank averaging to ensure that fraudulent samples are ranked higher than non-fraudulent ones, which reduces variability in predictions. They combine predictions from two models, BERT_first and BERT_last, trained on the first and last 512 tokens of the MD&A section, respectively, to capture both introductory summaries and future outlooks. To replicate their approach, we construct two models (BERT_first and BERT_last), train them on the first and last 512 tokens of the MD&A sections, and then average their predictions. Our final prediction is the rank average of the outputs from BERT_first and BERT_last, where pred_final = 1/2 * rank(pred_first) + 1/2 * rank(pred_last).

To validate the model and test Hypothesis 1 while staying consistent with Bhattacharya and Mićković (2024), we use a rolling window of consecutive five years for training, followed by the subsequent year as the test set. The validation set includes the years 1994 to 1999, in line with their study, to optimize our model parameters. For example, we train the model on data from 2000 to 2004 and test it on 2005. This approach is applied from 1994 to 2021, giving us a 23-year test period, extending beyond their 2013 cutoff. For evaluation, we also use the area under the ROC curve (AUC) as our primary metric. Given the class imbalance common in fraud detection, AUC is suitable as it measures the likelihood that a randomly selected fraud sample is ranked higher than a non-fraud sample, ensuring comparability with Bhattacharya and Mićković (2024).

To test Hypotheses 2 and 3, we use the word list from Loughran and McDonald (2023). We first split the dataset into fraud and non-fraud instances and then filtered the text based on the words from the Loughran and McDonald list. This list categorizes words into four sections: Positive, Negative, Litigious, and Uncertain. The dataset is organized by year, and we extracted the corresponding MD&A sections for each year. We then counted the occurrences of each word per year and categorized them according to Loughran and McDonald’s classification. Additionally, we calculated the total number of words per year to determine the percentage occurrence of each category annually.

To test Hypothesis 4, we use the Fog Index developed by Gunning (1952). We split the dataset into fraudulent and non-fraudulent filings per year and calculated the average Fog Index for each year, following Gunning’s method. We use an Ordinary Least Squares (OLS) regression analysis to test whether there are significant differences in word categories and readability between fraudulent and non-fraudulent MD&A sections. We also examine if these differences have changed over time, particularly with the rise of AI-generated financial reports.

Separate OLS regressions are conducted for the Fog Index and the occurrence of word categories (positive, negative, uncertainty, and litigious) to test for significant differences. We compare trends between two periods, 1994–2007 and 2008–2021, to see if word category usage has shifted. The regression includes variables for baseline measures, whether the MD&A section is non-fraudulent, year, and an interaction term to assess how trends differ between fraudulent and non-fraudulent sections over time.

We formulated Hypothesis 1 to assess whether a BERT model remains effective in detecting fraudulent activity in recent years, with accuracy comparable to or exceeding that of BM (2024). To gauge performance, we use their results as a benchmark, directly comparing our final model to their BERT models.

Table 1 displays the yearly and average AUC scores for our models from 2014 to 2021. Our BERT models, trained on MD&A sections from 10-K filings, are evaluated against the benchmark models. Our BERT_first, BERT_last, and BERT_final models achieved average AUC scores of 0.844, 0.661, and 0.797, respectively, compared to Bhattacharya and Mićković (2024), which reported AUC scores of 0.804, 0.816, and 0.826. While the AUC results for 2000–2013 are omitted for brevity, they are available upon request and show comparable performance between our models and Bhattacharya and Mićković (2024).

We tested Hypothesis 1 by comparing our BERT models to the benchmark models of Bhattacharya and Mićković (2024). The results indicate no significant differences between the models for the period 2000–2013, confirming that our BERT_first model performs comparably. Furthermore, our BERT_first model continues to be effective in detecting fraud from 2014–2021, with an average AUC score of 0.844, exceeding the benchmark’s 0.804. This confirms Hypothesis 1, showing that BERT remains effective in recent years.

Hypothesis 2 postulates that firms engaged in fraudulent activities use a different frequency of positive, negative, litigious, and uncertainty words in their MD&A sections compared to non-fraudulent firms. To test this, we examine the occurrences of these specific word categories in both fraudulent and non-fraudulent MD&A sections, aiming to determine whether linguistic characteristics differ between the two groups. Figure 3 presents the average percentage of each word category—positive, negative, litigious, and uncertainty—used per year from 1994 to 2021 for both fraudulent and non-fraudulent filings.

Figure 3.

Percentage of word occurrences in MD&A sections for fraudulent vs. non-fraudulent companies. Note: The y-axis represents the average yearly percentage of each word category occurring in the MD&A sections. The word categories (Positive, Negative, Uncertainty, and Litigious) are based on the classifications by Loughran and McDonald (2023).

We also conduct Ordinary Least Squares (OLS) regression analyses to test for significant differences in word category usage between fraudulent and non-fraudulent MD&A sections over the period 1994–2021. The results show that fraudulent MD&A sections contain significantly more positive words, with the difference being statistically significant at the 99% level. This finding aligns with the suggestion of Bhattacharya and Mićković (2024), confirming that fraudulent filings use more positive language. A separate test for the 1994–2013 period also showed a significant difference at the 5% level (not included in the manuscript for brevity).

However, the analysis reveals no significant differences in the usage of negative, uncertainty, and litigious words between fraudulent and non-fraudulent firms at the 95% significance level, leading us to partially reject H2 for these categories. There is some indication, at the 90% significance level, that non-fraudulent MD&A sections contain fewer negative words.

In Hypothesis 3, we propose that the frequency of word categories in non-fraudulent filings is more consistent over time compared to fraudulent filings. To test this, we examine whether the trend in the frequency of word categories in MD&A sections differs between non-fraudulent and fraudulent firms.

Interestingly, Figure 3 reveals a stabilizing trend in the occurrence of word categories in recent years. This aligns with the study by Oyewole et al. (2024), which suggests that the use of AI-generated financial reports may contribute to this stabilization. To investigate this, we tested whether the slope of the regression for word usage trends was less steep during the last 14 years (2008–2021) compared to the first 14 years (1994–2007). The difference in slopes is shown in Figure 4, which illustrates the trends in positive, negative, uncertainty, and litigious word usage across both periods. The figure presents scatter plots and trend lines, with the first time period T1 (1994–2007) indicated in red and second time period T2 (2008–2021) in blue, allowing for a clear visual comparison of word usage over time for each category.

Figure 4.

Trends in word usage for positive, negative, uncertainty, and litigious words across two time periods. Note: The figure presents scatter plots and trend lines for the usage of positive, negative, uncertainty, and litigious words in MD&A sections across two periods: T1 (1994–2007) shown in red, and T2 (2008–2021) shown in blue. It provides a visual comparison of word usage trends over time for each category, illustrating the difference in slopes between the two periods.

Our regression analysis (not tabulated for brevity) examined word category trends (positive, negative, uncertainty, and litigious) between two periods (T1 and T2), as shown in Figure 4.

The results show a significant decline in the use of positive words in both fraudulent and non-fraudulent MD&A during T1. However, in T2, the decline in positive word usage was less steep for fraudulent MD&A, indicating some stabilization, while the trend remained insignificant for non-fraudulent filings.

Negative word usage was significantly higher in the first period for both groups. Interestingly, there was a significant reduction in the use of negative words in the last period, suggesting a shift in language over time.

Both fraudulent and non-fraudulent MD&A showed an increase in uncertainty words during T1, but this trend reversed to a decline in T2. Litigious word usage showed an insignificant positive trend in fraudulent MD&A during T1, which shifted to a negative trend in T2. This suggests that fraudulent filings may be using less legalistic language over time, while the results for non-fraudulent filings were not significant.

In Hypothesis 4, we propose that firms engaged in fraudulent activities have a higher Fog Index, indicating more complex and difficult to read MD&A sections compared to non-fraudulent firms. To test this, we calculated the average Gunning Fog Index for the MD&A sections, with the results visualized in Figure 5. Contrary to our hypothesis, the results show that the average Fog Index of non-fraudulent firms is actually higher than that of fraudulent firms. This contradicts our expectation and the findings of Nakashima et al. (2022), who suggested that fraudulent filings are harder to read, and Moffitt and Burns (2009), who found that fraudulent 10-Ks tend to contain more complex language.

Figure 5.

Average Fog Index per year for fraudulent vs. non-fraudulent MD&A sections. Note: The y-axis displays the Fog Index, which measures the readability of the text. The blue solid line represents the average Fog Index for non-fraudulent MD&A sections per year, while the red solid line represents the average Fog Index for fraudulent MD&A sections per year.

We also conducted a regression analysis to further explore the relationship between the Fog Index and fraudulent vs. non-fraudulent MD&A sections over time. While Figure 5 suggests that the average Fog Index for fraudulent firms is lower in many years, the difference is not statistically significant. Therefore, we cannot conclusively state that firms engaging in fraudulent activities have a significantly lower Fog Index than non-fraudulent firms. However, the results do indicate a statistically significant increase in the Fog Index over time, suggesting that MD&A sections are becoming more difficult to read overall.