ORIGINAL RESEARCH

 

Development and comparison of payment behaviour prediction models for two South African state departments

 

 

Erika Fourie; Tanja Verster; Gary W. van Vuuren

Centre for Business Mathematics and Informatics, North West University, South Africa

Correspondence

 

 

---

ABSTRACT

BACKGROUND: No credit rating methodology currently exists for any of South Africa's sub nationals.
AIM: To develop a generic, quantitative credit rating methodology for the Department of Health and the Department of Education combined, as well as specific quantitative credit rating methodologies, for each department, individually.
SETTING: A comparison between generic and specific subnational credit rating methodologies to assess which fits the South African subnational environment best. Studies and results obtained from other nations were used to construct the approach.
METHODS: In a typical credit rating methodology, both quantitative and qualitative information is considered. In South Africa (as a developing economy), the quantitative information equates to a smaller portion of the final credit rating. A generic quantitative credit rating methodology, as well as specific credit rating methodologies, was developed. The appropriateness of these generic and specific models was tested with regards to prediction accuracies using Red, Amber or Green (RAG) statuses on a traffic light series. An illustration of the predicted versus actual ranks is provided, as well as an example to illustrate how model-predicted RAG statuses, based on quantitative information, may be overlaid with more recent qualitative information to derive a final ranking.
RESULTS: A generic, quantitative credit rating methodology for the Departments of Health and the Department of Education combined was developed, as well as specific credit rating methodologies for each department separately. The specific subnational credit rating methodology outperformed the generic methodology considerably; more precisely, the generic models predicted a maximum of 50% of the new cases correctly as opposed to the specific Health and Education models' 78%.
CONCLUSION: The primary contribution of this study was to develop and compare generic and specific subnational credit rating methodologies. A further contribution was to test the appropriateness of these models' prediction accuracies using RAG statutes. The specific subnational credit rating methodology was found to outperform the generic methodology considerably.

---

 

 

Introduction

Credit risk, defined by Scott (2003) as the risk that a borrower will be unable to make payments of interest or principal in a timely manner, should be assessed by banks and other investors before entering into a loan agreement. One way of assessing credit risk is the use of credit ratings. Scott (2003) terms a credit rating as 'a grading of a borrower's ability to meet its financial obligations in a timely manner'. Credit ratings can be used to rate issuers (e.g. companies, governments and subnational governments) and also issues (such as financial instruments or specific debt instruments).

Subnationals are defined as all tiers of government and public entities below the sovereign or national government, including states, provinces, counties, cities, towns, public utility companies, school districts and other special-purpose government entities that have the capacity to incur debt (Liu & Waibel 2008). South African subnationals are defined as provincial governments and departments for the purpose of this article. None of South Africa's defined subnationals are credit rated at present (2016).

It is evident from the review of subnational credit rating methodologies documented by Fourie et al. (2013) that special credit rating methodologies are required to rate subnationals because of the unique way that these governments are managed. One aspect that requires special attention is the rules and regulations set by the national government regarding financial management - especially rules and regulations related to borrowing. Another example is the subnational's economic profile, including demographics and social structures (Fitch 2011).

Credit rating methodologies can be divided into generic and specific credit rating methodologies. The specific credit rating methodologies are developed to credit rate a specific population (e.g. a specific country's subnationals) and the generic methodologies are developed to credit rate a wide range of different types of populations (e.g. a number of different countries' subnationals).

In a typical credit rating methodology, both quantitative and qualitative information will be considered and, specifically in South Africa as a developing country, the quantitative information will equate to a smaller portion of the final credit rating.

The primary contribution of this study is to develop and compare a generic with a specific credit rating methodology for South Africa's subnationals in order to predict payment behaviour, since no credit rating methodology currently exists for South Africa's subnationals as indicated by Fourie, Verster and Van Vuuren (2016). Note that because of data availability the developed models will only relate to the quantitative part of the final credit rating. The secondary contribution is to test the appropriateness of these generic and specific models by comparing the predicted payment behaviour with the actual payment behaviour by using Red, Amber or Green (RAG) statuses or traffic light series. An example is also provided to indicate how qualitative information can be taken into account.

The article is organised as follows: The 'Literature review' section provides a literature overview of existing subnational credit rating methodologies as well as South African laws and regulations applicable to subnational borrowing. The 'Data and methodology used' section describes the data and methodology used. The 'Results' section applies the methodology to develop a proposed generic quantitative credit rating methodology for the Departments of Health and Education combined, as well as specific credit rating methodology for each department separately. These two departments were selected because South Africa's National Treasury (National Treasury) indicates that these two departments' total expenditure equates to the biggest portion of the provinces' total expenditure. In the 2015-2016 financial year, the Department of Health's total expenditure equated to 32% of the provinces' total expenditure and the Department of Education's total expenditure equated to 41%, thus a combined proportion of 73% (National Treasury 2016). A total of eight linear regression models were built: These are tested in the 'Testing the appropriateness of the proposed models' section. The 'Conclusion and suggestions for future research' section concludes the article and provides recommendations and suggestions for future research.

 

Literature review

The literature overview summarises the required background information and is documented in three sections. Firstly, examples of existing generic and specific subnational credit rating methodologies are provided in the 'Existing subnational credit rating methodologies' section. The 'South African laws and regulations' section documents South African laws and regulations applicable to subnational borrowing. This is followed in the 'Data and methodology used' section by a discussion on the data and the methodology used to develop credit rating methodologies for South African subnationals.

Existing subnational credit rating methodologies

A credit rating is a grading of a borrower's ability to meet its financial obligations in a timely manner (Scott 2003). Credit ratings, or gradings, may be used to rate issuers (e.g. companies, sovereign governments and subnational governments), as well as issues, such as financial instruments or specific debt instruments.

Credit rating methodologies are required to rate the creditworthiness of issuers and issues. As mentioned earlier, these methodologies can be divided into generic and specific credit rating methodologies. Some of the advantages of generic models in general are that they are freely available and less expensive. One of the advantages of specific models is that they are more accurate at predicting outcomes (Burazeri, Dhuci & Kufo 2014).

The three largest international credit rating agencies are Fitch Ratings (Fitch), Moody's Investors Service (Moody's) and Standard and Poor's Financial Services (S&P). These credit rating agencies all generate generic subnational credit rating methodologies that could be used to determine the creditworthiness of any subnational government, regardless of the country within which the subnational government presides. Moody's, for example, rated 306 regional and local governments in 35 countries outside the United States (Liu & Tan 2009).

Theoretically, these methodologies can be applied to South Africa's provincial governments, departments and municipalities because of the generic definitions of subnationals (i.e. local and regional governments). However, no provincial government or department in South Africa is currently (2016) rated. On the other hand, all 278 of South Africa's municipal governments are rated by Municipal IQ, a data and intelligence service specialising in the monitoring and assessment of South Africa's municipalities (Municipal IQ 2015). Fitch and Moody's also rate the municipalities, but only a few (less than 5%).

In this study, proposed generic quantitative credit rating methodologies were developed for South African subnationals, more specifically, the Departments of Health and Education collectively as well as specific credit rating methodologies for each department separately.

The next section pays particular attention to some existing subnational credit rating methodologies, both generic and specific, because a thorough understanding of existing subnational credit rating methodologies is required in order to develop a credit rating methodology for South Africa's subnationals. This is largely based on the work of Fourie et al. (2013).

Generic subnational credit rating methodologies

The three international credit rating agencies' subnational credit rating methodologies are recapped in this section. These methodologies are considered to be generic because they could be used to determine the creditworthiness of a number of different subnational governments, regardless of which country the subnational governments reside in.

Fitch Ratings assesses the institutional framework within which the subnational government operates, as this provides the context within which the other factors interact. The four other factors are: debt and other long-term liabilities, finances, management and administration, and the economic status of the subnational government Fitch Ratings (2011).

Moody's (2008) assesses two fundamental aspects: the stand-alone credit quality of the subnational and the probability of the subnational receiving extraordinary support (i.e. assistance provided by the national government, any other higher tier of government or the subnational's peers to prevent the subnational from defaulting). The creditworthiness of a subnational is determined by evaluating six different factors, namely: the operating environment, institutional framework, financial position and performance, debt profile, governance and management factors, and the economic fundamentals of the subnational under review.

S&P employs eight factors to determine the credit quality of a subnational government. Firstly, the institutional framework factor reviews the institutional and legal background of the subnational government. The other seven factors are: economy, financial management, budgetary flexibility, budgetary performance, liquidity, debt burden and contingent liabilities. These focus on the subnational government itself and are used to determine the stand-alone credit quality of the subnational (S&P 2010).

Fourie et al. (2016) note that theoretically these can be applied to South Africa's provincial governments, departments and municipalities because of the generic definitions of subnationals (local and regional governments). However, to date no South African provincial government or department has been assigned a credit rating by Moody's, Fitch or S&P. Also, the data used in these existing credit rating methodologies do not necessarily exist, as will be seen in the 'Data and methodology used' section.

Comparison of the three international rating agencies' rating methodologies

A comparison of the subnational credit rating methodologies used by the three major international credit rating agencies provides insight needed to develop a credit rating methodology for South Africa's subnationals. Liu and Tan (2009) state that there are two similarities and three differences evident when comparing the subnational credit rating methodologies used by the three major international credit rating agencies.

Similarities: The similarities are the rating criteria and the rating processes:

• The rating criteria used by the three rating agencies can be divided into five principal or broad factors, as illustrated in Figure 1. These factors are the subnational's economic conditions, the fiscal performance of the subnational, the financial and debt position of the subnational, the management quality and institutional strength of subnational, and sovereign factors, intergovernmental relationships and fiscal arrangements. The first four factors involve only the subnational, while the fifth element embraces the influence of sovereign factors, intergovernmental relations and fiscal arrangements on the subnational. If a proposed credit rating methodology will not be used on an international level, the fifth factor may be neglected as the national government will have the same effect on all the subnationals. This observation will be taken into account when developing South African subnational credit rating methodologies.

• The rating processes followed to credit rate subnational governments are similar to common credit ratings processes, but more emphasis is placed on qualitative aspects because these may provide information not necessarily revealed by published information (Liu & Tan 2009).

 

 

Differences: The three main rating criteria differences between the credit rating methodologies are:

• Relative weights assigned to rating variables: The relative weights assigned to the rating variables, and thus broad rating factors too, differ according to the agency's view of the importance of the variables, although the variables used are roughly the same.

• Changes to relative weights over time: As a rating agency's view of the importance of a variable changes over time, the relative weight assigned also alters.

• Importance of qualitative variables: The importance of qualitative variables and factors vary between agencies (Liu & Tan 2009).

Specific subnational credit rating methodologies

This section focuses on existing regional subnational credit rating methodologies. These are considered to be examples of specific credit rating methodologies since it is used to determine the creditworthiness of subnational governments within specific countries or regions, i.e. it applies to specific populations. The two examples of specific or regional subnational credit rating methodologies for developed economies are Dominion Bond Rating Service Limited's (DBRS) rating methodology for Canadian subnational governments and Australia Ratings' methodology for Australian subnational governments. Fitch's credit rating methodologies for Indian subnational governments, as well as the methodology used by Credit Rating Information Services of India Limited (CRISIL) are discussed as examples of methodologies used for developing economies.

Dominion Bond Rating Service Limited assesses only two different types of factors to determine the credit rating of a Canadian provincial government, namely, province-specific operating risk and province-specific financial risk. This methodology does not apply to the three territories that also form part of Canada's subnational governments (DBRS 2011).

The funds needed to fulfil the principal responsibilities of the Australian subnationals, that is, to provide the necessary public services and infrastructure, are provided by the national government. Thus, Australia Ratings (2014) examines the current and future economic and financial situation of a subnational, as well as the strategy of the specific subnational government. The four different factors employed to do this are: economy, financial management, accounting and debt profile.

Fitch adjusted its international rating methodology to focus specifically on India. Fitch employs the following five factors to determine the stand-alone creditworthiness of Indian subnational governments: institutional and administrative aspects, economic and social profile, fiscal and budgetary performance, debt, liquidity and indirect risk, and management.

Credit Rating Information Services of India Limited (2011) assesses the credit quality of most of India's state governments by making use of three main factors: the state's economic structure, the government's finances and the economic management.

Note that regional subnational credit rating methodologies never review the national governments because all subnationals reviewed within a specific country share the same national government, and thus the outcome of the factor will remain static regardless of which subnational is being credit rated.

Importance of key factors: Developed versus developing economies

Although, the key factors used in the credit rating methodologies for developed and developing economies will generally be the same, the relative importance of these key factors will differ. The quantitative factors (financial fundamentals) and the qualitative factors (other key factors) included in the rating methodology for a developed country will be viewed as equally important. Thus, both will equate to 50% of the credit rating. In a developing country, the quantitative factors will only equate to 30% of the credit rating and the qualitative factors to 70% (Moody's 2007). Fourie et al. (2013) state that this principle is also relevant when developing a credit rating methodology for South Africa's subnationals.

Public sector risk management framework of South Africa

In the absence of a South African regional subnational credit rating methodology, the Public Sector Risk Management Framework of South Africa (PSRMF) could be reviewed in order to gain background knowledge for the development of such a methodology. National Treasury developed the PSRMF in response to the PFMA (Public Finance Management Act No 1 of 1999) to aid provincial governments and departments, among others, in their implementation and maintenance of effective, efficient and transparent systems of risk management and control (National Treasury 2010). However, this framework is considered to be vague. For example, it states that management should provide risk management reports but not how frequent these reports should be provided and to whom.

Key findings from literature review

Two key findings were drawn from the literature discussed above. Firstly, the variables used in subnational credit rating methodologies can be grouped into five broad factors, namely, the subnational's economic conditions, the fiscal performance of the subnational, the financial and debt position of the subnational, the management quality and strength of subnational institutions, as well as the influence of the sovereign factors, intergovernmental relations and fiscal arrangements on the subnational.

The other key finding is the difference in the importance of quantitative and qualitative information for developed and developing countries. Typically, for a developed country the quantitative and qualitative information will contribute equally to a credit rating, whereas for developing countries the quantitative information usually equates to the smallest portion of a credit rating.

Both observations will be applied in the remainder of this article. Currently, none of the South African subnationals are being credit rated (Fourie et al. 2016). This article develops the quantitative parts of credit rating methodologies for South Africa's subnationals, focusing on the Departments of Education and Health.

The current situation in South Africa

Although credit ratings are not assigned to the defined South African subnationals at present, the three largest international credit rating agencies do rate South Africa's national government. However, if the credit ratings resulting from the proposed methodology will not be used on an international level, the effect of the sovereign rating on the subnationals' credit rating may be neglected since the national government will have the same effect on all nine subnationals. As previously mentioned, South Africa's municipal governments are indeed being credit rated by Municipal IQ, Fitch and Moody's.

The National Treasury developed the PSRMF to aid provincial (subnational) governments in the implementation and maintenance of effective, efficient and transparent systems of risk management and control. However, the framework is vague and impractical. National Treasury (2010) states that the framework is under review to address some of the weaknesses.

Two concerns when developing credit scoring models, including subnational credit rating methodologies, are data availability and data quality (Siddiqi 2006). A preliminary investigation into this matter revealed three possible data sources that could be used within the South African context, namely, the subnational government's annual reports, the Auditor-General's reports, and, the Section 32 reports compiled by the National Treasury (see the 'Data' section for a detailed discussion).

South African laws and regulations

In addition to knowledge regarding existing subnational credit rating methodologies, an understanding of the regulatory framework within which subnational governments operate is also required when developing a country-specific subnational credit rating methodology. These laws and regulations should be investigated because these set the parameters within which subnational borrowing may take place.

The South African regulatory framework is summarised in the Borrowing Powers of Provincial Governments Act (No 48 of 1996) - that is, the Borrowing Act - and the PFMA, as well as the regulations that accompany this act, namely the Treasury Regulations for departments, constitutional institution and public entities. The Borrowing Act stipulates the borrowing powers of South Africa's subnational or provincial governments (South Africa 1996). The purpose of the PFMA and its Treasury Regulations is to 'secure transparency, accountability and sound management of the revenue, expenditure, assets and liabilities of the institutions to which the act applies' (South Africa 1999, 2001).

Each of the acts has a different purpose. South Africa (1996) states that the goal of the Borrowing Act is 'to provide for norms and conditions in respect of the borrowing powers of provincial governments and for matters incidental thereto', whereas the PFMA's purpose is stated as:

to regulate financial management in the national governments and provincial governments; to ensure that all revenue, expenditure, assets and liabilities of those governments are managed efficiently and effectively; to provide for the responsibilities of persons entrusted with financial management in those governments; and to provide for matters connected therewith. (South Africa 1999:1)

It is important to ensure that these laws and regulations are adhered to if it has been decided to extend credit to a subnational (based on a credit rating derived from applying the models to be developed), as non-adherence could lead to unenforceable loan arrangements. An example of such a specification is that any loan raised for the purpose of bridging finance must be redeemed within 30 days after the end of the financial year, making the maximum loan duration 13 months (South Africa, 2001). Another example is that the issuing of guarantees, indemnities and securities is usually not permitted.

 

Data and methodology used

This section encompasses the data used and the methodology applied in order to develop the proposed quantitative sections of a credit rating methodology. This is largely based on previous work by the authors - refer to Fourie et al. (2016) for more details.

Data

The data requirements were based on the information used in existing subnational credit rating methodologies and special focus was placed on the five broad factors identified in the 'Comparison of the three international rating agencies' rating methodologies' section. The final data sets included variables representing the four relevant broad factors (the fifth factor was omitted as all the departments reside within the same country).

Siddiqi (2006) states that data availability and data quality are concerning when developing credit scoring models. Furthermore, Liu and Tan (2009) caution that data not kept in a consistent form often cause problems when developing subnational credit rating methodologies for a developing countries. For these reasons, much attention has been paid to the identification of reliable data sources. That resulted in reports from four different institutions were used as data sources: the provincial departments, the National Treasury, Statistics South Africa (StatsSA) and the Auditor-General of South Africa (Auditor-General). However, most of the data used in the existing subnational credit rating methodologies do not exist in the South African context:

• Provincial Departments of Health and Education's annual reports: These reports are compiled per department positioned within the provincial governments. The necessary data were sourced from each province's Departments of Health and Education's annual reports, namely Gauteng Department of Education (2007-2013), Western Cape Department of Education (2007-2013), Eastern Cape Department of Education (2007-2013), Northern Cape Department of Education (2007-2013), North West Department of Education (2007-2013), Mpumalanga Department of Education (2007-2013), KwaZulu-Natal Department of Education (2007-2013), Limpopo Department of Education (2007-2013), Free State Department of Education (2007-2013) and Gauteng Department of Health (2007-2013), Western Cape Department of Health (2007-2013), Eastern Cape Department of Health (2007-2013), Northern Cape Department of Health (2007 - 2013), North West Department of Health (2007-2013), Mpumalanga Department of Health (2007-2013), KwaZulu-Natal Department of Health (2007-2013), Limpopo Department of Health (2007-2013) and Free State Department of Health (2007-2013). A total number of 108 annual reports were used.

• National Treasury: Two reports assembled on a regular basis by the National Treasury were used: Fourth Quarter Year to Date Provincial Budgets and Expenditure Report (Preliminary Outcome) 06/07 - 10/11 (National Treasury 2007a-2011a), as well as the Provincial Statements of Receipts and Payments for the 4th Quarter (end March 2007 to 2011) (National Treasury 2007b-2011b).

• StatsSA: The two reports used from this institution are the Gross Domestic Product. Annual estimates 2003 to 2012. Regional estimates 2003 to 2012. Third quarter 2013. Statistical release P0441 report (StatsSA 2013), as well as StatsSA's (2012) interactive data set containing the mid-year population estimate.

• Auditor-General: The Consolidated General Reports on the National and Provincial Audit Outcomes of 06/07 to 10/11 (Auditor-General 2007-2011) were also used.

Firstly, separate data sets were compiled for each department: the Department of Health and the Department of Education. The one interactive data set and 70 different reports containing 5 years' worth of data for the nine provinces resulted in 45 data points for each of the departments. Secondly, the department-level data were aggregated to construct provincial-level data, also referred to as the combined data set. This data set contained 90 data points. The data were captured manually, and data error identification and rectification measures were applied as an iterative process throughout the data compilation and model development phases. The constructed data sets are a secondary objective of this article and are available on request.

Note that the Health and Education data sets are department-specific, and thus contains data from a specific population. Any credit rating methodology resulting from the department-level data will therefore be a specific credit rating methodology, but any credit rating methodology resulting from the combined data set will be a generic one.

Specific credit rating methodologies were considered because Thomas (2000) as well as Anderson (2007) state that these types of models normally perform better than generic models. On the other hand, Siddiqi (2006) mentions that because of the costs involved it might not always be worthwhile to develop specific models. This specifically rings true for the South African environment because a number of different specific models need to be developed in order to assess all the provincial departments. Some provinces have a total of 12 departments (South Africa 2017) and an additional complication is the differences in the assignment of functions between these departments. For example, the Eastern Cape groups Road and Public Works together, whereas Gauteng groups Roads and Transport together and does not have a department specifically administering Public Works (South Africa 2017). For these reasons, generic models were considered too.

The constructed data sets contain only quantitative information. Therefore, the models developed from these will only relate to the quantitative part of the final credit rating. The qualitative part may be taken into account as an intuitive overlay, as will be illustrated.

Linear regression methodology

Linear regression modelling was chosen to predict the quantitative part of the credit ratings for South African provincial departments. The principal advantage of linear regression is its simplicity, interpretability, scientific acceptance and widespread availability. Linear regression is also widely available in statistical software packages and business intelligence tools (Chambers & Dinsmore 2014). Also, the results of a linear regression model could be summarised into the symbols used by the credit rating agencies to indicate relative rankings of creditworthiness, for example, AA−, BBB+ and Caa1.¹

The aim of the models developed in this study is to rank South Africa's subnationals in terms of future payment behaviour. To forecast using linear regression models, the independent variables used to predict the dependent variable for the next year have to lag by 1 year. This also applies to this work as the aim is to predict the provincial departments' credit ratings for the next financial year (e.g. 2015/2016) based on the current financial year's data (e.g. 2014/2015).

Dependent variable

The ultimate aim of a credit rating methodology is to predict default. Default can be viewed as a department's capacity and willingness to repay debt obligations in full and on time. The cash-based accounting system used by the provincial departments (National Treasury 2013b) resulted in insufficient information available to provide an indication of whether a department's debt obligations were paid in full and on time. The following proxy (Equation 1), suggested by the National Treasury, was used for the dependent variable instead (National Treasury 2012):

After various workshops and discussions with the National Treasury, no monthly or quarterly variables for the target variable could be found (National Treasury 2012, 2013a). Because of this constraint that the dependent variable was only available yearly, only yearly independent variables were used (even if some were available quarterly).

Independent variables

Three data sets were constructed containing either 42 or 43 (one extra for the combined data set) independent variables. As mentioned previously, the data requirements were based on the information used in existing subnational credit rating methodologies - especially the broad factors identified by Liu and Tan (2009). Seven of the independent variables included in the data sets relate to the broad factor economic conditions, 22 to fiscal performance, two to financial and debt position, and only one to management quality and institutional strength. The remainder of the independent variables were indicators and did not relate to a specific broad factor.

Methodology applied to develop linear regression models

The following methodology was applied to develop the 30% quantitative part of a credit rating methodology for the Combined, Health and Education data sets. Multicollinearity was dealt with first by using variable clustering - one variable was chosen from each cluster of variables as the cluster representative. Irrelevant independent variables were removed based on the variables' correlations with the dependent variable. Lastly, the stepwise variable selection technique and a significance level of 10% were used to choose the final independent variables. The models developed via this process will henceforth be referred to as the Basic Models.

The Basic Models were further investigated to improve the R² via two model advancement techniques, namely including interactions between the independent variables in the model (Advanced Models 1) and excluding outliers from the data (Advanced Models 2). All the models were developed using SAS Institute Inc. (2011) software.

Note that Kutner et al. (2005) state that models that include interaction terms are considered to be complex models and SAS Institute Inc. (2010) states that the inclusion of polynomial terms complicates interpretation. Hair et al. (2006) states that if there is no explanation available for an outlier, the analyses should be done with the outlier included and excluded since neither the inclusion nor exclusion can be warranted. As was the case for outliers found in this part of the study since outliers that were caused by data capturing errors they were corrected during the capturing phase.

Furthermore, one can argue that the models taking into account interactions and excluding outliers will always be the better models when used for explanation purposes. However, the aim of this study was to predict and therefore the developed models need to be able to generalise too. A common pitfall in predictive modelling is to overfit to the data. An overly complex model (which may be the case for Advanced Models 1 and 2) might be too sensitive to peculiarities in the development sample data set and therefore may not be able to generalise well to new data. However, using too simple a model (possibly the Basic Models) can lead to underfitting, where true features are disregarded (SAS Institute Inc. 2010).

For these reasons, the results of all three models per data set are reported next and are also later on compared in terms of predictability.

 

Results

The results of the basic and advanced linear regression models from all three data sets are documented below. The parameter estimate of an independent variable indicates the change in the dependent variable per one unit increase of the corresponding independent variable when the other independent variables are held constant (Kutner et al. 2005). The explanations provided for the independent variables' effects on future payment behaviour assume that all other independent variables remain constant.

Generic linear regression models

The methodology discussed in the previous section was applied to the combined data set to develop the generic linear regression models.

The Basic Model has an R² of 0.49, which means that it explains 49% of the variance of the dependent variable. The p-values of all six chosen independent variables are smaller than 0.1, the highest being 0.07 (see Table 1).

The resulting Generic Basic Model's formula is:

Where i = 1,2,···,18.

The R² of Generic Advanced Model 1 is 0.55, thus it improved from 0.49 to 0.55. The highest p-value is 0.08. The details are in Table 2. No outliers were identified, thus no Generic Advanced Model 2 was developed.

The equation of the Generic Advanced Model 1 including interaction terms is:

where i = 1,2,···,18.

Specific linear regression models

The results of the models specifically developed for the Departments of Health and Education are documented next.

The Basic Model developed specifically for the Department of Health (Health Basic Model) has an R² of 0.51. The p-values of all five independent variables chosen for this model are smaller than 0.1 (see Table 3). The model can be expressed as:

where i = 1,2,···,9.

The Basic Model developed specifically for the Department of Education (Education Basic Model) has an R² of 0.48, which means that it explains less than 50% of the variance of the dependent variable. Details in Table 4 below. The highest p-value of the independent variables chosen for this model is 0.06. The model can be expressed as:

 

Table 4 - Click to enlarge

 

where i = 1,2,···,9.

For both the Department of Health and the Department of Education, two unique models resulted from the model advancement investigation. More details of these two models can be found in Table 5 below.

 

Table 5 - Click to enlarge

 

The R² of the Health Advanced Model 1 increased to 0.74 and Health Advanced Model 2 to 0.80. The p-values are all smaller than 0.1.

The equation of the Health Advanced Model 1 is:

 

where i = 1,2,···,9.

The equation of the Health Advanced Model 2 is:

where i = 1,2,···,9.

Both of the Education Advanced Models' R²s (0.70 and 0.74) were higher than that of the Education Basic Model (0.48). All the independent variables' _p-values were smaller than 0.1. Details of these two models are in Table 6 below.

Table 6 - Click to enlarge

 

The equation of the Education Advanced Model 1 is:

where i = 1,2,···,9.

The equation of the Education Advanced Model 2 is:

where i = 1,2,···,9.

In total, two generic models and six specific models were developed. In the 'Testing the appropriateness of the proposed models' section, these eight models will be compared and tested for appropriateness.

Assigning Red, Amber or Green statuses

The ultimate aim of the developed models is not to predict a department's numerical value of the dependent variable (i.e. the proxy of payment behaviour), but rather to rank the departments' expected future payment behaviours in terms of RAG statuses. RAG statuses can be assigned based on absolute or relative rankings.

Absolute rankings make use of the absolute values of the dependent variable. For example, a red status is assigned when the ratio of accruals 30 days plus/total expenditure is higher than 150% and a green status is assigned if the ratio is lower than 40%. Literature suggests a number of different cut-offs that could be used to assign RAG statuses in this manner, see for example Moody's (2008) and S&P (2010).

However, as a proxy was used to determine payment behaviour, RAG statuses are assigned based on relative rankings for the purpose of this article. The following procedure was used to allocate the RAG statuses: the bottom third departments (bad payment behaviour based on high values of the payment behaviour proxy) are assigned red statuses, the middle third departments (average payment behaviour) are assigned amber statuses and the top third departments (good payment behaviour based on low values of the payment behaviour proxy) are assigned green statuses.

 

Testing the appropriateness of the proposed models

The aim of this section is to test the appropriateness of the developed models, that is, to decide on a model to be used in future to predict payment behaviour. With this as aim, this article includes a comparison of, among others, the number of correctly predicted RAG statuses of the Basic Models and the Advanced Models for both the generic and the specific developments (Sections 'Validation' and 'Model comparison'). This comparison is used in the 'Model recommendation' section to provide recommendations with regard to which model should be used in future to rank the departments in terms of expected payment behaviour. The recommended model is then used to compare the RAG statuses predicted by the model with the actual RAG statuses of the financial year 2012/2013 in the 'Predicted versus actual ranks and Red, Amber or Green statuses' section. Lastly, an example is provided in the 'Example: Deriving the final ranking of Gauteng's Department of Health' section. This illustrates how the RAG statuses predicted by the models based on quantitative information can be overlaid with some more recent qualitative information to derive a final ranking or RAG status. Dashboards were constructed to illustrate this.

Validation

Kutner et al. (2005) recommend that, after development, new data should be collected and the model should be reapplied to verify its predictive ability. This method was used to validate the developed payment behaviour models. The data from the previously discussed sources were used, but for the next available financial years. Thus, the 2011/2012 financial year's data were used for the independent variables to predict the values of the dependent variable for 2012/2013. Note that typically a 2012/2013 data set is only available in 2014.

The RAG statuses predicted were compared with the actual RAG statuses and the number of correctly assigned RAG statuses was counted as a measure of the model's predictive ability. The lowest number of RAG statuses correctly predicted by the models was two out of seven (22.2%) and the highest number was seven out of the nine (77.8%).

Model comparison

The model comparison is based on the R², the number of data points deleted, the number of terms and independent variables included in the models, the number of data sources needed to update the models, the number of factors represented by the independent variables, and the number of correctly predicted RAG statuses of each of the three resulting models. This information is summarised in Table 7.

The model with the highest R² is the Health Advanced Model 2 at 0.80, with only one data point omitted. The minimum number of terms included in the models is four and the maximum is nine. The number of independent variables included range between two and 10. The data sources needed to update the models range between three and five. The independent variables included in the final models represented two to three of the broad factors. The two Health Advanced Models and Education Advanced Model 2 predicted seven of the nine (77.8%) RAG statuses correctly.

Model recommendation

When considering the generic models, Generic Advanced Model 1 predicted nine out of 18 (50%) RAG statuses correctly, taking into account that these models are used to predict the nine Departments of Health plus the nine Departments of Education's payment behaviour simultaneously. Health Advanced Model 2 and Education Advanced Model 2 predicted 77.8% correctly.

Thus, the specific models outperformed the generic models and it is therefore recommended that the specific models (Health Advanced Model 2 and Education Advanced Model 2) should be used in future. Literature also indicated that models tailored for specific populations outperform generic models (Anderson 2007; Thomas 2000). One explanation is that the difference in the functions that the departments perform equates to two different types of populations and causes different information to be important for the different departments. Thus, different independent variables (and therefore models) are needed to predict the payment behaviour of the different departments.

Predicted versus actual ranks and Red, Amber or Green statuses

Health Advanced Model 2 was the model with the highest R² and also one of the models predicting the highest number of RAG statuses correctly. Thus, this model will be used in the illustrations provided. The ranks and RAG statuses of the provinces' Departments of Health predicted by the interaction model without independent outliers are shown in Table 8. The predicted values for 2012/2013 are based on the information of the independent variables contained in the data of the financial year 2011/2012. The actual values are the ranks and RAG statuses based on the true outcomes of the dependent variable, as documented in the annual reports of the financial year 2012/2013. Only two provinces' RAG statuses were predicted incorrectly, namely Limpopo and the Eastern Cape.

 

 

Example

Deriving the final ranking of Gauteng's Department of Health

An example (focusing on Gauteng's Department of Health) is provided next to demonstrate how the developed models could be used in future. The predicted ranking with regard to the other provinces, based on quantitative information, was nine.

As mentioned previously, credit rating methodologies are normally based on a combination of quantitative and qualitative factors. Qualitative information will be assessed by a team of analysts and then used as input in the model. Typically, credit rating methodologies for developing countries comprise 30% quantitative factors and 70% qualitative factors (Moody's 2007). Thus, qualitative information should also be taken into account.

An example of qualitative information that should form part of the intuitive overlay is the news article issued by SABC News on 30 April 2014. This article states that the Zola-Jabulani Hospital building project was finished 5 years later than planned and the actual costs were three times higher than what was budgeted (SABC News 2014). This might indicate a lack of management sophistication and poor financial planning. Another example of inadequate management and financial planning is provided by a press release issued by the Gauteng Department of Health (2014), where the department states that it did experience challenges of late or non-payment in the past, but the problem is being rectified. Also, the department's annual report of 2012/2013 reported that a turnaround strategy has been deployed, focusing, among other things, on financial management. The figures reported in this annual report did in fact illustrate an improvement in the debt repayment numbers. The ratio of accruals in excess of 30 days to total expenditure did improve in recent years (10.05% in 2011/2012, 4.49% in 2012/2013, 3.36% in 2013/2014). However, it is yet to be seen if this improvement is sustainable.

Typically, the above-mentioned information, as well as any other available qualitative information, will be reviewed by a team of analysts. The analysts will then rank Gauteng's Department of Health's expected payment behaviour with regard to the other provinces by taking into account their qualitative information as well. However, this example focuses only on Gauteng and therefore the other provinces' qualitative information is not provided. For illustrative purposes, it is assumed that Gauteng has an average red status, thus a ranking of eight, based on provided qualitative information.

Taking both the quantitative and qualitative rankings into account, the final ranking of Gauteng's Department of Health is about eighth. This is derived as follows: 30%×9+70%×8=8.3. Thus, the RAG statuses based on the quantitative (ranking = 9) and qualitative (ranking = 8) information, as well as the final RAG status (ranking = 8) were all red. In this case, the qualitative information reinforced the red status that was assigned based on the quantitative information (illustrated in Figure 2).

 

 

Conclusion and suggestions for future research

The primary contribution of this study was to develop and compare both a generic and a specific subnational credit rating methodology (to date no such methodologies have been developed for South African provinces and departments). A further contribution was to test the appropriateness of these generic and specific models about prediction accuracies using RAG statuses, where the specific subnational credit rating methodology was shown to outperform the generic methodology by far.

Models resulting from the combined data sets (i.e. generic models) should not be used for future predictions. The generic models predicted a maximum of 50% of the new cases correctly as opposed to the specific Health and Education models' 78%. The payment behaviour models developed specifically for the Department of Health or Education therefore performed better than the generic models in terms of predicting new data and should rather be used.

Thomas (2000) and Anderson (2007) confirmed that models tailored for specific populations outperform generic models. Again, one possible explanation in this specific case is the difference in the functions that the departments perform, which equates to two different types of populations and results in different information to be important to the different departments. Thus, different independent variables (and therefore models) are needed to predict the payment behaviour of the different departments. For example, in order for the Departments of Health to deliver the required health services, sophisticated medical equipment is needed. In this case, independent variables containing surplus/deficit information is of importance since surpluses could be used to purchase the needed medical equipment. The Department of Education, however, must ensure that compensation to employees do not fall into arrears to avoid political unrest. Therefore, personnel expenditure information is of interest. However, the independent variables containing surplus/deficit information and the independent variables containing personnel expenditure information represents the same broad factor: fiscal performance. Thus, the same broad factors are represented by the independent variables included in the different department-specific models.

Lehmann (2003) has shown that the use of qualitative variables improves predication quality, which has been confirmed by the example used in the 'Example: Deriving the final ranking of Gauteng's Department of Health' section. Because of the nature of the data used, the payment behaviour or subnational credit rating methodology developed in this article assesses only quantitative information, so models could be improved by also incorporating qualitative variables. Other data sources should be investigated to collect more data to represent the broad factors for financial and debt position as well as management quality and institutional strength. These two principal factors are only represented by two and one variable(s), respectively.

 

Acknowledgements

The authors gratefully acknowledge the valuable input given by anonymous referees. This work is based on research supported in part by the Department of Science and Technology (DST) of South Africa. The grant holder acknowledges that opinions, findings and conclusions or recommendations expressed in any publication generated by DST-supported research are those of the author(s) and that the DST accepts no liability whatsoever in this regard.

Competing interests

The authors declare that they have no financial or personal relationships that may have inappropriately influenced them in writing this article.

Authors' contributions

This article originated from E.F.'s PhD. T.V. was the PhD promotor and G.W.v.V. was the PhD co-promotor.

 

References

Anderson, R., 2007, The credit scoring toolkit, Oxford University Press Inc., Oxford.         [ Links ]

Auditor-General, 2007-2011, Consolidated general reports on the national and provincial audit outcomes 2006/07 to 2010/11, The Auditor-General of South Africa, Pretoria.         [ Links ]

Australia Ratings, 2014, State territory government methodology, viewed 03 February 2014, from http://www.australiaratings.com/stateterritorygovernmentmethodology        [ Links ]

Burazeri, A., Dhuci, O. & Kufo, A., 2014, 'Credit scoring process avoiding the excessive risk', Journal of Applied Economics and Business 2(1), 92-104.         [ Links ]

Chambers, M. & Dinsmore, T.W., 2014, Predictive analytics techniques, Pearson Education, FT Press, Upper Saddle River, NJ.         [ Links ]

Credit Rating Information Services of India Limited (CRISIL), 2011, Rating criteria for state governments, viewed 21 November 2011, from http://crisil.com/Ratings/SectorMethodology/MethodologyDocs/CRISIL-Ratings-crieria-stategovernments2007.pdf        [ Links ]

Dominion Bond Rating Service Limited (DBRS), 2011, Methodology: Rating Canadian provincial governments, DBRS, Toronto.         [ Links ]

Eastern Cape Department of Education, 2007-2013, Annual report, Eastern Cape Department of Education, Zwelitsha.         [ Links ]

Eastern Cape Department of Health, 2007-2013, Annual report, Eastern Cape Department of Health, Zwelitsha.         [ Links ]

Fitch, 2011, International local and regional governments rating criteria: Outside the United States, Fitch Ratings, New York.         [ Links ]

Fourie, E., Styger, P., De la Rey, T. & Van Vuuren, G., 2013, 'Review of subnational credit rating methodologies and the applicability in the South African context', Politeia 32(3), 74-99.         [ Links ]

Fourie, E., Verster T. & Van Vuuren, G., 2016, 'A proposed quantitative credit rating methodology for South African provincial departments', South African Journal of Economic and Management Sciences 19(2), 192-214.         [ Links ]

Fourie, E., Verster T. & Van Vuuren, G., 2017, 'Die toepassing op en validering van 'n voorgestelde subnasionale kredietgraderingsmetodologie vir die Departement van Onderwys, met 'n gevallestudie', LitNet Akademies (Ekonomiese en Bestuurswetenskappe) Jaargang 14(3), viewed n.d., from http://www.litnet.co.za/die-toepassing-op-en-validering-van-n-voorgestelde-subnasionale-kredietgraderingsmetodologie-vir-die-departement-van-onderwys-met-n-gevallestudie/        [ Links ]

Free State Department of Education, 2007-2013, Annual report, Free State Department of Education, Bloemfontein.         [ Links ]

Free State Department of Health, 2007-2013, Annual report, Free State Department of Health, Bloemfontein.         [ Links ]

Gauteng Department of Education, 2007-2013, Annual report, Gauteng Department of Education, Johannesburg.         [ Links ]

Gauteng Department of Health, 2007-2014, Annual report, Gauteng Department of Health, Johannesburg.         [ Links ]

Gauteng Department of Health, 2014, Press release: Gauteng health ensures the availability of food in its facilities, viewed 13 November 2015, from http://www.gov.za/gauteng-health-ensures-availability-food-its-facilities        [ Links ]

Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E. & Tatham, R.L., 2006, Multivariate data analysis, 6th edn., Pearson Prentice Hall, Upper Saddle River, NJ.         [ Links ]

Kutner, M.H., Nachtsheim, C.J., Neter, J. & Li, W., 2005, Applied linear statistical models, 5th edn., McGraw-Hill/Irwin, New York.         [ Links ]

KwaZulu-Natal Department of Education, 2007-2013, Annual report, KwaZulu-Natal Department of Education, Pietermaritzburg.         [ Links ]

KwaZulu-Natal Department of Health, 2007-2013, Annual report, KwaZulu-Natal Department of Health, Pietermaritzburg.         [ Links ]

Lehmann, B., 2003, Is it worth the while? The relevance of qualitative information in credit rating, EFMA 2003 Helinski Meetings, Helsinki, viewed 04 June 2014 from https://ssrn.com/abstract=410186        [ Links ]

Limpopo Department of Education, 2007-2013, Annual report, Limpopo Department of Education, Polokwane.         [ Links ]

Limpopo Department of Health, 2007-2013, Annual report, Limpopo Department of Health, Polokwane.         [ Links ]

Liu, L. & Tan, K., 2009, Subnational credit ratings: A comparative review, Policy Research Working Paper 5013, The World Bank, New York, NY.         [ Links ]

Liu, L. & Waibel, M., 2008, Macro federalism and local financing, insolvency and regulation, The World Bank, Washington, DC.         [ Links ]

Moody's, 2007, Bank financial strength ratings: Global methodology, Moody's Research, London.         [ Links ]

Moody's, 2008, Rating methodology: Regional and local governments outside the US. Updated rating methodology, Moody's Investors Service, London.         [ Links ]

Moody's, 2014, Rating symbols and definitions, Moody's Investors Services, Inc., viewed 04 June 2014, from https://www.moodys.com/researchdocumentcontentpage.aspx?docid=PBC79004        [ Links ]

Mpumalanga Department of Education, 2007-2013, Annual report, Mpumalanga Department of Education, Nelspruit.         [ Links ]

Mpumalanga Department of Health, 2007-2013, Annual report, Mpumalanga Department of Health, Nelspruit.         [ Links ]

Municipal IQ, 2015, Welcome to Municipal IQ, viewed 11 November 2015, from www.municipaliq.co.za        [ Links ]

National Treasury, 2007a-2011a, Fourth quarter year to date provincial budgets and expenditure report (preliminary outcome) 2007/8 to 2010/11, The National Treasury of South Africa, Pretoria.         [ Links ]

National Treasury, 2007b-2011b, Provincial statements of receipts and payments for the 4th quarter ended 31 March 2007 to 2011, The National Treasury of South Africa, Pretoria.         [ Links ]

National Treasury, 2010, Public sector risk management framework, National Treasury of South Africa, Pretoria.         [ Links ]

National Treasury, 2012, Preliminary results of risk grading of provincial governments (Workshop held on 8 November 2012), The National Treasury of South Africa, Pretoria.         [ Links ]

National Treasury, 2013a, Workshop discussing: A proposed credit rating methodology for South African provincial governments, The National Treasury of South Africa, Pretoria.         [ Links ]

National Treasury, 2013b, Structure of the government accounts. Website annexure to the 2013 budget review, The National Treasury of South Africa, viewed 02 August 2013, from http://www.treasury.gov.za/documents/national%20budget/2013/review/Annexure%20W2.pdf        [ Links ]

National Treasury, 2016, Fourth quarter year to date provincial budgets and expenditure report (preliminary outcome) 2015/16, The National Treasury of South Africa, Pretoria.         [ Links ]

North West Department of Education, 2007-2013, Annual report, North West Department of Education, Mafikeng.         [ Links ]

North West Department of Health, 2007-2013, Annual report, North West Department of Health, Mafikeng.         [ Links ]

Northern Cape Department of Education, 2007-2013, Annual report, Northern Cape Department of Education, Kimberley.         [ Links ]

Northern Cape Department of Health, 2007-2013, Annual report, Northern Cape Department of Health, Kimberley.         [ Links ]

S&P, 2010, Methodology for rating international local and regional governments, McGraw-Hill Companies, New York.         [ Links ]

S&P, 2011, Guide to credit rating essentials: What are credit ratings and how do they work? McGraw-Hill Companies, New York.         [ Links ]

SABC News, 2014, Minister admits to R1 billion 'error' at Zola-Jabulani hospital, viewed 10 June 2014, from http://www.sabc.co.za/news/a/c30a830043d3f5f4beacbef0c0fe2c4c/Minister-admits-to-R1-billion-'error'-at-Zola-Jabulani-hospital-20143004        [ Links ]

SAS Institute Inc., 2010, Predictive modelling using logistic regression course notes, SAS Institute Inc., Cary, NC.         [ Links ]

SAS Institute Inc., 2011, The SAS system for windows release 9.3 TS level 1M0, Copyright© by SAS Institute Inc., SAS Institute Inc., Cary, NC.         [ Links ]

Scott, D.L., 2003, Wall street words: An A-Z guide to investment terms for today's investor, 3rd edn., Houghton Miffin Company, Boston, MD.         [ Links ]

Siddiqi, N., 2006, Credit risk scorecards: Developing and implementing intelligent credit scoring, Wiley, Hoboken, NJ.         [ Links ]

South Africa, 1996, Borrowing Powers of Provincial Governments Act 48 of 1996, viewed 11 November 2015, from http://www.gov.za/sites/www.gov.za/files/Act48of1996.pdf        [ Links ]

South Africa, 1999, Public Finance Management Act 1 of 1999, viewed 21 January 2011, from http://www.treasury.gov.za/legislation/PFMA/act.pdf        [ Links ]

South Africa, 2001, Treasury regulations for departments, constitutional institutions and public entities. Issued in terms of the Public Finance Management Act, 1999, viewed 21 January 2011, from http://www.treasury.gov.za/legislation/pfma/regulations/gazette22219.pdf        [ Links ]

South Africa, 2017, Provincial Government, viewed 21 April 2017, from http://www.gov.za/links/provincial-government        [ Links ]

StatsSA, 2012, Mid-year pop estimates by province, gender, age group and year, viewed 15 November 2012, from http://www.statssa.gov.za        [ Links ]

StatsSA, 2013, Gross domestic product. Annual estimates 2003-2012. Regional estimates 2003-2012. Third quarter 2013. Statistical release P0441, Statistics South Africa, Pretoria.         [ Links ]

Thomas, L.C., 2000, 'A survey of credit and behavioural scoring: Forecasting financial risk of lending to consumers', International Journal of Forecasting 16, 149-172. https://doi.org/10.1016/S0169-2070(00)00034-0        [ Links ]

Western Cape Department of Education, 2007-2013, Annual report, Western Cape Department of Education, Cape Town.         [ Links ]

Western Cape Department of Health, 2007-2013, Annual report, Western Cape Department of Health, Cape Town.         [ Links ]

 

 

Correspondence:
Gary van Vuuren
vvgary@hotmail.com

Received: 17 Nov. 2016
Accepted: 24 May 2017
Published: 10 Oct. 2017

 

 

1. Credit rating agencies summarise their opinions about the creditworthiness of an issuer or issue in ratings that are graded by symbols such as AA−, BBB+ and Caa1. Issuers or issues with the same rating symbols will portray the same credit characteristics and hence have the same relative creditworthiness. See for example Moody's (2014) and S&P (2011).