Empirical Evaluation of a Preliminary Model to Identify Low-Risk mba Applicants

This article reports on the second stage of the model, namely to empirically evaluate the model’s performance and validity across all three of the identified categories. These categories are ‘Low-to-no risk’ applicants for the mba; and those applicants who did not complete the degree in 3 years, classified as ‘Medium-to-low risk’ applicants who are expected to complete their degree in extended study year, and ‘Highrisk’ applicants who are not expected to complete their degrees and who drop out of the programme. The final-year mba students at the pbs in 2004 and 2005 served as the research population. The results were very satisfactory. Concerning the categories Low-to-no risk and Medium-tolow risk applicants, the model can be used as predictive tool, presenting a validity higher than 60% (p = 0.9) and 90% (p = 0.7) respectively. Caution, however, looms at the category of High-risk applicants where the model judges too harshly with an error of 13.7% (p = 0.7).


Introduction
This article empirically evaluates the findings of the research by Bisschoff (2005, 300-309) on a predictive model for mba applicants.The focus of this article is, therefore, to report on the empirical evaluation of the already developed predictive model.Although a short summary of the model and history of the research is provided as a frame of reference to readers, those readers who wish to obtain more detailed information about the stages of model development and the model itself, are referred to the article as published in the South African Journal of Economic and Management Sciences.Table 1 shows the research project and the progress.
In addition, the results are discussed and contextualised with similar studies that aimed to predict mba success rates by making use of selection criteria.This discussion follows after the results of this study were presented.

Historical Perspective and Context
Throughout the world academic institutions are constantly aware of the discrepancy between the number of students who enrol for tuition, and those who actually obtain the qualification.This erosion effect of students is two-fold, namely the complete academic dropouts and those students who change their study discipline to some related or unrelated course.In both events, this erosion results in the loss of revenue for the academic institution, ineffective time spent by lecturers on students not completing or passing the course and possible damaging perceptions to the image of the institution.On the part of the student, a loss of money and time, damaged self-esteem and lower self-confidence are but a few of the consequences of failing or dropping out of academic programmes.
Regarding the degree Masters in Business Administration (hereafter referred to as mba), the market value and image surrounding it, makes it a favourable option for post-graduate management studies.As a result, a vast number of applicants strive for acceptance at business schools on their mba programmes.This market demands further emphasis on the responsibility of business schools to admit suitable applicants.
However, although most business schools in South Africa and the First World apply clear and above board criteria in their admission process, few have been able to predict the success of their students during admission to the programme.The Potchefstroom Business School (hereafter referred to as pbs), therefore, aimed to construct a tailormade model that could fulfil this purpose, namely to predict outcomes of mba students applying to be admitted on the mba programme.The model aimed to: • Primarily: introduce a preliminary model with the purpose of predicting the academic success of an mba applicant (thus the risk to

Managing Global Transitions
drop out or fail) by categorising them into 'medium to low risk' (consisting of 230 cases) and 'low to no risk' applicants (consisting of a total of 368 cases); and to • Secondarily: identify 'high-risk' applicants (those applicants that show a high probability to fail or drop out of the mba programme consisted of a total of 118 cases).The initial research to construct the model was conducted amongst a total of 729 respondents, consisting of mba graduates and mba dropouts.All the mba students who studied at the pbs between 1998 and 2003 were used in the research.The first stage of research aimed to construct the predictive model, using Discriminant Analysis as a statistical tool (Bisschoff 2005).The specific choice to use discriminate analysis, as an evolved multiple regression technique, was based on its strong classification capabilities which are required to classify the mba students into the three risk groups envisaged (Youngblood andMartin 1982, 1157-8;Clayton and Cate 2004, 238;Steyn 2006).
This article serves as the second stage of the research, and reports on the validation of the aforementioned predictive model, thus, to evaluate its correctness in its predictions in quantitative terms.Obviously this is a first step in the validation process, and the process needs to be repeated to substantiate the results obtained in this round of evaluation.For this part of the process, 2004 and 2005 graduates and drop-outs were evaluated.

Problem Statement
Although the primary problem (as stated in the mentioned article on the construction of the model) remains to admit the 'right' students to the mba programme, this article is more concerned in determining whether the model could be used as a reliable predictive tool to assist in the admission process.In essence, the research in this article reports on the results obtained in the scientifically conducted evaluation and resulting conclusions regarding the validity of the predictive model.Therefore, the problem on is faced with is if the model is suitable to make predictions to admit the 'right' students onto the mba programme of the pbs.As a result, this article reports on the second stage of the research project, namely to empirically evaluate the soundness of the model's predictive powers.

Theoretical Base: A Summary of the MBA Admission Tests
As discussed by Bisschoff (2005, 302-4) business schools try (with varying success) to increase the quality of applicants via specified admission criteria, and an array of screening tests.Although these criteria and tests may be successful in increasing 'quality' or suitability of admittees, their predictive capabilities are unproven in the fields of management and leadership (a core discipline in mba education).
In summary, these tests include: • General Management Admission Test (gmat).Although being one of the most popular admission tests, the gmat is also fairly criticised.Closer investigation reveals not only a debate on the validity of tests and their predictive value, but also contrasting research results.The major controversy seems to be not the validity of gmat, but rather the significance of the positive correlation between performance in the test and business success.Hong (2001:1) pointed out that strong performance on gmat does not necessarily translate into success in business, while Arbor (2001: 1) highlights the value of gmat to be an internationally standardised test.Standard gmat scores are also influenced by experience, and although Tyson (2002, 33) regards it to be a fair predictor of performance, it is pointed out by Sternberg, et al. (2000) that in a study by Hedlund (1999Hedlund ( -2001) ) it was found that disparity exists in the scores by gender, race and nationality.Although various researchers such as McCloud (1991), Nebel (2002) and Wilt (in Hindo 2002) scrutinised gmat as a predictive tool, some other researchers (Sternberg 1997(Sternberg -2001;;Hedlund 1999Hedlund -2001)), according to Tyson (2002, 34-7), engaged in new research on predictive models.
• Test of English as a Foreign Language (scored on a scale of 200 to 677 and commonly referred to as the toefl test).This test is important in the assimilation of studies, and a number of universities employ it (in addition to other tests) to ascertain that language does not constitute educational barriers (University of Texas 2004).
• shl tests is an application test that is utilised in a number of business schools in the rsa such as the pbs, Graduate School of Business and Graduate School of Management (usb 2006;up 2006 andpbs 2006).The company shl is well established in more than 40 countries on a global scale, and is also active in the human resources sciences.The tests closely resemble gmat and in admission applications report on communicative skills, numerical ability and group functioning to name but a few of the test areas (Scholtz 2006).
• Standardised and/or in-house tests such as the Thinking test at the

Managing Global Transitions
University of Arcadia ( 2004) require applicants to apply basic science knowledge, maths and verbal skills to reading comprehension, biology, verbal skills, quantitative analysis and chemistry (thus also testing ability in non-business related fields for admission to their mba programme).
• Common Admission Test (cat) predicts academic success in the business application setting by categorizing student admissions on mba (and other) programmes.According to the University of Arcadia (2004) in Philadelphia, usa it is a pre-requisite to qualify for any of the post-graduate programmes.The cat, which is regarded to be one of the toughest qualifying or entrance tests in the world, tests two subjects, namely English and Maths.These subjects are subdivided further into areas of Quantitative Ability, Data Interpretation, Data Sufficiency, Logical Reasoning, Reading Comprehension, Verbal Reasoning and Verbal Ability (ims Learning resources 2004).
• Successful Intelligence Assessment (sia) by Sternberg and Hedlund which consists of a model to predict successful business, workplace and academic performance (Holmes 2004, 1-2).The core of the model resides in problem-solving abilities.Two core problems are employed, namely case-based and situational-judgement problems (Holmes 2004, 2).The focus of the test concentrates primarily on the quality of the response, and secondarily, on the actual correctness thereof (Oswald et al. 2004, 3;Tyson 2002, 1).The researchers found that the higher the scores on these two criteria, the better mba students fared in their grade point averages (Arbor 2001, 1), as well as in academic discussions and in leadership positions (Holmes 2004, 2;Hong 2001, 1).
In addition to these tests, Truitt (2002, 2) points out that the use of the undergraduate Grade Point Average (gpa) can also serve as a predicting variable of student success as it shows significant positive correlations between success on undergraduate level and success on the mba programme.However, caution should be taken because a student with a high gpa point should perform better in any post-graduate programme (including mba).The gpa is, therefore, a useful but general predictor of academic performance, and not per se a predictor of success on an applied managerial education programme such as the mba (Truitt 2002, 3).
Typically, entrance to an mba programme has certain pre-requisites.Analysis of three Ivy League business schools reveals that both gmat and toefl tests are employed in their admission procedure.Harvard Business School requires a gmat score of 705 or higher, Stanford requires a gmat (with no minimum score) as well as a toefl score of 250 or higher, while at Wharton Business School 'satisfactory' scores on both gmat and toefl tests are required (Harvard Business School 2004; Stanford Graduate School of Business, 2004;Wharton Business School 2004).Be it as it may, there are as many different admission criteria as there are different business schools.This is also true for business schools in South Africa where some of the business schools make use of gmat and/or other supplementary tests.
A new avenue of research was pursued in a joint research project between the Universities of Yale and Michigan in the usa by the researchers Sternberg (1997) and Sternberg andHedlund (1997-2001).The latter of the two projects is regarded to be one of the most significant relating to predictability of business and academic success (Tyson 2002, 34).
In conclusion to pre-testing with the purpose of predicting performance, it is clear that various models and tests do exist.Naturally, some seem to perform better than others.Generic or standardised tests have the advantage of a wide application setting.Customized tests, on the other hand, have a limited application setting, but could lead to better results designed for specific business schools.

Research Methodology
The second stage research, namely to validate the model, evaluated 378 student cases (the actual academic record of a student and the prediction by the model for this student represents a case).A total of 18 students have dropped out of the mba programme, either by their own choice or by disqualification due to unsatisfactory academic performance.They were discarded from the research project.The remaining 360 students obtained their mba degrees at the end of either 2004 or 2005.This represented a continuous database of 2 years of mba graduates that served as research cases.The research population was limited to pbs students.All the students either graduated or exited the mba programme (due to poor academic performance or otherwise) by December 2005.Table 2 shows the research subjects employed to evaluate the validity of the predictive model.
The research makes use of a comparative analysis between the predictive powers of the model and the actual outcome of the specific student.It thus employs 'hindsight as perfect science' to validate whether the model correctly or incorrectly classified a student (based on his/her historical data during application for admittance onto the mba programme) as being a high-, medium-or low-risk applicant.This classification is then compared to the actual outcome and study record of the specific student to verify if the model succeeded in correctly classifying the student.

Managing Global Transitions
As such the predictive model is empirically evaluated case-by-case, and the validity is determined.The statistical analysis was performed by the software Statistical Analysis System (sas), while data-recording and the comparative spreadsheet analysis made use of Microsoft Excel.
The research methodology to evaluate the validity of the predictive model is shown in figure 1.
The cut-off value applicable to the comparative analysis has been set at a minimum probability of 0.50, signifying that at least 50% of cases need to be predicted correctly for the model to be even considered useful.Clearly, with a success rate of less than 50%, the model would not be suitable for further use (Steyn 2006).In addition, the probabilities have also been increased to 0.60, 0.70, 0.80, 0.90 and eventually to 0.95 in an attempt to identify the point-of-breakdown for the predictive model.

Results
The analysis has been performed on all three of the identified categories, namely to identify Low-to-no risk applicants, Medium-to-low risk applicants, and High-risk applicants.

low-to-no risk applicants
The analysis made use of the data representing all cases where the students obtained their mba degree within the minimum time of 3 years.This constituted a research population of mba graduates from 2004 (n = 149) and 2005 (n = 163).The probability of success of the predictive model for both the years 2004 and 2005 is shown in table 3. From the table it is clear that the model is successful in predicting if the applicants will complete their degrees within the 3 years as required.
Table 3 clearly shows that up to a probability of 0.9 the predictive

medium-to-low risk applicants
The results from the analysis of the Medium-to-low risk applicants' category revealed that the predictive model also succeeds in predictinf if an applicant is not suitable to complete the degree in 3 years, he/she will complete it in 4 years.Once again, table 4 shows that very satisfactory predictions result from applying the model to the second category of mba applicants (above 0.90 for 2005 while above 0.85 for 2004).A total of 22 and 24 cases were respectively analysed for the years 2004 and 2005.
The model, however, breaks down at the 0.80 probability level, and is thus highly predictive up to a probability of 0.70.The years 2004 and 2005 follows the same trends once again in the decline of the predictive powers.
Although these predictions are very satisfactory and well above expectations, it is also important to employ the model inversely, namely to determine to what extent the model disqualifies applicants from the programme when these applicants actually completed the programme successfully.These applicants would fall into the high-risk category.

high-risk applicants
The model was used to analyse all the cases (360) to identify high-risk applicants.Regarding the No-to-low risk category (312 cases), the model did identify 3 cases incorrectly to be of a Medium-to-low risk (p = 0.6 and p = 0.7).However, when the analysis was performed on the remainder of the cases, namely the Medium-to-low risk applicants, the model incorrectly identified another 4 cases to be of high-risk.Although this constitutes an error of only 1.9% of the total population, it results in a predictive error of 13.7% (p = 0.6 and p = 0.7) in the Medium-to-low risk category where, in total, 7 applicants were classified incorrectly.See table 5.
Although an error of 13.7% is statistically acceptable, the fact remains that the model disallowed 7 people to follow their dream and to study towards an mba.Therefore, the model should be very carefully used as screening tool if it is used as such to disqualify applicants to enrol for an mba.
In summary (with reference to tables 3 and 4), it is clear that the discriminant functions 1 and 2 (Bisschoff 2005, 306) display very satisfactory predictive powers in the categories No-to-low risk applicants and Medium-to-low risk applicants.However, concerning the discriminant functions dealing with the high-risk applicants, the functions seem to be a bit too harsh a judge of the applicant.These functions, although delivering statistically acceptable margins of error, incorrectly disallow 13.7% of the Medium-to-low risk applicants as students.The model is, therefore, valid to be employed in the first two categories as predictive tool, but care has to be taken when employing the model as predictive tool in the high-risk category.

Discussion
Academic institutions all over the world are pressurised to increase the flow-through of students on academic programmes.This is especially relevant to public universities and colleges who receive, according to the subsidy formula, funding from the state (Elof 2011).Amongst an array of possible interventions such as better facilities, improved educational techniques, educator education levels, and more, features also the striving for a better quality student, and allowing students onto programmes they have an aptitude for.This is also true for the mba degree.
Resultantly, as early as 1974, research was already focused on how to ensure that the applicants allowed onto the mba degree of the Carnegie-Mellon University in New York, would be successful (Weinstein and Srinivasan 1974).Other schools and researches soon followed suit.Some of these studies also used discrimant analysis (as a variant of multiple regression) to predict mba success (Clayton and Cate 2004), other researchers used multiple regression (step-wise and logistic) (Weinstein and Srinivasan 1974;Yang and Lu 2001;Truitt 2002), while the neural network was also employed as predictive tool in one study (Naik and Srinivasan 2004).Other researchers sought merely for relationships between success and selection criteria, following the less complicated statistically pathway of correlation analyses (Sulaiman and Mohezar 2006;Siegert 2008).The studies that employed correlations, however, largely lost the Managing Global Transitions advantage of prediction, as correlations merely show relationships between variables and not the inter-relationships and importance of each of the variables in a unified predictive formula (such as a regression or discriminate analysis formula) (Steyn 2006).
In the study by Weinstein and Srinivasan (1994, 208) the step-wise regression revealed that solely using academic criteria (junior grades and mathematics, statistics of engineering as majors) for admission to mba, leads to ignorance of other important variables that pose predictor powers.These variables are the applicant's fields of interests, analytical ability, extra-curricular activities (time?) and years of work experience.These authors also placed a lot of emphasis on the gpa and the in-house ability test at the Carnegie-Mellon University called the Administration Test for Graduate Study (atgsb).It is interesting to note that a New York judge ruled against the wide use of the Scholastic Aptitude Tests (sat) for admission to colleges and universities because it violated equal protection rights of the us constitution (Truitt 2002, 2).This brought a new dimension to admission tests and admitting students onto programmes via selection criteria.Tests and admission criteria evolved towards the obligation to (Truitt 2002, 3): • Prove relevance to the performance on the specific programme; and to • Show applicability of the test to the specific programme.
Research by Yang and Lu (2001, 17) showed that although the gmat (when applied as admission test) does possess predictive powers towards successful completion of the mba and graduation, the gpa is an even better predictor for success (especially if the gpa refers to a degree in the economic and management sciences faculty).Clayton and Cate (2004, 236) support Yang and Lu in their views on gmat's predictive powers, and continue to state that the most important academic predictors are the gpa and gmat, and that the gpa should be employed as minimum entry level.When applying these two variables together in a range of discrimant analyses scenarios with other variables, these researchers found that these two variables, when applied together, have a predictive power of 94.2% in predicting if the student will graduate or not (Clayton and Cate 2004, 242).
Following a new avenue in research, Naik and Srinivasan (2004, 144) employed neural networks as a predictive statistical tool in their quest to identify variables with predictive powers on mba success rates.Their re-search substantiated the findings of the previous researchers on the gpa and gmat, and achieved an eventual success rate of 89.13% with their learning artificial intelligence model in predicting if a student will graduate with an mba degree or not.Errors in prediction led to a 19.1% Type I error (Classified as successful, but they were not) and 6.62% Type ii error (Classified as unsuccessful, but they graduated).From the error analysis, it thus showed that the model would disallow 6.62% of students admitted that would have been successful, while 19.1% that would have been admitted would have failed.
Moving away from the sole academic approach, Sulaiman and Mohezar (2006, 331-4) also investigated correlations between demographic and psychographic variables, and the success on an mba programme.Their research found that: • Work experience is a predictor of mba student performance; • Age is a predictor of mba student performance; • Undergraduate gpa is a predictor of mba student performance; • Undergraduate disciple is a predictor of mba student performance; • Gender is not a predictor of mba student performance; and that • Ethnicity is not a predictor of mba student performance.This research substantiated the importance of research by Truitt (2002), Yang and Lu (2001), Clayton and Cate (2004) and Naik and Srinivasan (2004).However, the research also showed that, already in 1994, Weinstein and Srinivasan (1994, 208) correctly commissioned other variables (such as work experience) as academic performance predictors.
Research in 2008 by Siegard, however, reverted back to the pure academic approach, and analysed the individual components of the gmat as predictors for mba success.In this analysis, Siegard found that the verbal and quantitative components accounted for 30% and 36% of the variance, respectively, in the prediction model.This means that although both these scores are important, the quantitative scores are more important than the verbal scores.Siegard (200,226) adds that interviews should be added in the admission process, especially so when marginal applications are reviewed.These interviews should be chaired by experienced personnel such as deans, school directors and senior academic staff.
The research reported on in this article employed a wide array of variables (in excess of 30 per category).These variables were academic as well as demographic and psychographic variables.From the initial analysis Managing Global Transitions (Bisschoff 2005, 306-8), the statistical analysis identified the following variables as ones possessing predictive powers: 1. Complete mba studies in minimum time of extended studies with one year • Undergraduate degree at reputable university • Undergraduate degree at distance learning university From the variables above, it is clear that the predictive powers reside with the academic variables as the literature discussion on the previous studies also indicated.In this study, all the variables relating to being classified as a no-risk applicant (completing the degree in 3 years) are academically orientated.To be classified as a low-risk (completing the degree in 4 years) includes only one non-academic variable that positively counters the negative variables.This is the variable 'work experience;' meaning that good work experience can positively influence the applicant with no degree or low mathematics grades.
The results from the empirical analyses are very satisfactory.Concerning the category Low-to-no risk applicants, the model can be used as predictive tool.It presents a validity of 63.7% for 2004 (p = 0.9) and 75.7% for 2005 (p = 0.9) respectively.The model can also be used for the category and Medium-to-low risk where the model presents a validity of 86.4% for 2004 (p = 0.7) and 91.3% for 2005 (p = 0.7) respectively.In both cases the model breaks down at a probability of 0.9 and 0.8 respectively for the categories Low-to-no risk and Medium-to-low risk.Concerning the category of High-risk applicants, although the model only displays errors in 1.9% of the cases of the total population, it does judge the category of students who complete their mba degree in 4 years too harshly.The model displays an error of 13.7% (p = 0.7) in this discriminant function.

Summary and Conclusion
The results in this research are a function of the data from the pbs, thus an mba degree and a specific business school.It would be unwise to use the model in another application setting without validating the model first.The application of the model in another business school environment is one of later stages of the model development, and is scheduled as a future research project.
The research presents a valuable insight into the validity of the predictive powers of the model.Furthermore, although far from complete, the empirical evaluation clearly indicated that the model shows promise for refinement and development.Although the research does not represent a complete evaluation, it does supply a sound scientific base from which incremental improvements could result.As a result, future research is directed towards the continued statistical validity of the model and its predictive powers embedded in the discriminant functions.
Finally, it is concluded that although the research and the model provides valuable insights and very satisfactory predictive powers, it is not yet ready to implement as predictive tool in the mba selection process at the pbs.The model still needs to be validated further before it determines the fate of any prospective student future.

table 1
Design and Progress of Research Project

table 2
Classification of research population (N = 360)