A proposed best practice model validation framework for banks

Model risk management comprises robust, sensible model development, sound implementation, appropriate use, consistent model validation at an appropriate level of detail and dedicated governance. Each of these broad components is accompanied and characterised by unique risks which, if carefully managed, can significantly reduce model risk. Model risk management is also the process of mitigating the risks of inadequate design, insufficient controls and incorrect model usage. According to McGuire (2007), model risk is ‘defined from a SOX (USA’s Sarbanes-Oxley Act) Section 404 perspective as the exposure arising from management and the board of directors reporting incorrect information derived from inaccurate model outputs’. The South African Reserve Bank (SARB 2015b) uses the definition of model risk as envisaged in paragraph 718(cix) of the revisions to the Basel II market risk framework:


Introduction
Model validation is concerned with mitigating model risk and, as such, is a component of model risk management. Since the objective of this article is to provide a framework for model validation, it is important to distinguish between model risk management and model validation. Below, we define and discuss both these concepts.
Model risk management comprises robust, sensible model development, sound implementation, appropriate use, consistent model validation at an appropriate level of detail and dedicated governance. Each of these broad components is accompanied and characterised by unique risks which, if carefully managed, can significantly reduce model risk. Model risk management is also the process of mitigating the risks of inadequate design, insufficient controls and incorrect model usage. According to McGuire (2007), model risk is 'defined from a SOX (USA's Sarbanes-Oxley Act) Section 404 perspective as the exposure arising from management and the board of directors reporting incorrect information derived from inaccurate model outputs'. The South African Reserve Bank (SARB 2015b) uses the definition of model risk as envisaged in paragraph 718(cix) of the revisions to the Basel II market risk framework: two forms of model risk: the model risk associated with using a possibly incorrect valuation methodology; and the risk associated with using unobservable (and possibly incorrect) calibration parameters in the valuation model. (n.p.) In its Solvency Assessment and Management (SAM) Glossary, the Financial Services Board (FSB) defines model risk as 'The risk that a model is not giving correct output due to a misspecification or a misuse of the model' (FSB 2010). In a broader business and regulatory context, model risk includes the exposure from making poor decisions based on inaccurate model analyses or forecasts and, in either context, can arise from any financial model in active use (McGuire 2007).
Background: With the increasing use of complex quantitative models in applications throughout the financial world, model risk has become a major concern. The credit crisis of [2008][2009] provoked added concern about the use of models in finance. Measuring and managing model risk has subsequently come under scrutiny from regulators, supervisors, banks and other financial institutions. Regulatory guidance indicates that meticulous monitoring of all phases of model development and implementation is required to mitigate this risk. Considerable resources must be mobilised for this purpose. The exercise must embrace model development, assembly, implementation, validation and effective governance.
North American Chief Risk Officers (NACRO) Council (2012) identified model risk as 'the risk that a model is not providing accurate output, being used inappropriately or that the implementation of an appropriate model is flawed' and proposed eight key validation principles. The relevance of model risk in South Africa is highlighted by the Bank Supervision Department of the SARB in its 2015 Annual Report, where it is specifically noted that some local banks need to improve model risk management practices (SARB 2015a). Model validation is a component of model risk management and requires confirmation from independent experts of the conceptual design of the model, as well as the resultant system, input data and associated business process validation. These involve a judgement of the proper design and integration of the underlying technology supporting the model, an appraisal of the accuracy and completeness of the data used by the model and verification that all components of the model produce relevant output (e.g. Maré 2005). Model validation is the set of processes and activities intended to verify that models are performing as expected, in line with their design objectives and business uses (OCC 2011b). The Basel Committee for Banking Supervision's (BCBS) minimum requirements (BCBS 2006) for the internal ratingsbased approach require that institutions have a regular cycle of model validation 'that includes monitoring of model performance and stability; review of model relationships; and testing of model outputs against outcomes'.
In this article, we assess the available literature for validation practices and propose a coherent 'best practice' procedure for model validation. Validation should not be thought of as a purely mathematical exercise performed by quantitative specialists. It encompasses any activity that assesses how effectively a model is operating. Validation procedures focus not only on confirming the appropriateness of model theory and accuracy of program code, but also test the integrity of model inputs, outputs and reporting (FDIC 2005).
The remainder of this article is structured as follows: The next section provides a brief literature overview of model risk from a validation perspective. 'Overview of the proposed model validation framework' section establishes an overview of the proposed framework for model validation and, in 'model validation framework discussion' section; this framework is discussed in more detail. Guidelines for the development of scorecard tools for incorporation in the proposed best practice model validation framework are presented in 'model validation scorecards' section. Some concluding remarks are made in 'conclusions and recommendations' section. Examples illustrating the importance of proper model validation are given in Appendix 1 and scorecards for the evaluation of the main components of the validation framework are provided in Appendix 2.

Brief overview of model risk from a validation perspective
Banks and financial institutions place significant reliance on quantitative analysis and mathematical models to assist with financial decision-making (OCC 2011a). Quantitative models are employed for a variety of purposes including exposure calculations, instrument and position valuation, risk measurement and management, determining regulatory capital adequacy, the installation of compliance measures, the application of stress and scenario testing, credit management (calculating probability and severity of credit default events) and macroeconomic forecasting (Panko & Ordway 2005).
Markets in which banks operate have altered and expanded in recent years through copious innovation, financial product proliferation and a rapidly changing 1 regulatory environment (Deloitte 2010). In turn, banks and other financial institutions have adapted by producing data-driven, quantitative decision-making models to risk-manage complex products with increasing ambitious scope, such as enterprise-wide risk measurement (OCC 2011b).
Bank models are similar to engineering or physics models in the sense that they are quantitative approaches which apply statistical and mathematical techniques and assumptions to convert input information -which frequently contains distributional information -into outputs. By design, models are simplified representations of the actual associations between observed characteristics. This intentional simplification is necessary because the real world is complex, but it also helps focus attention on specific, significant relational aspects to be interrogated by the model (Elices 2012). The precision, accuracy, discriminatory power and repeatability of the model's output determine the quality of the model, although different metrics of quality may be relevant under different circumstances. Forecasting future values requires precision and accuracy, for example, rank ordering of risk may require greater discriminatory power (Morini 2011). Understanding the capabilities and limitations of models is of considerable importance and is often directly related to the simplifications and assumptions used in the model's design (RMA 2009).
Input data may be economic, financial or statistical depending on the problem to be solved and the nature of the model employed. Inputs may also be partially or entirely qualitative or based on expert judgement [e.g. the model by Black and Litterman (1992) and scenario assessment in operational risk by de Jongh et al. (2015)], but in all cases, model output is quantitative and subject to interpretation (OCC 2011b). Decisions based upon incorrect or misleading model outputs may result in potentially adverse consequences through financial losses, inferior business decisions and ultimately reputation damage: These developments stem from model risk, which arises because of two principal reasons (both of which may generate invalid outputs): • fundamental modelling errors (such as incorrect or inaccurate underlying input assumptions and/or flawed model assembly and construction) and • inappropriate model application (even sound models which generate accurate outputs may exhibit high model risk if they are misapplied, e.g. if they are used outside the environment for which they were designed).
Model risk managers, therefore, need to take account of the model paradigm as well as the correctness of the implementation of any algorithms and methodologies to solve the problem as well as the inputs used and results generated. NACRO Council (2012) (Rajalingham 2005). As far as the model paradigm is concerned, the model needs to be evaluated in terms of its applicability to the problem being solved, and the associated set of assumptions of the model needs to be verified in terms of its validity in the particular context. Example 1 in Appendix 1 gives an illustration of the inappropriateness of the assumptions of the well-known Black-Scholes option pricing model in a South African context. Clearly, all listed assumptions may be questioned, which will shed doubt on the blind application of the model in a pricing context. Some models have been implemented using spreadsheets (Whitelaw-Jones 2015). Spreadsheet use in institutions range from simple summation and discounting to complex pricing models and stochastic simulations. Madahar, Cleary and Ball (2008) questioned whether every spreadsheet should be treated as a model, requiring the same rigorous testing and validating. Spreadsheet macros require coding and may be used to perform highly complex calculations, but they may also be used to simply copy outputs from one location to another (Galletta et al. 1993;PWC 2004). Requiring that all macro-embedded spreadsheets be subject to the same validation standards can be onerous (Pace 2008). Example 2 in Appendix 1 highlights some examples of formulas in Excel that provide incorrect answers. In addition, the European Spreadsheet Interest Group (EUSIG) maintains a database of such errors. EUSIG (2016) and Gandel (2013) provide examples of high impact Excel errors that occurred as a result of inadequate model validation. Therefore, the validation of the code is of extreme importance, and should be validated using not only ordinary but also stressed inputs.
Model risk increases with model complexity, input assumption uncertainties, the breadth and depth of the model's implementation and use. The higher the model risk, the higher the potential impact of malfunction. Pace (2008) identified challenges associated with effective model risk management programmes. Assigning the correct model definition to models is important, but challenging, because model types (e.g. stochastic, statistical, simulation and analytical) and model deployment methods (ranging from simple spreadsheets to complex, software-interlinked programmes) can sometimes straddle boundaries and defy easy categorisation (PWC 2004 Haugh (2010) presents practical applications of model risk and emphasises the importance of understanding a models' physical dynamics and properties. Example 3 in Appendix 1 illustrates a strong correlation between two variables that clearly does not share any causal relationship. Incorrect inclusion of such variables in models can lead to nonsensical conclusions and recommendations. The dangers of calibrating pricing models with one type of security and then pricing other types of securities using the same model can be disastrous. Model transparency is important and substantial risks were found to be associated with models used to determine hedge ratios. These conclusions, although specifically focussed on structured products (collateralised debt obligations) and on equity and credit derivative pricing models, could be equally applied to all models (Haugh 2010;PWC 2004). Example 4 in Appendix 1 gives some risk-related loss examples. These examples clearly illustrate that even simple calculation errors and incorrect models and assumptions can result in devastating losses. Actively managing model risk is important, but also costly, because not only does the validation of models requires expensive and scarce resources, but also the true cost of model risk management is much broader than this. The cost of robust model risk management processes includes having to maintain skilled and experienced model developers, model validators, model auditors and operational risk managers, as well as senior management time at governance meetings, opportunity costs (because of delays in time-to-market because of first having to complete the model risk management process before a new model supporting a new product can be deployed) and IT development cost of the model deployment.
Although model risk cannot be entirely eliminated, proficient modelling by competent practitioners together with rigorous validation can reduce model risk considerably. Careful monitoring of model performance under various conditions and limiting model use can further reduce risk, but frequent revision of assumptions and recalibration of input parameters using information from supplementary sources are also important activities (RMA 2009). Deloitte (2010) addressed internal model approval under Solvency II. Model validation was identified as a key activity in model management to ensure models remain 'relevant', that is, they function as originally intended both at implementation and over time.
Ongoing monitoring to determine models' sensitivity to parameter changes and assumption revisions helps to reduce model risk. Deloitte's (2010) proposed validation policy includes a review of models' purpose and scope (including data, methodology, assumptions employed, expert judgement used, documentation and the use test), an examination of all tools used (including any mathematical techniques) and an assessment of the frequency of the validation process. Independent governance of the validation results, robust documentation and a model change policy (in which all changes to the model are carefully documented and details of changes are communicated to all affected staff) all contribute to effective model management (PWC 2004;Rajalingham 2005).

Overview of the proposed model validation framework
Despite the broad market requirement for a coherent model risk management strategy and associated model validation guidelines, the literature is not replete with examples. The Basel Accord and some regulatory authorities have attempted to establish this but, according to our knowledge, no definite set of global standards exists. However, although the literature places varying emphasis on different aspects of model governance, there are encouraging signs of cohesion and broad, common themes emerging. One of these common themes is the role of the three-lines of defence governance model, as developed by the Institute of Internal Auditors in 2013 (IIA 2013). In the context of model risk management, the first line of defence would be model development, the second line would be model validation and third line would be internal audit. A fourth line of defence is also suggested by the Financial Stability Institute (FSI) as external audit and supervisors (FSI 2015). In addition, operational risk management has a second line duty in respect of model development and model validation.
Model validation embraces two generic views of the modelling landscape that should be considered to cover all possible validation elements (refer to Figure 1), namely: • the modelling life cycle, comprising three stages (development, implementation and operation) as well as the elements that form part of each stage and • the modelling process elements namely input, output and process (Rajalingham 2005).
As illustrated in Figure 1, the model life cycle starts with the model development phase. This phase commences with the formulation of the problem and model, followed by the specification of the user requirements. This phase is usually followed by a prototyping phase, in which especially the more risky or uncertain modelling aspects are researched and tested. Based on the results of this phase, the formulation phase may be re-assessed and various iterations may be possible before alignment with user requirements is achieved. Once the modelling concept is clear, the development of the model can commence. In this phase, amongst others, the model outputs are defined and the inputs required clearly specified. The testing phase involves testing of functional components of the model. As soon as the model is completely assembled integration testing can start on the completed model. This could involve out of sample and backtesting in order to ensure that the model performs well for the purpose it was designed for. As soon as testing is completed, the model is reviewed internally and externally by independent experts and then accepted if it adequately meets all validation criteria (see 'model validation framework discussion' section for more details on the validation framework proposed as well as the validation criteria). After the model has been accepted, it should be implemented. Depending on the complexity of the model and speed requirements it might entail the recoding of a model in computationally efficient programming language and using appropriate database query languages. This could entail a complete redesign and specification of the model in IT terms, recoding, User Acceptance Testing (UAT), review and acceptance. Once the model is implemented and running the model is put into operation. In this phase, the model should be validated in terms of its performance against the original design specifications and tested on a regular basis. Ownership of the model should be identified through the validation process, as should the appropriateness of the model governance. Efforts to validate models should be proportional to model output materiality and complexity, and should involve validation of model components and relevant documentation as well as third-party validation where possible (Elices 2012).
A graphical representation of a proposed framework is presented in Figure 2 below, by providing an alternative view of the model life cycle. The framework presented in Figure 2 consists of the following components.

Model governance
This includes model governance (on the left of Figure 2) and related management activities and will be discussed in more detail in the 'model validation governance' section and will be extended into a scorecard in the 'model validation governance scorecard' section.

Model validation policy
Note that seven main elements are covered in this (on the right of Figure 2) namely, the scope, an independent review, the roles and responsibilities, relevant model documentation, proof of ongoing validation, details of performance standards and remediation plans and audit oversight. This will be discussed in more detail in the 'model validation policy' section and will be extended into a scorecard in 'model validation policy scorecard' section.

The 'validation process'
This consists of three distinct elements (in the middle of Figure 2) namely: • conceptual soundness and developmental evidence • process verification and ongoing monitoring and • outcomes analysis.
This 'validation process', with its three elements, will be discussed in more detail in the 'model validation process' section and will be extended into a scorecard in 'model validation process scorecard' section.
In 'model validation framework' section, each of the abovementioned components are discussed in more detail.

Model validation framework discussion
As stated at the end of the previous section, the model validation framework is discussed in more detail in the following subsections: • model validation governance • model validation policy • model validation process.
This subsection contains a discussion of conceptual soundness and developmental evidence, process verification and ongoing monitoring, and outcomes analysis.

Model validation governance
As mentioned in 'overview of the proposed model validation framework' section, the FSI outlines that regulated financial institutions require a four lines of defence model (FSI 2015) to effectively manage the risk it is exposed to. Model validation is seen as the second line of defence in the context of model risk management. Under model validation governance, the adequacy of the governance structure should be evaluated. In the model validation policy, clear roles and responsibility should be assigned to role players and committees (OCC 2011b). This includes identifying who amongst the stakeholders in the model risk management process should perform, for example, benchmarking, independent review and monitoring.
Model governance (which should involve the board of directors, senior management and line-of-business managers) requires that the organisation's governance policies, procedures and processes support its controls and provide the requisite oversight to manage the model (Glowacki 2012

Model validation policy
Seven main aspects are covered in the model validation policy namely, the scope, independent review, roles and responsibilities, relevant model documentation, proof of continuing validation, details of performance standards and remediation plans and audit oversight (Rajalingham 2005).

Scope
Institutions should have a written, enterprise-wide policy for validating model risk (RMA 2009). The rigor and sophistication of validation should be commensurate with the institution's overall model use, the complexity and materiality of its models, and the size and complexity of the organisation's operations (OCC 2011b).

Independent review
The validation process should be subject to independent review (OCC 2011b; RMA 2009) and should be organisationally separate from the activities it is assigned to monitor. The head of the validation function should be subordinated to a person who has no responsibility for managing the activities that are being monitored. Remuneration of validation function staff should not be linked to the performance of the activities that the validation function is assigned to monitor (CEBS 2008).

Roles and responsibilities
The validation policy should be owned by the chief risk officer (RMA 2009) and should identify roles and assign responsibilities based upon staff expertise, authority, reporting lines and continuity. According to OCC (2011b) and Green (2012), model validators should have appropriate incentives, competence and influence (e.g. authority to challenge developers and users or to restrict model use).
Model owners should ensure that models employed have undergone appropriate validation and approval processes and promptly identify new or altered models by providing all necessary information for validation activities (OCC 2011b).

Performance standards and remediation plans
Backtesting, benchmarking and stress testing should be conducted and results assessed. Model accuracy and precision should be evaluated and results should be compared with those provided by other models. Model output sensitivity to inputs, model assumptions and stress testing should also be considered (Green 2012;OCC 2011b). If significant model risk is found remediation efforts should be prioritised. Ongoing monitoring of areas of concern to ensure continued success is also required (McGuire 2007).

Audit oversight
Internal audit should verify that no models enter production 2 without formal approval by the validation unit and should be responsible for ensuring that model validation units adhere to the formal validation policy (Pace 2008). Records of validation -to test whether validations are performed in a timely manner -should also exist and the objectivity, competence and organisational standing of the key validation participants should be evaluated (OCC 2011b).

Model validation process
As per Figure 2, the model validation process comprises three distinct elements, namely conceptual soundness and developmental evidence, process verification and ongoing monitoring and lastly, outcome analysis.

Conceptual soundness and developmental evidence
This first main element of the model validation process can be subdivided into nine sub-elements, which will be briefly discussed in Table 1.

Process verification and ongoing monitoring
The second element of the model validation process can be split into two distinct stages, namely monitoring and test and evaluation as discussed in Table 2.

Outcome analysis
The third and last element of the model validation process will be discussed under outputs and backtesting in Table 3.

Model validation scorecards
Model validation scorecards comprise three components (in line with the elements introduced in the previous section): 2.To clarify this point further, the reader should note that internal audit is typically not in a position to police the day-to-day moving of models into production since they only do periodic audits of business areas. Internal audit should therefore pick up that models entered production without following due process, however only sometime after the event. All the lines-of-defence (refer to the four-line of defence in 'Overview of the proposed model validation framework' section) including operational risk managers need to play a role to ensure that models follow the correct process before entering production.
the model validation governance scorecard, the model validation policy scorecard and the model validation process scorecard. These tools may be used to ascertain whether the proposed best practice model validation framework has been adequately assembled and implemented. All three scorecards use numerical scores ranking from 1 (no evidence) to 4 (fully evident). A four-grade scale, in line with that used by the Regulatory Consistency Assessment Programme (RCAP) (BCBS 2016) was specifically chosen to avoid the midpoint. Although most inputs in any kind of scorecard are subjective, there is a danger in using a 3-point or 5-point scale, as many respondents is likely to choose the midpoint or average. This might result in a failure to identify specific weaknesses. 3 A typical use of these scores in a management information environment would result in a colour-coded dashboard, for example, associating a score of 1 with Red, 2 with Orange, 3 with Yellow and 4 with Green. This would be a powerful tool in highlighting validation areas in need of urgent attention (i.e. 'Red'). Tracking individual scores, as well as the distribution of these scores, over time can give an indication of the model validation framework's maturity within an institution.
3.Backtesting is not always possible, for example with capital models where the unexpected loss is modelled. Here the modelled result typically represents a 1-in-1000 year annual loss and therefore backtesting is not practical due to needing several thousand years of data perform a credible backtest. In these cases benchmarking replaces backtesting (see Table 1 for more information on benchmarking).
Note that no weights were added to the scorecards, as the purpose is not to combine the elements together in an aggregate score. Instead, the ultimate goal for each institution should be to achieve 'compliance' in each one of the suggested areas, that is, a score of 4 (fully evident) for each sub-element of each scorecard, similar to the grading methodology of the RCAP programme.
The three different scorecards can be applied at different levels. The policy scorecard could, for instance, be applied at the highest level for which an applicable validation policy exist, be that at enterprise, risk type or business unit level. The process scorecard, on the contrary, should be applied at model level.

Model validation governance scorecard
This generic validation governance scorecard (Table 1-A2) provides a tool that may be used to determine whether the firm's validation governance is in place (according to the model validation framework discussed in 'model validation governance' section). The main elements of this scorecard are as follows: • Clear roles and responsibility assigned to role players and committees.

Sub-element Description
Methods/theory/approaches The quality of the empirical evidence supporting the methods used and variables selected for the model should be assessed (OCC 2011b). This should include a comparison with alternative theories and approaches by means of an independent review (CEBS 2008;FDIC 2005;OCC 2011b).

Assumptions/variables/ sensitivity
Key assumptions and the choice of variables should be assessed, with analysis of their impact on model outputs and particular focus on any potential limitations. This should include a sensitivity analysis to check the impact of small changes in inputs and parameter values on model outputs and to make sure they fall within an expected range (FDIC 2005;OCC 2011b).

Data
Input data used to build models should be assessed to ensure they are reasonably representative of market conditions. Data inputs should be representative of normal and stressed market conditions.

Mathematical calculations/ algorithms
Key assumptions, choice of variables and a review of quantitative techniques employed should be assessed, with an analysis of their impact on model outputs and a particular focus on any potential limitations (Madahar et al. 2008;Maré 2005). Focus should be placed on independent methodologies to ascertain the accuracy of algorithms and calculations.
Code generation Code should be rigorously tested by independent construction of an identical model or by testing against a well validated benchmark model. For complex models, technical proofreading of code is advised (OCC 2011b; Pace 2008).

Scenarios
Specific guidelines must be set for scenario generation and all scenarios should be appropriately vetted. An independent review should address the completeness of risk factors included as well as the effect of extraordinary changes in these risk factors (FDIC 2005;RMA 2009).
Outputs Support for the reasonableness and validity of model results should be provided (FDIC 2005). Outputs should be verified over a range of inputsresults of derived quantities such as hedging ratios should be checked for reasonability.

Benchmarking
Benchmarking includes the comparison of the model's inputs and outputs to estimates from alternative internal or external models. This can range from industry surveys, Basel Quantitative impact studies or even third-party vendor models. Vendors should provide documentation of their validation methods and results and institutions should ensure that there are appropriate processes in place for selecting and retaining vendor models (OCC 2011b;RMA 2009

Monitoring
Ongoing monitoring should confirm that the model is appropriately implemented and is being used and is performed as intended. Each model extension (beyond original scope) should be validated and placed under configuration control (OCC 2011b; RMA 2009).

Test and evaluation
A program for ongoing test and evaluation of model performance should be designed, including checks that all model components are functioning as designed (Madahar et al. 2008;Maré 2005) within predetermined tolerance levels. Only approved parties should approve changes and all changes should be logged and audited (Morini 2011). Analysis of internal and external information integrity should be performed regularly (OCC 2011b; Pace 2008).

Sub-element Description
Outputs A comparison of model outputs against corresponding actual outputs should be conducted regularly, including the assessment of forecast accuracy, appropriateness of statistical tests, expert judgement of outputs produced and confirmation that outputs make business sense (FDIC 2005 • Adequate oversight and participation by internal audit.
• Validation function independent in terms of remuneration.
• Defined assignment of authority for approval.
• Adequate board and senior management involvement.
The rest of the validation governance elements are addressed in the validation policy scorecard (e.g. roles and responsibilities, independent review and audit oversight).

Model validation policy scorecard
The generic validation policy scorecard may be used as a tool to check to what extent the firm's validation policy is in place (according to the model validation framework discussed in 'model validation policy' section). This scorecard comprises seven elements, indicated in Table 2-A2 and can be summarised as follows: Scope: • Separate, enterprise-wide validation policy exists.
• Validation policy provides guidelines for input validation, processing and reporting. • Vendor model validation included in policy scope.
Independent review: • Policy provides for models to be independently evaluated prior to implementation. • Independent review is performed by suitably skilled experts.
Roles and responsibilities: • Problems identified during the independent review are reported on. • Appropriate responsibility to act on such reports is assigned. • Necessary actions are scheduled and managed properly.

Model documentation:
• Documentation is reviewed in terms of completeness, transparency and scope. • A completely new model can independently be reproduced from documentation.
Ongoing validation: • The existence of a monitoring plan for implementation is ascertained. • A program for ongoing testing and evaluation of model performance has been designed.
Performance standards and remediation plans: • Documentation specifies tolerances/thresholds for implemented model performance. • Procedures for the management and control of remediation activities are in place.
Audit oversight: • Verification that no models enter production without validation unit approval.
• Evaluation of objectivity/competence/organisational standing of key validation participants. • Verification that validation process is carried out according to policy in a timely manner.

Model validation process scorecard
The generic validation process scorecard may be used as a tool to check if the validation process has been correctly established (according to the model validation framework discussed in 'model validation policy' section). Validation process comprises conceptual soundness and developmental evidence, process verification and ongoing monitoring, and outcome analysis. The validation process scorecard comprises seven elements as indicated in Table 3-A2. Note that this is a generic scorecard and will change per product, per institution and also whether the model is in the development, implementation or monitoring phase.
The main elements of this scorecard are: Paradigm: • • Was it ascertained that assumptions are clearly formulated? • Was the appropriateness and completeness of assumptions checked? • Was it checked that all variables employed have been clearly defined and listed? • Have the causal relationships between variables been documented? • Have input data been determined and assessed in terms of reasonableness, validity and understanding? • Has it been ascertained that outputs are clearly defined? • Has the design been evaluated in terms of model parsimony? • Has model builder benchmarked design against existing best practice models? • Was design independently benchmarked against existing best practice models? • Are special cases dealt with appropriately? (e.g. terminal conditions or products with path-dependent pay-off) Data/variables: • Have input data been checked to gauge reliability/ suitability/validity/completeness?
• Have data that involve subjective assessment of expert opinion been appropriately incorporated? • Was the procedure for the collation of expert opinion scrutinised? • Has expert opinion been validated in terms of logical considerations? • Has expert selection process been assessed as sound?
• Were data verified that they are representative of relevant (general and stressed) market conditions? • Was it verified that data are representative of the company's portfolio? • Have inadequate or missing data been re-assessed and reviewed for model feasibility? Algorithms/code: • Was the algorithms/code checked against the model formulation and underlying theory? • Were key assumptions and variables analysed with respect to their impact on model outputs? • Was an independent construction of an identical model undertaken?
• Was the code rigorously tested against a benchmark model? • Was technical proofreading of the code performed? Outputs: • Was model output benchmarked against best practice models (e.g. against a vendor model using the same input data set)?

Conclusions and recommendations
Institutions should classify, design, implement, validate and govern their models robustly on an ongoing basis if they want to effectively minimise model risk. Institutions that fail to implement a regular, consistent model risk management framework risk penalties from regulatory authorities and reputation risk in the contemporary era of strict model risk management standards.
This research provided a comprehensive literature study, which provided a background to the complexities of effective model management and focussed on model validation as a component of model risk management. A best practice model validation framework for institutions has been proposed.
The proposed best practice model validation framework is designed to assist firms in the construction of an effective, robust and fully compliant model validation programme and comprises three principal elements: model validation governance, policy and process.
A set of scorecards -detailing the principles of model validation governance, model validation policies and model validation processes -was proposed. These scorecards may be used as tools to determine whether the proposed best practice model validation framework has been established and is effective. This includes the provision of detailed supporting documentation to substantiate assertions that models are aligned to business and regulatory requirements to supervisory authorities.

APPENDIX 1 Model risk examples Example 1: Real world calibration assumptions
Suppose a contingent claim on a South African equity within the Black-Scholes paradigm was required - Table 1-A1 highlights the classical theoretical model assumptions used and details the practical reality 'calibrated' to the South African market environment. Model validators typically need to assess the gap between theory and practise and understand the model misspecification. Typically, a certain amount of capital could be set aside to cover the gap. The misspecification could also mean that a product is entirely unsuitable for a specific institution or lead to strict limits imposed on its use. Significant losses can be incurred as a result of using the incorrect paradigm (see Cont 2006).

Example 2: Spreadsheet-based implementation examples
'Let's not kid ourselves: the most widely used piece of software in Statistics is Excel' (Ripley 2002).
Spreadsheet applications abound in the modern financial services industry. Users generally understand the need for careful practices when using spreadsheets to ensure a control environment and avoid basic mistakes or logical errors. Users might not always suspect that the functionality offered by a spreadsheet program has not been thoroughly tested. We highlight some examples from the literature. Consider the simple problem of calculating the standard deviation of three numbers, say [m, m, + 1, m + 2]. The correct answer is trivially equal to m + 10 8 ; Higham (2013), however, showed that Google Sheets produces a 0 answer for m = 10 8 . Sawitzki (1994) reports a similar problem for EXCEL 4.0. McCullough and Wilson (1999)  It is clear from the above that one should view spreadsheetbased analyses with some caution -In particular when the spreadsheet is used as an independent control for validation purposes.
Example 3: Illustration of the necessity of confirming business sense Figure 1-A1 demonstrates the need to explore the true causal relationship between variables. The Wood index is plotted against the Naspers index. The variables in the example exhibit significant positive correlation (+0.92), however, their economic relationship is entirely spurious. The wrong conclusion might be that Naspers are highly correlated to the Wood index because of the relationship of Naspers to print media (i.e. paper), and paper again comes from wood. However, the real constitutes of Naspers has very little to do with the Wood index. Naspers is a global platform operator with principal operations in: • Internet services, especially e-commerce (i.e. classifieds, online retail, marketplaces, online comparison shopping, payments and online services) • pay television (direct-to-home satellite services, digital terrestrial television services and online services) • print media.

Example 4: Model error examples and associated loss impacts
A few different model risk related loss examples will be explained next and summarised in Table 2-A1 below. A computing error at the Fidelity's Magellan fund resulted in a net capital loss of $1.3 billion (Godfrey 1995). In March of 1997, NatWest Markets, an investment banking subsidiary of National Westminster Bank, announced a loss of £90 million because of mispriced sterling interest rate options (Simons 1997). Real Africa Durolink, a smaller bank in South Africa, but major player in the equity derivatives market, failed within days of the introduction of the skew, as they were completely unprepared for the dramatic The so-called local volatility is assumed constant over the life of the option contract. Equity market volatility is stochastic with mean reversion.
Volatility is assumed constant over time and independent of the required strike of the option.
A volatility surface exists. Volatility is term-dependent and strike-dependent.
No-arbitrage opportunities. Arbitrage opportunities typically exist.
Derivative contracts are hedged by virtue of continuous trading (rebalancing). Hedging at daily/weekly close. †, As correctly noted by a referee, a lot of these assumptions would be equally invalid for any equity market.  impact the new methodology would have on their margin requirements (West 2004). The number three on a list of the eight worst spreadsheet blunders are listed as the financial institution, Fannie Mae that discovers a $1.3 billion 'honest' mistake (Wailgym 2007

APPENDIX 2 Model validation scorecards
The detailed scorecards that are described in 'model validation governance', 'model validation policy' and 'model validation process' sections are presented Tables 1-A2; 2-A2 and 3-A2. To what extent does the validation policy provide guidelines for input validation, processing and reporting?
To what extent is vendor model validation included in policy scope?

Independent review
To what extent does the policy provide for models to be independently evaluated prior to implementation?
To what extent is the independent review performed by suitably skilled experts?

Roles and responsibilities
To what extent are problems identified during the independent review reported on?
To what extent is appropriate responsibility to act on this report assigned?
To what extent are necessary actions scheduled and managed?

Model documentation
To what extent is documentation reviewed in terms of completeness, transparency and scope?
To what extent can a complete new model be reproduced independently from documentation?

Ongoing validation
To what extent has the existence of a monitoring plan for implementation been ascertained?
To what extent has a program for ongoing testing and evaluation of model performance been designed?

Performance standards and remediation plans
To what extent does documentation specify tolerances/thresholds for proper, implemented model performance?
To what extent are procedures for the management and control of remediation activities in place?

Audit oversight
To what extent has it been verified that no models entered production without validation unit approval?
To what extent has it been evaluated that the objectivity/competence/organisational standing of key validation participants is adequate?
To what extent has it been verified that the validation process was carried out according to policy in a timely manner? To what extent were clear roles and responsibility assigned to role players and committees?
To what extent does internal audit have appropriate oversight and participation?
To what extent is the validation function independent in terms of remuneration?
To what extent is the assignment of model approval authority defined?
To what extent is the board and senior management involved? To what extent was the conceptual soundness of paradigm checked?
To what extent was the review performed by suitably skilled experts?

Methods/theory
To what extent is the underlying model theory consistent with published research and sound industry practice?
To what extent were research publications considered of appropriate quality/standing?
To what extent was the methodology benchmarked against appropriate industry practice?
To what extent are approximations made within agreed tolerance levels?

Design
To what extent was it ascertained that assumptions are clearly formulated?
To what extent was the appropriateness and the completeness of assumptions checked?
To what extent was it checked that all variables employed have been clearly defined and listed?
To what extent have the causal relationships between variables been noted?
To what extent have input data been assessed in terms of reasonableness, validity and understanding?
To what extent has it been ascertained that outputs are clearly defined?
To what extent has the design been evaluated in terms of over-complexity/over-simplification?
To what extent has the model builder benchmarked the design against existing best practice models?
To what extent was the design independently benchmarked against existing best practice models?
To what extent have special cases been dealt with appropriately? (e.g. terminal conditions or products with path-dependent pay-off)

Data/variables
To what extent have input data been checked to gauge reliability/suitability/validity/completeness?
To what extent has it been checked that data involving subjective assessment of expert opinion been appropriately incorporated?
To what extent was the procedure for the collation of expert opinion scrutinised?
To what extent has expert opinion been validated in terms of logical considerations?
To what extent has the expert selection process been assessed as sound?
To what extent was it verified that data are representative of relevant (general and stressed) market conditions?
To what extent was it verified that data are representative of the company's portfolio?
To what extent have inadequate or missing data been re-assessed and reviewed for model feasibility?

Algorithms/code
To what extent was the algorithms/code checked against the model formulation and underlying theory?
To what extent were key assumptions and variables analysed with respect to their impact on model outputs?
To what extent was an independent construction of an identical model undertaken?
To what extent was the code rigorously tested against a benchmark model?
To what extent was technical proofreading of the code performed?

Outputs
To what extent was model output benchmarked against best practice models (e.g. against a vendor model using the same input data set)?
To what extent was the reasonableness and validity of model outputs assessed?
To what extent has a comparison of model outputs against actual realisations been performed?
To what extent has a range of outputs been examined vs. a range of inputs (e.g. are solutions continuous or jagged? What is the behaviour of hedging quantities and/or derived quantities over the same range?) To what extent are all results repeatable? (e.g. Monte Carlo simulations)

Monitoring
To what extent has the model been monitored for appropriate implementation and use?
To what extent has the model been monitored to check whether it is performing as intended?