The accuracy of fiscal projections in South Africa

Forecasting accuracy is important for fiscal policy credibility. Three questions are posed. Firstly, are the forecasts by South Africa’s National Treasury good, compared to those of non-government economists? The paper compares Treasury’s forecasts to non-government projections and to those of other countries and over time. With reference to the mean absolute error and the root mean square error (van der Watt, 2013), it is concluded that nongovernment economists do not necessarily forecast GDP and inflation better than Treasury. Secondly, have the forecasts by National Treasury been good, over time and compared to those of other countries? The forecast error (the final figure minus the budget estimate) is calculated, using data for 2000/01-2010/11. This is most relevant because retrospectively the outcome of fiscal policy is analysed and judged with reference to final figures. National Treasury’s budget forecast errors are found to be significant. Margins of error in forecasting revenue, expenditure and GDP have partially neutralised each other in terms of their impact on the budget balance as a percentage of GDP. Except towards the end of the period, the fiscal balance was better than budgeted. On average and calculated as a percentage of GDP, revenue forecasting inaccuracies made the biggest contribution to inaccurate estimates of the budget balance, but this is largely explained by GDP forecasting inaccuracies. SA fiscal forecasts show a smaller forecast error than that of 14 member countries of the European Union. Thirdly, has the forecasting ability of National Treasury improved over time? A trend line shows higher Treasury forecast errors towards the end of the period and an underestimation bias for GDP and revenue forecasts. A simple example of the dynamics of fiscal politics is presented to demonstrate that a persistent underestimation of revenue could also erode fiscal credibility.


INTRODUCTION
From a macroeconomic perspective, South Africa's fiscal policy has been appraised as credible and transparent. Two important measures of credibility are the accuracy and impartiality of the forecasts used to set the annual national budget. In order to assess these, we compare the South African government's forecasts against those of non-government economists. We also investigate its performance relative to European Union (EU) countries and New Zealand, which surpassed South Africa in 2012 to claim first place in the Open Budget Index 2 . This paper focuses on the period 2001/02 to 2010/11, and asks three questions about the forecasts of the South African National Treasury: (1) How accurate are the macroeconomic forecasts compared to those of nongovernment economists? (2) How accurate are the fiscal forecasts compared to other countries? (3) Has forecast accuracy improved over time?
We find that, on balance, neither private sector economists nor the International Monetary Fund (IMF) and the highly reputable Bureau of Economic Research (BER) outperform the National Treasury regarding forecasting accuracy of macroeconomic variables. Fiscal forecasts by the National Treasury also show a smaller margin of forecast error than those of member countries of the European Union, and New Zealand.
Nevertheless, National Treasury's budget forecast errors are significant when compared to final figures for budget revenue, budget expenditure and GDP, and have increased during the period 2000/01-2010/11. If such errors were to coincide in any particular year, fiscal credibility would be severely tested.
The National Treasury has tended to underestimate revenue and overestimate expenditure with the result that, outside of the global financial crisis period, the fiscal balance turned out better than was budgeted. Although an underestimate of budget revenue may at first glance seem to be a good tactic to combat omnipresent expenditure pressures, it is not a risk-free strategy. A persistent track record of revenue underestimation can change the dynamics of the fiscal game in the sense that spending departments grow in certainty that National Treasury is playing a game of hide-and-seek. The net result might be higher spending pressures than in a scenario in which forecast errors were smaller and less biased.

HOW ACCURATE ARE THE MACROECONOMIC FORECASTS COMPARED TO THOSE OF NON-GOVERNMENT ECONOMISTS?
In an unpublished research note, Van der Wath (2013) compared the accuracy of the macroeconomic 3 forecasts of the BER with those of the National Treasury, the IMF, the consensus forecast of the panel of economists participating in the annual Sake24 Economist of the Year competition 4 , and two mechanical benchmarks (a naïve 5 and a trended forecast). The comparison is between the forecast values in March and October and the first official figures published for the particular variables. These official (first vintage) data are subject to further -and sometimes quite significant -revision, but remain useful as a basis for accuracy comparison. The mean absolute error (MAE) and the root mean square error (RMSE) are used as measures of accuracy. 6 The number of observations depended on the number of future years for which forecasts were done, as well as on the year in which the forecasting commenced. 7 In forecasting the economic growth rate of the nearest year, the BER was on average 0.48 percentage points off target, while the discrepancies of the IMF, National Treasury and Sake24 all came close to 0.6 percentage points. In respect of all the forecasting periods, National Treasury on average outperformed the BER's forecast for economic growth by roughly 2%. Van der Wath (2013: 7) concludes that "all the forecaster's models come remarkably close to each other in terms of accuracy, and that in the first two years all of them perform significantly better than a purely trended or naïve forecast. For periods longer than that they lose their competitive advantage (in terms of pure accuracy)." Regarding inflation, the forecasting errors (MAE) of National Treasury come close to that of the BER, but both institutions missed the spikes in inflation of 2002 and 2008. All the forecasting models appeared to suffer from an inability to anticipate directional changes (Van der Wath, 2013: 12). Although Van der Wath's comparison was between the forecast of the BER and each of the others individually, each with its own sample size, the individual forecasting discrepancies enable the conclusion that neither the group of private sector economists nor the IMF appears to have outperformed the National Treasury regarding forecasting accuracy. 3 The assessment was with reference to GDP growth, inflation, the prime interest rate, the rand-dollar exchange rate, and the gold price. 4 Sake24 Economist of the Year competition tests the forecasting capabilities of a group of well-respected private sector economists. 5 The naïve method leaves the last recorded value unchanged over the forecast period. 6 The formulae are: ∑ , and ∑ , , i=1,…,m. 7 For economic growth forecasts the number of observations was 40 for four-year forecasts (comparing the IMF and the BER), 82 for three-year forecasts (comparing the BER and National Treasury) and 42 for two-year forecasts (comparing the BER with Sake24). In the case of inflation, the corresponding numbers of observations were 38, 69 and 39, respectively.

HOW ACCURATE ARE THE FISCAL FORECASTS COMPARED TO OTHER COUNTRIES?
Next we investigate the accuracy of National Treasury forecasts, in current prices, of GDP (Y), national budget revenue (T 8 ), national budget expenditure (G) and national budget balance (B) in more detail. We identify three sets of figures.  The subscript '-1' (e.g. T -1 ) refers to forecasts for a particular budget year made before the start of the budget year, that is, the data which appear in the Budget Review when the Finance Minister presents the Budget to Parliament in February. We refer to these as budget estimates.  The subscript '0' (e.g. T 0 ) refers to forecasts for the same budget year, as revised during the course of the budget year and which appear in the Budget Review for the next budget year. We refer to these as revised estimates.  The subscript 'f' (e.g. T f ) refers to final official figures. We refer to these as final figures.
Following Beetsma, Bluhm, Giuliodori and Wierts (2001), we distinguish three types of forecasting error. The implementation error (IE) is the revised estimate minus the budget estimate. It measures the extent to which budget design had to be adjusted on account of updated information about the economy and other relevant information (including expenditure pressures) during the fiscal year. The forecast error is the final figure minus the budget estimate. This presents a harsher judgement than Van der Wath's first vintage comparison, but is most relevant because retrospectively the outcome of fiscal policy is analysed and judged with reference to final figures. The revision error is the final figure minus the revised estimate and indicates the error (gap) which remained notwithstanding the budgetary adjustment in response to updated economic and other information. Here we focus on the forecast error. Source: Authors' calculations based on data from National Treasury Budget Reviews and SARB Quarterly Bulletins. 8 We use the conventional symbol T, although tax and non-tax revenue are measured. Budgeted revenue was markedly lower than the final outturn in all the years from 2001/02 to 2007/08 (see Figure 1(a)), a period during which revenue overruns were attributed inter alia to the efforts by the South African Revenue Service to improve tax compliance (SARS, 2012). Actual expenditure consistently exceeded budgeted outlays by a smaller amount than the revenue excess (Figure 1(b)). Hence, the budget balance turned out more favourable than budgeted for seven consecutive years (Figure 1(c)). This trend was only reversed with the recession from 2008Q4 to 2009Q2, which resulted in large revenue under-collections in both 2008/09 and 2009/10.
Revenue forecast errors were responsible for most of the forecast error in the budget balance-to-GDP ratio, namely 72.1% (average) and 285.6% (highest in any year), followed by expenditure 22.1% (average) and 301% (highest), and GDP 5.8% (average) and 84.6% (highest) (cf. Table 1). In many of the years, the errors partly neutralised each other in terms of their impact on the final budget balance-to-GDP ratio; on average government expenditure turned out to be lower than budgeted, while revenue and GDP were higher 9 .
Given the obvious high correlation between GDP (Y) and government revenue (T), to what extent can the revenue forecast error (T f -T -1 ) be attributed to GDP (Y f -Y -1 ) forecast errors? A simple regression analysis generates the following: Although there is a relatively high correlation of 79.3 per cent between the two sets of forecast errors, the regression indicates that a significant part of the forecast error in T cannot be explained by (Y f -Y -1 ), thus suggesting that an important driver of the forecast was on the revenue side itself.
The data in lines (6) and (7) indicate the biggest forecast error (over-and underestimate) in respect of all three variables (comparing budget estimates and final figures) for any year. In some years these errors were very significant, with revenue underestimated by roughly R42 billion in 2005/06, expenditure by R24.9 billion in 2008/09, and nominal GDP by R120.3 billion in 2002/03. The forecast errors of nominal GDP have been particularly lop-sided, with GDP being overestimated only once in the past 11 years, reflecting a clear bias.
Final budget balance estimates have often been revised downwards as nominal GDP estimates come in above expectations. In subsequent years, further upward revisions to GDP have reinforced this trend, producing even smaller final budget balance outcomes.
Despite these examples of significant error, the South African budget balance forecasts are accurate by international standards. Frankel (2011) reviews the accuracy of 33 countries from the first available forecasts up to 2009, and finds that South Africa's deficit forecast errors are in line with the international average. While South Africa tends to overestimate the size of the deficit, the average tendency for all 33 countries is to underestimate the deficit. Looking just at the past decade, the average MAE in respect of the South African national budget was 26 per cent smaller than the average of 1.83 for 14 European Union countries (the so-called EU15 except Luxembourg) 10 .
The New Zealand Treasury has a transparent and credible budget process, and has recently surpassed South Africa in 2012 to claim first place in the Open Budget Index (International Budget Partnership, 2013). As such, it provides a useful benchmark for international accuracy. Using only 1-year ahead forecasts for the period 2000/01 to 2010/11, we find that South Africa's forecasts have been relatively more accurate, with both a smaller MAE and the associated standard deviation. The MAE for New Zealand was 2.3 percentage points (standard deviation of 2.6), against a South African average of 1.4 percentage points (standard deviation of 1). Figure 2 shows that the absolute forecast error in respect of the budget balance was bigger during the middle than the first part of the previous decade, although the error was on the safe side in the sense of overestimating the eventual deficit. During the latter part of the decade, the error was in the other direction and showed higher margins than at the beginning of the decade. Notwithstanding the limitations of the low number of data points, the trend line of the budget balance forecast error is distinctly upwards. A similar trend is observable for revenue and expenditure, even when removing the recession year (2009/10) from the sample. There is thus no evidence of improved forecasting ability, although the upward trend may reflect the inability of most models to predict turning points in the business cycle.

IMPLICATIONS FOR THE CREDIBILITY OF FISCAL POLICY
The analysis raises a key question: how reliable are the budgets presented by the Minister of Finance as indicators of the professed fiscal policy stance? There are significant deviations between the ex ante and ex post budget balance. Over the whole period the absolute average forecast error amounted to 1.35 percentage points of GDP. To put this in perspective: an increase in the budget deficit as a percentage of GDP of 1.35 percentage points from one year to the next would be regarded as a significant fiscal stimulus. To be wrong on average by such a margin over an eleven-year period, does not give a good signal of the reliability of fiscal policy announcements at the time of the budget.
Fortunately, the biggest revenue, expenditure and GDP errors have never coincided. Table 2 presents a few counterfactual calculations. It indicates the difference between the actual deficit-to-GDP ratio (column (2)) and what the final ratio would have been if in 2010/11, each of the following would have materialised: the average historical error (column (3)), all of the biggest historical errors which together would have resulted in the lowest budget deficit ratio (column (4)) and all of the biggest historical errors which together would have resulted in the highest budget deficit ratio (column (5)). This illustrates two things. Firstly, if the forecast error follows the average path, which entails that forecast errors neutralise each other to a large extent, the error in the budget balance as a percentage of GDP is not that big and the credibility risk to fiscal policy would be small. However, should all the maximum errors of the past occur in the same year, the actual deficit would have been either underestimated by 3.8 percentage points (i.e. difference between -1.1 and -4.9 in last row) or overestimated by 3.3 percentage points (i.e. difference between -8.2 and -4.9 in last row). Such extreme errors would damage fiscal credibility. Given the evidence of bias towards underestimating actual revenue, a teasing question is whether such "errors" are more or less serious than the opposite. A common problem in non-credible fiscal authorities is overoptimistic revenue forecasts. South Africa's consistent underestimation of budget revenue might be regarded as an important instrument for managing expenditure expectations. Moreover, such an approach embodies a real possibility that a surplus could be achieved and a buffer be built up to counter future fiscal shocks.
By all accounts this scenario would appear to be risk-free and no threat to fiscal credibility. The snag lies in the dynamic response to a repetition of such an approach. When revenue exceeds the forecast for six out of seven years, as happened in South African between 2001/02 and 2007/08 -another kind of bias, the spending ministries may well get the impression that the fiscal authority is deliberately underestimating revenue. This would change the dynamics of the budgeting game: champions for higher spending would grow in "certainty" that the National Treasury is sounding false alarms about budget constraints. The government may even end up locked into higher expenditure levels than would have been the case under more reliable revenue estimates. Persistently underestimating revenue could therefore also erodes fiscal credibility.

CONCLUSION
The accuracy and impartiality of budget forecasts are two important measures of fiscal credibility. On balance, neither private sector economists, the IMF nor the highly reputable BER appear to outperform the National Treasury regarding forecasting accuracy. Forecasts of the budget balance show a smaller margin of error than those of member countries of the European Union, and of New Zealand.
Nevertheless, National Treasury's budget forecast errors are significant and increased during the period 2000/01-2010/11. Underestimates of revenue together with overestimates of expenditure had the result that, outside of the global financial crisis period, the fiscal balance turned out better than was budgeted.
However, the margin of forecast error in respect of each of revenue, expenditure and GDP had been quite big at times, and if such errors were to coincide in any particular year, fiscal credibility would be severely tested.
Although an underestimate of budget revenue may at first glance seem to be a good tactic to combat the omnipresent expenditure pressures, it is not a risk-free strategy. A persistent track record of revenue underestimation can change the dynamics of the fiscal game in the sense that spending departments grow in perception that National Treasury is playing a game of hide and seek, with the latter losing credibility. The net result might be greater spending pressures than in a scenario in which forecast errors were smaller and less biased.