Economic evaluation of public programmes : lessons from the Expanded Public Works Programmes in South Africa

This review aims at assessing the economic evaluation of public programmes such as the expanded public works programmes (EPWP) in South Africa. The South African government earmarked the EPWP for departments and municipalities to implement projects that are meaningful for economic transformation and inclusive growth. This study argues that economic evaluation of public programmes must consider the interplay of complex decisions making on resource allocations and take into account consequences thereafter in a systematic way. This review paper adopted a qualitative document analysis, where data is drawn from research reports on programme evaluation, policy documents, EPWP evaluation reports, books and articles drawn from accredited journals. Key findings from this study draw attention to unfulfilled great expectations to sustain job creation in an emerging economy in South Africa. Results also revealed that although the M&amp;E design was suitable for the evaluation, it was not compared to any other alternative cost-effective measurement strategy to assess the economic value of the EPWP in South African public service. Based on the lessons from EPWP, this study recommends an integrative approach to evaluate job creation programmes in order to settle on the economic value of EPWP.


INTRODUCTION
This paper reviews the economic evaluation of the Expanded Public Works Programmes in South Africa.The intention of this paper is to provide recommendations for improvement in the adoption and application of a framework for the evaluation public programs.Globally and nationally, the adoption of neo-liberal policies is based on the government expectations to induce the economy that will be favorable for job creation.The EPWP was implemented in South Africa to create jobs that will enhance a growth in GDP of the broader society.The economic evaluation of public program in South Africa is still at the incipient stage (Abrahams, 2015).The economic evaluation on the efficiency and effectiveness of public programs can justify whether the government can sustain the program or abort it (Levin et al., 2011).The EPWP was implemented in various departments and local municipalities to train the youth to be more prepared for better job opportunities in a short term.and spending on goods and budget capacity (Public Works, 2014).While there are existing models to evaluate the economic value of the EPWP, public works adopted the monitoring and evaluation framework to assess the effectiveness and efficiency of government performance.The application of M&E is informed by the transversal approach used by the Presidency of South Africa.However, the Presidency adopted a systematic national evaluation framework that underpins planning, budgeting, implementation management and accountability reporting.Various departments are grappling with the application of tools and techniques for monitoring and evaluating public programs due to lack of capacity, on their side, to conduct the evaluation process.
The South African governing party responded to economic problems by adopting neoliberal policies like Growth Employment and Redistribution (GEAR) and other policy initiatives but yielded less progress in solving the economic problems in South Africa.The implementation of EPWP emerged as a response towards addressing job scarcity along those lines of neo-liberal policies.EPWP was set to be implemented in various spheres of government, by developing skills and capacity of the youth to make them to be employable.This review paper starts with the conceptual framework of Monitoring and Evaluation, literature review, discussions of the challenges of evaluating the EPWP in South Africa, conclusions and recommendations.Evaluation may be conducted from various perspectives, for example an ex ante facto evaluation may seek to ascertain the impact of a program before it is implemented, its process may investigate aspects of the program operations, while they are in place.While an ex post facto evaluation focuses on programs after they have been implemented (Shafritz et al., 2011).The monitoring of govern-ment performance focuses primarily on measuring the connection of inputs to costs, the value of cost reduction activities in a system, adapting techniques from the broader discipline and observing outcomes.

The systems theory and evaluation
The theoretical foundation of this article is informed by the systems theory which is favored for its analytical ability to review programs effectiveness.Systems theory is based on the premise of holism, which implies that a system is made up of interdependent components or parts that cannot be acted upon separately.In the context of this study, a program constitutes part of a government agency that relies on both its internal and external environments which is constantly impacted by these factors during its implementation.Programs are directed by policy determinations relating to the way in which they should be implemented.Anderson (2006) maintains that a systematic evaluation is ap-propriate for evaluating policy to determine causeand-effect relationships and rigorously measure the results of the policy in question.However, a program may be conceived of as a system in which each element is dependent on the other.
In the context of this study, a system is composed of inputs, conversion and outputs (Figure 1).
Proponents of the systems theory accept its premise that is grounded on a public policy process paradigm.Ille et al. (2012) agree that systems theory departs from the general understanding of a political system in terms of which elements such as inputs are converted into outputs.Cloete, De Coning, Wissink, and Rabie (2018) assert that feedback information is imperative for the interaction process as it indicates the extent of interdependency of the parts in the whole system.According to Lessem and Shieffer (2010, p. 168), the use of systems theory focuses on systems breakdowns and malfunctions as well as the identification of subsystems which require change.In the context of this study, the systems theory enabled the evaluators to assess the EPWP effectiveness and its vulnerability to economic and political environmental factors during its implementation.

THE COST-BENEFIT APPROACH
To investigate the efficiency of the program, a cost and benefit approach can be applied.It is necessarily to investigate the EPWP costs and benefits of beneficiaries.In principle, the benefits must outweigh the costs of a program to avoid the failure of the program in achieving its objectives.Performance measurement also provides information on the budget

CONVERSION INPUTS OUTPUTS
allocation and the cost and benefit of the interventions like EPWP.This study evaluates the following direct and indirect benefits of youth work: • Direct benefits measured through: • economic value of volunteering and paid work; • the multiplier impacts of youth organization expenditures.
• Indirect benefits measured from the estimated longer-run costs avoided by the State through the provision of EPWP and service providers, performing in the following areas: • education related benefits; • public works; • social security; • municipal services.
It is expected that in the long run, the benefits must outweigh the program costs.Thus, the determination of a benefit-cost (BC) study would be to provide accurate measure of costs and benefits to determine the worth and value of the program.In terms of job creation and skills development, the authors seek investments with the highest return to the taxpayer and to the society.In the past, BC studies have been limited only on social benefits to masquerade government success on the reduction of job scarcity when the economy is growing.Sometimes evaluators use a Cost-Utility Analysis (CUA), which can be mistaken for cost-effectiveness analysis.The reason for using a CUA is to integrate the dimension of quality of life into the measurement of benefits.Benefits are measured as "quality-adjusted life-years", or QALYs, where gain is expected in a lifespan resulting from the program and is weighted by the quality of that life, as assessed through systematic surveying of perceptions of the affected (beneficiaries) or target population.

RESEARCH METHODOLOGY AND PROCEDURES
This research employed a qualitative document analysis as a method of collecting and analyzing the requisite data through content and discourse analysis.
In the context of this review, data collection and da-ta analysis involved analysis and review of literature from various policy documents, legislation, parliamentary report and books.This also includes filtered data using content and thematic analysis strategy.Bryman and Bell (2015) maintain that analytical induction begins with a broadly defined research question or questions, moves onto a hypothesis (if necessary), and then continues to the collection of data and the examination of a relevant case study.The limitation of this study is that evaluation concentrates on the evaluators and less on the beneficiaries of the program.Documents were drawn from the empirical studies focusing on implementation and monitoring and evaluation of EPWP to create a discourse on program monitoring and evaluation.
In addition, the researcher used a case study of the Extended Public Works Programme to assess whether the monitoring and evaluation framework was a viable tool to assess the effectiveness of poverty alleviation programs such as the EPWP.The researcher used other lenses to analyze the problem by means of the systems theory and cost-benefit approach.This study also adopted a deductive approach, which is not straightforward, since theory is often being used as a background to qualitative investigations (According to Bryman & Bell, 2015).

RESULTS AND DISCUSSION
In assessing the evaluation of the EPWP, several factors emerged from the literature as challenges of implementing the EPWP in South Africa.Some of these challenges can be grouped as institutional capacity, program costs and the matching of the EPWP with beneficiary needs.

EPWP FRAMEWORK AND PREPAREDNESS
The department of public works surveyed the problem regarding the evaluation of the EPWP by conducting research on the M&E of EPWP (DPW, 2014).However, it is clear from evaluation reports and meta-data that M&E was not compared with CEA to assess its suitability for evaluating EPWP.The DPW choice was to adopt the M&E approach since the infrastructure for GWM&E was already existing.There was no alternative evaluation approach determined, information was collected based on the standard procedures of M&E, which used a transversal approach of systematic data collection, costing and accountability based on existing policy framework on public financial accountability.The framework for evaluation was adopted with the intention to discharge the duties related to the monitoring and evaluation of the EPWP in all the phases.Figure 2 depicts the escalating costs on the framework of M&E that was tasked to evaluation efficiency of the EPWP.This study also rejects the evaluators observed ethical accountability during the systematic data collection on meta-data, such as salaries, inflation, discount rates, CPI and GDP trends, monitoring reports from the Ministry of Public Works and the performing instructions and Statistical Offices.
The framework increased government spending on salaries and instead of saving costs for sustaining the EPWP in various municipalities.For instance, the share on employee remuneration and the transfer to households grew up from the period of 2009 to 2014 (Public Works, 2014).This trend can be interpreted against the background of government spending, which also increased the GDP in those years.There are lessons on the merits of EPWP evaluation, the government earmarked improvement of performance in various entities through the EPWP projects was a success, with increased budget vote of various EPWP projects across the departments and municipalities.
There're limited accountability measures to key stakeholders since program monitoring relied on reports from the departments and municipalities, public budgeting must include stakeholder's engagement.Economic evaluation of EPWP also requires managerial leadership skills coupled the understanding and knowledge on budget and financial transactions.Program managers should comply with financial procedures and legal guidelines with a combination of skillful, both conceptually and treasury guidelines on budget preparation and spending.In addition, Public Financial Management Act (PFAM) was used as a guiding policy for managing finances.
To achieve this goal, the EPWP was expected to deliver the required public goods and services, labor intensively, at the required standards and primarily through public resources, as well as public and private joint capital for implementation.The framework applied for monitoring and evaluation was broad and complex, hence budget allocation in the first and second phases of the ERWP budget was decentralized in various departments.
In assessing the EPWP effectiveness, Mohapi (2013) observed the unmet outcomes of EPWP on its effectiveness in the North West Province.
Mohapi asserts that the program team and specialists such as engineers were expected to be technically educated and trained in the specific skills required for the program implementation.There was very low labor content of some infrastructure projects with less commitment from the officials to the EPWP concepts and principals of managing the projects.In some cases, officials mismatched the interorganizational planning boundaries with that of EPWP, where in some certain cases they become blurred leading to EPWP projects being loaded with other non-core activities.

Budget planning and costs
In the case of EPWP, monitoring and evaluation should have focused more on measuring its costs and benefits to come up with decisions regarding its economic value.The common problem with other evaluation models is the exclusion of costs and more concentration on benefits.Attention should be given to the cost of the EPWP since costs are a typical feature of the economic models of public policy and program evaluation.An evaluation of costs of EPWP would influence government policy makers to be transparent and accountable towards the use of public resources.Saving on the cost of public programs would benefit government in the long term since South African economy is struggling to grow while there is job scarcity.On the other hand, it should be noted that pro poor programs are part of rectification programs to enhance equity in the society.However, the justification of such programs lies on the evaluation results and the methods applied by government to improve the social and economic well-being of the beneficiaries.Program evaluation may be viewed as a politicized activity instead of a technical process that assesses social change in the society.This means that M&E involves a systematic decision-making that might be influenced politically.
Decisions regarding the cost effectiveness of the EPWP can be demonstrated by balancing the benefits against the financial costs of the pro-gram (Mid-term Report on EPWP, 2013).The government adopted the monitoring and evaluation framework (MEF) for the EPWP to track the progress of the programs.However, the paucity of beneficiaries as participants in all the evaluation of phase one and phase two of the EPWP are clouding evaluation findings.The first impression on the evaluation reports was on the success of job creation and infrastructure in a short term, which can be linked to the attainment of socio-economic development.The benefits of job creation for the youth was to provide services and stimulation of the informal and formal-sector economic activities.
Budget planning and financial management seem to be considerable imperative for conducting research.Besides political interference, it has emerged that the most powerful factor affecting the EPWP implementation in all the spheres of government is the availability of funds needed for conducting the evaluation (Henderson, 2016).
According to the Mid-term Review Synthesis Report on the Expanded Public Works Programme (2013), the evaluation on the worth of the program was conducted with systematic data collected to provide evidence supporting the implementation of the program in various provinces so that findings can be supportive.The evaluation framework for the EPWP costed the government almost R29.9 million over the five-year period if 1% of the beneficiaries are interviewed using cross-sectional surveys and 0.5% of participants from the EPWP are interviewed.Information was collected empirically using various strategies.However, evaluation reports failed to provide the qualifications of the participants and recipients of the EPWP, to establish the attributes attained by the recipients before they were part of the intervention and be compared with what they have gained after the intervention.
According to Moeti (2014), the Labor Force Survey of 2005 provided inconclusive results that isolated the impact of the program on costs.Data collection and analysis were systematically done by the evaluation committee, but DoPW is silent about the use of tools like rubrics by evaluators to make determinations and sharpen their focus evaluation.The results of the EPWP cross-sectional study (2013) provided evidence that would demonstrate impact of the program in real terms of borrowing, expenditure and saving.However, these were limitations since they do not provide detailed analysis of the levels of household income and expenditure.Table 1 illustrates costs incurred by government in evaluating the EPWP.It is unclear why the costs of the evaluation were escalating.In economic terms, the value for money has depreciated in South Africa due to currency volatility and increasing prices of goods.Pauw (2015) asserts that during the purchasing of public goods, managers must be more responsible towards the promotion of high value for money by achieving more outputs or less costs than what was budgeted for.
The evaluation reports also excluded the information about the number of provincial reports on projects that failed due to limited funding and continuity.It was difficult to analyze all transactional costs associated with those projects, since several projects which were not funded by EPWP were included on their lists.This elevated interrogations about the data reliability.According to the Human Research Council (2016), there are economic gains from the implementation of the EPWP even though some costs calculations were omitted.

The matching of EPWP objectives with beneficiaries needs
The EPWP assessment criteria would have worked better if it matched the economic impact of EPWP with the needs of the beneficiaries.Crouch (2012) argues that failures and mistakes in policy implementation can be gauged through evaluation of the economic worth and the program effectiveness.It should be noted that the preparedness for EPWP was assessed before the implementation of phases 1 and 2 of the EPWP.At that time, the experts were not aware of any environmental changes like economic factors such as recession and political instability.Theron (2014) asserts that the EPWP in the Western Cape had challenges that were observed in the implementation of the Western Cape Province community care givers since the deployees displayed their dissatisfaction towards the conditions of work and employment rewards.
As it appears in Table 1, the costs of evaluation were high.However, the challenge of the M&E framework was the burden of recovering costs on the application of the framework to measure the success of the program in the long term by assessing the benefits of the program as far as its economic and social impact on the beneficiaries.According to the EPWP (2010), the impact on the recipients implied program success in five years, as well as the extent to which EPWP had impacted the economy.Contrary to that, the official unemployment rate for the youth between 15 years and 24 years is 20% higher than for population.Chakwizira (2010) argues that the results of the EPWP must interrogate and go beyond quantitative data to include the qualitative examination of the conditions.The monitoring of financial accountability and credibility of useful information was analyzed to assess the perceptions of different stakeholders.Crouch (2012) also agrees that evaluators should focus on assessing the efficacy of the transfers in terms of direct or indirect microeconomic impact on participating households.It looks like there was no survey conducted to evaluate the microeconomic impact of EPWP on household.However, the EPWP had more policy implications that were credible to government officials than they would have been to

CONCLUSION
This paper assessed the economic value of monitoring and evaluating the EPWP in South Africa.There are lessons to be learnt from this research, as it reveals that monitoring and evaluation of public programs require a systematic generated information that may engender an awareness of and promote debates on the efficiency and economic value of public programs.For some interventions like EPWP, it is expected that the social and economic benefits will surpass the costs.The taxpayers also can only see the investments on the use of implementing public programs such as EPWP when the intervention is effective, and there are jobs created over a period.
The monitoring and evaluation framework may provide some indications of the progress and/or failure of a program.However, there was no other alternative economic evaluation tool studied and compared as an alternative against M&E.In addition, the tool may also signal to policy makers whether the implementation of a policy through the program has been successful or not.The reviewed literature also indicates that the adoption of neo-liberal policy as a strategy to create jobs through economic growth is not working.The evaluation of EPWP has raised mixed feelings on their effectiveness.It also emerged that program evaluation can hold government accountable on its mechanisms in place to provide feedback on how money was spent, and it implies changes in the budget planning, resource allocations and efficiency of the programs in question.It may be deduced that the monitoring and evaluation of EPWP framework for phases 1 and 2 revealed that South Africans could learn on how the EPWP enhanced economic growth and development during the difficult period of 2009 to 2012.Without EPWP phase 2, economic growth would have been lower and the unemployment rate, poverty and inequality higher than they were in the short term.
However, the simulations of the two phases of the EPWP reveal that the growth path that underpinned the EPWP II scenario benefited the youth in the short term.It may be deduced that the EPWP II and III scenario was not viable in the long term to support the ailing economy and increasing unemployment although there was a rationale for EPWP to be further implemented in phases 3 and 4. The economic evaluation of EPWP is a trial and error as some reports show that there were errors in the evaluation conducted.

RECOMMENDATIONS
1) Designing of long-term cost-benefit evaluations: Economic evaluation must be integrated, by assessing the cost-benefits of public programs in the long term, the risks were investigated associated with public programs, program preparedness before they are implemented.There is some merit in disseminating the program evaluation results to avoid repeating problems in the future.This review paper recommends that the drivers of program monitoring and evaluation must learn lessons from the EPWP and be willing to promote a culture of accountability and transparency to enhance program efficiency and effectiveness.
2) Isolation of program objectives and political mandates: Evaluators must separate proram objectives and political mandates since such mandates might not represent the interests of all the beneficiaries.
3) Professionalisation of economic evaluation is important.Evaluators must conduct further research to assess suitability of evaluation strategies to assess the economic value of programme evaluation in diverse sectors.Public agencies and departments must collaborate with academic institutions to foster training and skills development on programme evaluation.Both academics and practitioners have a responsibility to professionalise monitoring and evaluation knowledge area to consolidate its position in governance studies and related fields.

Figure 1 .
Figure 1.The simple system model The program roll-out of EPWP has been in three phases.In phase 1, one million work opportunities were targeted by 2008.Although jobs were created, unemployment rate remained high.During 2007, which was a period half-way to the end of the first cycle for the EPWP roll-out, the EPWP organized with the Expanded Public Works Support Programme (the Support Programme) custom-built a midterm review that would assess the EPWP performance to date, with the intention to monitor its progress and recommend correction plans and provide future direction and structure of the program.In the second phase, the EPWP was earmarked to create 4.5 million work opportunities from 2009 to 2014.However, there were developments during this phase, which were characterized by the introduction of the non-state sector, with two programs such as the provincial and municipal EPWP and the community work program.However, it was observed during phase two that the government new growth path policy (2010) managed to break the divide between the first and the second economy (Public Works, 2014).
Abrahams (2015)agrees that program monitoring and evaluation can be used as a governance tool in South Africa.This meant that the developed M&E framework involved review of existing monitor-Public and Municipal Finance, Volume 7, Issue 3, 2018 http://dx.doi.org/10.21511/pmf.07(3).2018.03

Table 1 .
The cost of evaluating the Expanded Public Works Programme Source: Employment and Economic Policy Research Unit, HSRC (2009).