RESEARCH ARTICLE

 

The appropriateness of a realist review for evaluating the South African Housing Subsidy Programme

 

 

Matodzi M. Amisi^I; Lochner Marais^II; Jan S. Cloete^II

^ISouth African Department of Performance Monitoring and Evaluation, Pretoria, South Africa
^IICentre for Development Support, University of the Free State, Bloemfontein, South Africa

Correspondence

 

 

---

ABSTRACT

Conducting meta-reviews of government programmes has become common practice. In South Africa, the national Department of Human Settlements and the national Department of Performance Monitoring and Evaluation recently commissioned a team to review the extent to which the Housing Subsidy Programme had provided assets to municipalities and the poor and whether these assets had helped poor households escape from poverty. A realist approach was employed to conduct the review. We argue that, given the complex nature of housing programmes, the realist review methodology was an appropriate approach to follow in answering the review questions. We explored how the realist review method allowed us to work with the uneven and contested nature of the housing literature and how the review nonetheless enabled elucidation of the factors that had contributed to the expected outcomes. Because this case was the first time that this method was used in a government-commissioned evaluation of housing, there were some practical challenges involved in its use. Some of the challenges were related to the nature of the questions that were asked. At the time of the review, the Department of Human Settlements was in the process of reviewing the 1996 White Paper and, to inform this process, the Housing Subsidy Programme review included a copious number of questions set by the Department of Human Settlements and Department of Performance Monitoring and Evaluation, which made the review rather large and, in some cases, complicated the analysis. In some cases, because the Departments wanted clear-cut answers, the commissioners perceived the theoretical strength of the method, such as offering explanatory instead of conclusive judgement, as a weakness. The paper reveals some limitations of the realist review method for evaluating the multifaceted outcomes of a complex programme, particularly the practical difficulty of dealing with large quantities of data. We do however consider this method to have potential for further reviews.
Significance:
•Housing research in South Africa is uneven which makes any review process difficult.
•The review was unable to offer judgement on the effect that the Housing Subsidy Programme has had on the asset base of the poor.
•The review was useful for making clear which factors will help the Programme to achieve the intended outcomes and also for pointing out on what government should focus to build assets for the urban poor.

Keywords: critical realism; housing assets; housing policy; poverty

---

 

 

Introduction

Evaluation and review of policy has become a common government practice across the globe. Many of these reviews take the form of meta-reviews, in effect studies of studies, in which the literature pertaining to specific policy concerns is closely examined. The demand for policy reviews has spawned an array of review methods: systematic, realist, scoping, critical, mapping - to mention but a few. In this paper, we assess the 'realist review' method, originated by Pawson and Tilley¹. For simplicity we have chosen to use the term 'realist review', while noting that this method is also referred to as 'critical realist review', as it stems from critical realist philosophy.

Globally, there is a growing body of work of evaluation of conventional review methods such as systematic reviews.² Some common criticisms are that the available evidence is often 'mixed or conflicting' and provides 'little or no clue as to why the intervention worked or did not work when applied in different contexts'³, that there are difficulties in striking a balance between rigour and relevance, and that 'few review types possess prescribed explicit methodologies and many fall short of being mutually exclusive'². Substantially more work is needed to evaluate review methods⁴, particularly in the health sciences⁵.

In South Africa, as elsewhere, evaluations and policy reviews have become the norm now that policy is increasingly expected to be evidence based.⁶ The national Department of Performance Monitoring and Evaluation (DPME), established in the Office of the President in 2010, has been mainstreaming reviews of policies and programmes in various line departments.

By the end of 2016, the DPME had completed 65 evaluations, 2 of which were meta-reviews. In 2014, the national Department of Human Settlements (DHS) and the DPME commissioned a review of South Africa's Housing Subsidy Programme. The review was to investigate the extent to which the Programme had succeeded in providing assets to the poor and whether these assets had helped poor households escape from poverty. We initially suggested the use of systematic review methodology to answer the review questions. Discussions with both the DPME and the DHS alerted us to the limitations of the systematic review approach in regard to the Housing Subsidy Programme. Because this Programme is implemented non-uniformly by the nine provinces - with each province using different implementation protocols in response to particular local contexts and moreover doing so in a variety of communities - our main problem was that we needed a review methodology that would be more flexible and would emphasise different contexts. In the end, we opted for the realist review method.

Apartheid planning left South African cities not only with large numbers of informal settlements and housing backlogs but also with municipalities that were ill prepared to accommodate rapid growth. The Housing White Paper released in 1995 was one of the first post-apartheid policy responses to the housing challenges faced by South African communities. Although multifaceted, the policy chiefly emphasised three things: ownership, a focus on the poor (only households with incomes of below ZAR3500 per month are able to access the subsidy) and a fixed-amount capital subsidy. (In 1995, the USD:ZAR exchange rate was 1:3.61 and about 1:13 at the time of writing in July 2017.) The original capital subsidy amount in 1995 was ZAR15 000 for those households with the lowest incomes. A revised policy, namely 'Breaking New Ground: A Comprehensive Housing Plan for the Development of Integrated Sustainable Human Settlements'⁷, has retained the above three elements while re-emphasising informal settlement upgrading and rental accommodation, and drawing attention to the need to establish sustainable settlements and to develop the property market. The South African Housing Subsidy Programme has delivered approximately four million housing opportunities (subsidised houses and site-and-services) in slightly more than two decades, mostly by providing a capital subsidy and homeownership to households at the lower end of the market.⁸

Despite the growing number of reviews and internal evaluations in South Africa there has been virtually no critical assessment of their methods. Against the above background, we critically assess the method we used and then discuss its appropriateness in terms of evaluating the multifaceted outcomes of the Housing Subsidy Programme. The fact that we as the authors represent both the commissioning department (the first author) and an academic department should ensure a balanced view. While we acknowledge that our closeness to the review process influenced our evaluation of the review, we did attempt to take a step back. We reflected with hindsight on what had helped or hindered the review process and its outcomes. In this paper, we discuss the limitations, and some benefits, of the realist review method.

 

Realist reviews: An overview

Realism is a school of thought that lies between positivism and constructivism.⁹ Pawson and Tilley^1,5 are credited with applying realist philosophy to programme and policy evaluation. The value of the realist method lies in its ability to deal with complexity³, to synthesise evidence while accepting that 'no deterministic theory can always explain or predict outcomes in every context'¹⁰. Evidence-based policy development is commonly described as wanting to determine 'what works'. However, in a realist review, we ask a more complex question: What is it about this programme that works for whom in what circumstances?³ In a realist review, the reviewers are able to engage with context and the human element in the implementation of interventions. There is an acceptance that different conditions contribute to programme success or failure^1,7,11 and that while diverse results are problematic, various outcomes are inevitable because the mechanisms that create change are not necessarily embedded within a specific programme but are often present in the thought processes of the programme's participants.¹ These diverse results must thus be explored rather than controlled.¹² A realist review therefore emphasises 'what works for whom, in what circumstances, in what respects and how'^3. Realist evaluators can use both quantitative and qualitative research methods.³

Realist reviewers engage with evidence by studying the interaction between contexts, mechanisms and outcomes, in what are called CMO (context-mechanism-outcome) configurations.^8,9 A CMO configuration is 'a proposition stating what it is about an initiative that works', in other words, an hypothesis to be tested.¹³

Conventionally, evaluators find it difficult to deal with how context mediates and moderates the results of a programme. Context is both perceived and treated as a threat to the external validity of evaluation where evaluators are concerned with isolating how programme interventions produced observed outcomes.¹⁴ Realist review methodology, however, allows evaluators to explore a variety of contexts and they try not to be judgemental.³ Understanding how context mediates and moderates programme performance is thus core to realist reviews. Mechanism is another central component of realist reviews. A realist review looks at the underlying causes of change that are not directly observable.⁷ Mechanisms could involve multiple individuals engaged in a sequence of processes.³ Mechanisms connect programmes to their outcomes. Realist review sees the outcomes as the result of interaction between the resources or opportunities the programme provides, the reasoning of its target population, and the context. The change process is studied to provide explanations for how change happens, not just to state what change has been observed.

Other principles besides the CMO configurations underpin a realist review. Firstly, a realist evaluator sees programmes as theories.¹ People design programmes on the basis of their beliefs about the nature of the problem and how change happens. This design is then translated by practitioners who are responsible for delivering services to programme beneficiaries. Thus, programmes are always inserted into existing social systems that have produced the negative conditions that necessitated the programme.¹ Because an intervention may involve multiple theories, using traditional review methods is difficult. In this regard, Pawson et al.³ note that 'the review question must be carefully articulated so as to prioritize which aspects of which interventions will be examined'. Programme motivations and designs usually make statements about how the programme or policy should be implemented and what results can be expected. Because a realist review usually starts by adopting the programme or policy design as the theoretical base, it must therefore consider the theory's underlying assumptions.

Secondly, as programmes are embedded in social systems, it is 'through the workings of entire systems of social relationships that any changes in behaviours, events and social conditions are effected'¹. A realist review therefore recognises and accepts the existence and interplay of multiple social systems. To understand the process of change, the reviewer needs to investigate beyond what the programme offers so as to understand how the wider social systems affect the programme. Traditional review methods are often unable to deal with this multiplicity and with interconnections in society. The realist review accepts that the relationship between mechanisms and outcomes does not have to be linear; in many cases it could be a reverse relationship. In accepting the existence of non-linear relationships, the realist reviewer notes and examines the 'flows, blockages and points of contention'^3. For example, the outcomes in societies that emphasise self-help might prove to be totally different from those in societies in which the state is required to play a dominant role. A second example relates to the fact that while the South African Housing Subsidy Programme grants individual households decision-making status, the decisions that households make might not be all that similar.

Thirdly, programmes are active. Implementation of a programme requires the active participation of individuals.^1,7 This principle is important and has methodological implications. For the realist reviewer, there is no need to control and remove the human influence. Instead, the reviewer needs to explore and understand how the human influence produces change in the intended programme.¹ In a realist review the literature review can therefore be broader than in a traditional review in which control and adherence to predefined programme components, population, types of studies, and so on, are critical. A realist review includes literature on the basis of relevance rather than restricting itself to a pre-identified finite set of sources. It generally uses a simple search strategy based on purposive sampling but multiple search strategies can also be used, and grey literature can be given a more important role than in other review types.

Lastly, because programmes are open systems, realist reviewers accept that externalities will always influence the way in which a programme is implemented, with benefits varying according to location. The programme implementer is an active agent in the implementation of the programme and context will constrain what is implemented.^1,7 Programmes can also be self-transformational. As the programme is implemented, it may be altered according to lessons learnt and may be adapted to context changes that have resulted from the introduction of the programme. A realist review must therefore be able to account for this adaptability. This aspect was important in the review that is the topic of the present paper, as the Housing Subsidy Programme policy had evolved significantly since 1994. From having an initial focus on starter houses in whose growth households were required to invest, the policy now makes provision for fully built houses of good quality that are aimed at incentivising market take-off.¹⁵

Realist reviews are not free of limitations. Realist review methods have been criticised for not being able to provide definitive answers to policy issues. The practical applicability of the realist approach has also been called into question, with some arguing that although, theoretically, the method offers useful lenses with which to look at programmes, it is difficult to apply these lenses with the methodological rigour and precision required of evaluators. A widely contested issue is how realist reviewers define and interpret causation. Realist reviews tend to emphasise contextual knowledge (what works for whom in what context) over normative positions; and then, too, the nature of causation is often debatable.^16,17 Effectively, realist reviews should pay attention to how existing world views influence specific studies and researchers' interpretations of the results. The danger further exists that researchers will choose literature that is in line with their own epistemological and ontological assumptions. Further criticism is that there is too little emphasis on the question 'does it work?' (as opposed to what works under what conditions) and an over-emphasis on contextual factors.¹³ It is these very criticisms that have necessitated this paper, which reflects on the practical use of the method while attempting to answer a policy question in a complex government programme.

 

Background to the Programme and implications for the review

South Africa's government-subsidised Housing Subsidy Programme is a complex intervention both in design and mechanisms for implementation (Figure 1). It is complex firstly because it has to respond to dysfunctionalities inherited from the apartheid government. The Group Areas Act of 1950 moved most black people from the core urban areas to impoverished and marginalised townships. Landownership for black people was revoked during the 1950s and only selectively reinstalled in the second half of the 1980s. The resulting inequality between black and white households should not be underestimated. As a result, the Housing Subsidy Programme was central to the political negotiations during the transition from apartheid to democracy and was important for restorative justice.¹⁸ Housing is now both a constitutional right that the state has an obligation to realise progressively (as affirmed in the Constitutional Court case of the Government of the Republic of South Africa vs Grootboom in 2000) and an individually owned asset that functions in the property market.^15,19,20 Responding to apartheid property-ownership biases (in urban areas), the Housing Subsidy Programme adopted an ownership model designed to redistribute wealth, ensure the participation of the poor (particularly black and coloured people formerly denied ownership in urban areas), and enable households to access and benefit from the workings of the property market.^21,22 The intervention logic or the theory of change was thus always more than the mere provision of accommodation. The provision of accommodation was a means to reduce asset poverty, address the failings of the market, give the poor equitable access to the property market and create wealth for those previously excluded (Figure 1).²³ The 2004 Human Settlements Strategy added to this a clear focus on asset creation as a means of poverty alleviation.²⁴ The theory of change was thus a market-based approach to asset building. Furthermore, when the Programme started, we had to accept the theory of change because it was the policy position adopted by the DHS. Later in the paper we note that during the review process we started to question this one-dimensional asset-building approach.

A second source of complexity is that the outcomes of the Housing Subsidy Programme are contingent on factors beyond its control or influence. Among these factors are macroeconomic conditions (employment, interest rates, and so on), concomitant investment in public spaces by local government, provision of municipal services, and the socio-economic conditions of the beneficiaries.

Thirdly, the intervention is complex because of its delivery arrangements. Nine provincial Departments of Human Settlements annually deliver housing by means of thousands of construction projects using a range of delivery arrangements with municipalities and private contractors. The nine provinces vary considerably in the way in which they package housing projects, select and appoint building contractors, monitor adherence to policy objectives, work with local governments to secure the spatial planning and other planning approvals necessary for project delivery, and provide bulk services such as water and sanitation. They also vary in the way they plan development so as to integrate low-income households with the rest of the municipality. A further complication is that architects and town planners make decisions about settlement design and land-use schemes (that in turn influence the development trajectory of a settlement). These decisions are made on a project-to-project basis so as to optimise the effective use of land and other resources.

Finally, the households that benefit from government housing subsidies vary in terms of economic circumstances, size and composition, level of education, and so on. To qualify for a subsidy a household must have a combined monthly income of no more than ZAR3500. But households in this income category may be unemployed and dependent on government grants, or formally employed with the possibility of upward economic mobility. They may be single-parent or two-parent households. The type of household determines or influences the extent to which a house will be an asset to that household and how well it will use the resources provided by the Programme. Variation in outcomes is thus only to be expected. Isolated studies on whether housing is elevating people out of poverty are likely to reach different conclusions.

All these complexities had implications for the review. In addition to the ideological context, we had to know the background of papers on housing delivery, such as in which province the research was conducted and the terms of the contractual relationships between developers, contractors and the provincial governments. We also had to take into account the fact that most housing research is currently being done in urban contexts and chiefly in four or five of the largest metropolitan areas, which, although not necessarily a negative, could give our review an urban bias. These factors significantly influence the ability of the Housing Subsidy Programme to achieve its policy objectives and tend to make the delivery mechanisms unduly dependent on context.

 

The review

The review was commissioned by the DHS and the DPME as part of the cabinet-approved National Evaluation Plan of 2013/2014. The DPME is the custodian of the Plan, as part of the implementation of the National Evaluation System. After 20 years of implementing the Housing Subsidy Programme, DHS reviewed its housing policy to respond to the transition to a broader human settlements approach initiated by the 2004 Breaking New Ground strategy and mandated in 2009 with the name change from 'Department of Housing' to 'Department of Human Settlements'. Our review was one of seven evaluations that the DHS conducted in partnership with the DPME, intended to influence and inform this policy review process. The need for a review emanated from this policy need.

The review questions

The review's specific focus was to 'determine if the provision of state subsidised housing [had] addressed asset poverty for households and created assets for municipalities'. More specifically, the review questioned whether subsidised houses were 'growing in value' and whether beneficiaries were indeed obtaining and benefitting from this growth. A set of 14 secondary questions pertained to the theoretical and conceptual understanding of housing and assets, asset generation for individual households and asset generation for municipalities (see Appendix 1 in the supplementary material for a full list of questions).

We had two difficulties with the review questions. Firstly, the focus on asset generation for both households and municipalities required us to combine two methods. Whereas to assess housing assets we could refer to the existing literature, to assess municipal assets we had to do new empirical work because little had been done. While these two types of assets are obviously linked, they are distinctly different issues for which a range of different assumptions exists. Secondly, each of the 14 secondary questions added a different emphasis. Although most of these questions were related, during the review process it proved difficult to devote sufficient attention to all of them. For example, the question about whether title deeds do indeed provide poor people with a platform for market access was a specific focus that required much attention - one that proved to be difficult to answer given that title deeds had to date been issued to only 50% of those households which had received a housing asset as part of the Housing Subsidy Programme. The wide range of questions necessitated a wide range of literature searches on the assumption that a considerable body of research is already available on each of the issues.

The review process

The main research question of the review was whether the Housing Subsidy Programme had provided assets to the poor and whether these assets had helped poor households to escape from poverty. The review process evolved in four phases over an originally envisaged period of 6 months. In the end, the process took more than 1 year to complete. In Phase 1, the DHS framed the questions in collaboration with the DPME and an evaluation steering committee, and subsequently appointed an external review team based at the University of the Free State to conduct the review. The review team had to suggest a review method. Originally, the review team proposed the idea of conducting a systematic review to the commissioning departments. In the inception phase of the project, the limitations of the proposed methods were pointed out by the commissioning departments; the weaknesses of this approach soon became apparent in the initial literature scan conducted by the review team. Most of the literature in housing was to be found in grey literature sources and not in academic studies. The existing research also varied in design, so that while many case studies had rich qualitative data, they suffered from a lack of randomised control trials or other impact-evaluation measures - a situation often encountered in health-related research. This situation provided further justification for the review team to change the initial method and a critical realist review was thus proposed to the commissioning departments. The commissioning departments, in approving this methodology, noted that it provided the necessary flexibility and also presented a methodologically defensible approach to respond to the review questions. Phase 1 also saw the introduction of a review team to the evaluation steering committee, one that was established by the commissioning departments in line with the requirements of the National Evaluation Plan. The evaluation steering committee comprised staff from the DPME, the DHS, National Treasury, a number of officials from local municipalities and a number of hand-picked academic researchers. The DPME also appointed two external peer reviewers to comment on the work of the review team at different stages of the review process.

Phase 2: Conceptualisation and search strategy

Once the review team was familiar with the terms of reference, the team familiarised itself not only with the housing theory of change pertaining to asset building but also with the various theories of asset building. The review team had to indicate from which paradigm it would view asset building. The team argued that it largely accepted the framework of asset building portrayed by the theory of change. Yet, it was also made clear that it would adopt a more critical and normative stance in this regard. The main point is that, as reviewers, we had to work with the theory of change prescribed by the Housing Subsidy Programme. In line with the realist position that programmes should be regarded as theory, the review team at this stage also spent time with the commissioning departments in reviewing and attempting to understand the theory of change that had been developed by the commissioning departments. After this, the review team developed a detailed methodology chapter in which it set out the literature search strategy, where the search would be conducted and how the information would be synthesised. This was an expansion of what the review team had presented to the commissioning departments during the project inception phase. In line with the realist approach, the search strategy comprised a set of search terms, databases to be searched and other information. The strategy, however, allowed for the review team to use other manual search processes like reference lists of studies reviewed and word-of-mouth suggestions by experts in the field of housing, which enabled the process to remain open and flexible as new literature was found and added to the review.

Phase 3: Search process

Phase 3 was a structured literature search using not only various databases but also documents provided by the DHS. In line with the realist review method, we formulated the following 28 CMO configurations, i.e. hypotheses, directly related to the theory of change provided by the DPME and the DHS:

•Housing subsidies improve social networks and create social capital.

•Housing subsidies improve health outcomes.

•Housing subsidies improve educational outcomes.

•Housing subsidies create security of tenure for women.

•Housing subsidies create security of tenure for the aged.

•Housing subsidies create security of tenure for the disabled.

•Housing subsidies reduce mobility.

•Housing subsidies improve household stability.

•Housing subsidies result in a higher degree of citizenship responsibility.

•The Capital Housing Subsidy results in a feeling of improved security of tenure.

•Housing subsidies engender feelings of belonging.

•Housing subsidies improve social inclusiveness and integration.

•Housing subsidies result in positive attitudes towards one's own 'asset' (house).

•Housing subsidies help restore people's dignity.

•Housing subsidies allow households to trade their units.

•Housing subsidies enable households to 'climb the housing ladder'.

•Housing subsidies allow people to raise collateral for other business activities.

•Housing subsidies make it possible to obtain mortgage finance.

•Housing subsidies reduce expenditure on transport if the houses are well located.

•Housing subsidies have a positive impact on home-based enterprises.

•Housing subsidies help increase household income.

•Housing subsidies can result in rental income.

•Housing subsidies lay the foundation for increased investment in housing.

•Housing subsidies lead to an increase in the property values of units.

•The informal trading of subsidised housing units mitigates their potential value.

•Housing subsidies improve households' access to employment.

•Housing subsidies alleviate poverty.

•Housing subsidies increase poverty.

The search process we followed was iterative and flexible, and continued throughout all the stages of the review. Unlike conventional review methods in which literature searches cover a specific period and follow a strict process that is articulated in a search strategy, in our review process, literature was included as and when it came to the notice of the review team. This iteration process enriched the review process and ensured that no important seminal studies were left out of the review process.

The realist approach requires contending with four main ideological viewpoints: the neoliberal, the Marxist, the American welfare-policy view and the developing country asset-accumulation view. The neoliberal view sees housing and asset building largely in terms of the market, whereas the Marxist view is that housing should in no way be commodified. Between these two extremes, we find two main schools of thought - one originating from research on asset building in the USA, emphasising the importance of investing in housing to pay for education and retirement²⁵, and the other from research in developing countries, emphasising the importance of asset building for poor people in urban areas, for health, employment and stability, and particularly for stability for migrants²⁰.

These ideological presuppositions dominate much of the research on housing. In contrast to the practice in the health professions, housing research findings do not originate from randomised control trials but mainly from case studies, and are influenced by the researchers' ideological presuppositions. Given South Africa's apartheid past, a large portion of housing research is situated within critical theory that is known to be sceptical of markets. During a feedback workshop one person remarked that 'these academic papers originate from non-market ideologies' - a strong statement, but it does indicate the extent to which ideology is involved in deciding whether housing has succeeded in creating assets for the poor.

In reviewing the literature we thus also had to understand the researchers' ideologies. We often had to make decisions about the value of a contribution solely on the basis of its authors' ideological presuppositions or had to take into account ideologically opposite findings. Overall, we could divide the studies into two categories: theoretically thorough work based on rather scant empirical results, and work based on large empirical data sets but theoretically shallow and moreover riddled with methodological concerns. The ideological problem was further complicated by the fact that the theory of change was based on the assumption of an ideal condition: increased access to the housing market for the poor. Table 1 shows - by means of an overview of the main findings from our sources - how we tested some of the CMO configurations.

We found approximately 1160 relevant sources with which to test our hypotheses; some sources were relevant to more than one hypothesis. The DHS also provided existing research and evaluations that they had previously commissioned. Then, we examined and assessed the titles and the available abstracts for relevance to the review questions. We found 320 research reports and papers to be relevant to the review questions. These sources included both academic and grey literature identified by means of the process described above. Towards the end of the search, we added new papers that we had found during the research process - a practice commonly followed in realist reviews. The existing research was found to have a number of shortcomings. Firstly, most of the already existing research focused on the early stages of housing development processes. Earlier studies tended to focus on variables or on the immediate outcomes of the housing development processes on households and neighbouring communities. Longer-term assessments were few. Because asset-generation is a long-term activity, the absence of long-term assessments was a major shortcoming. Evaluations over more than one generation are more likely to reflect on issues pertaining to asset building. Secondly, two paradigms of research dominate South African housing research. The first pole, critical theory, has been instrumental in challenging apartheid housing policies. The second pole contains research largely based on a positivist research paradigm or in some cases 'ideologically neutral'. Housing research is generally either conceptually or theoretically rich but empirically underdeveloped or empirically rich but conceptually poor. Thirdly, the notions of housing and asset accumulation are not a prominent research direction in South Africa. Asset-based welfare or asset-based development has received scant attention in South Africa. The majority of the research on asset generation has to date originated from NGOs and individuals not affiliated to universities. The majority of the research has moreover hitherto been narrowly focused on housing as an economic asset. Asset building is also not viewed in a more holistic framework - which happens to be the conceptual framework used in the present review. Lastly, because housing research in metropolitan areas dominates the housing research landscape, we also know very little about housing issues in smaller urban settlements.

Phase 4: Hypothesis testing

In Phase 4 we used these sources to test our 12 hypotheses. We identified the specific research contexts, mechanisms and outcomes related to each source linked to a specific hypothesis. In our review, we noted the extent to which housing practice as revealed by these sources was based on specific case studies and was therefore not necessarily generalisable. Finally, the DPME asked the project team to test whether the data collected supported the existing hypotheses. Having done this, we then sought further clarification through interviews with the authors who wrote the initial texts. Because the review was part of the National Evaluation Plan that adopted utilisation-focused evaluations, the participation of users of evidence was important. The review was therefore carried out with active participation of the implementing departments and their key stakeholders, including National Treasury. Like the previous deliverables, the results from Phase 4 were presented in an evaluation report that was submitted to the DPME and the DHS for review. To test our analysis, this report was also presented in a number of workshops attended by government officials, prominent academics working in this specific field and by people in the NGO sector. Although we, as the reviewers, had a certain level of independence, the stakeholders shaped the review questions and the different outputs of the review, such as including interpretation, analysis and recommendations.

Analysis

Having provided an overview of the process, we turn to an analysis of the review method.

Working with a contested theory of change

The national housing theory of change has a number of outcomes for which there is not always consensus. Although the literature on housing and the theory of change have a number of pathways through which households that receive fully subsidised houses are able to escape poverty and build wealth, one pathway has been dominant in research and evaluation. This pathway is that which argues that a functional property market will be created through the following ways: the subsidised housing appreciates in value; subsidised houses are incorporated into the property market; subsidised houses enter municipalities' rates rolls; the value of the poor's share of the property market grows; and the poor move up the housing ladder. The dominance of this particular pathway could, theoretically, and from a measurement point of view, be ascribed to the fact that it is relatively well established in the literature. However, the theory also acknowledges several factors that block this pathway: racially skewed participation in the property market (because apartheid determined suburbs along racial lines), biased distribution of resources and wealth, high levels of poverty and unemployment, minimal private sector investment in low-income areas, and a dearth of research on how black people - with little experience of dealing in the property market because apartheid prevented black ownership of property - function in the property market. On the positive side, the theory acknowledges factors that clear the pathway: well-located land, effective planning and deeds registration, the creation of functional neighbourhoods, access to private sector finance, and good quality housing.

The anomaly between the intended outcome and the contextual limitations entailed the risk that the reviewers could easily align with either a pro-market or an anti-market perspective. Probably more problematic is the fact that some of these inhibiting factors could prove to be so overwhelming that the theory of change might not be practically possible. It also provides only a single mechanism by means of which asset building can take place, namely the housing market. However, existing research suggests a range of alternative ways of creating assets²⁶, such as education, settlement stability and intergenerational transfers. Focusing a theory of change only on the market does not engender a holistic understanding of assets. The review team early on pointed this fact out and the DPME and the DHS accepted a broader understanding of housing assets. This revision highlights the importance of reaching agreement on the theory of change on which the review process is focused.

The importance of review questions

Pawson and Tilley¹ argue that reviews need clear policy questions which are suited to the approach. The review questions with which we were working were not developed with the realist evaluation approach in mind and there were also too many review questions. Because the review was commissioned by government departments, there was furthermore no flexibility to adjust the review questions. Having too many questions meant that a number of CMO configurations needed to be tested. This turned out to be a challenge and we were not always able to subject the CMO configurations to thorough testing. Also, the combination of a review and questions requiring primary research made the project difficult to manage. The lesson is that even when reviews have to respond to pressing policy questions, it is important that the questions be streamlined and that the commissioners of the research should not expect the reviewers to respond to all the pressing policy questions at the same time.

Synthesising and reporting issues

Although the realist review method is theoretically sound, in practice, the analysis of the relationships between context, mechanism and outcome requires much effort. This is a limitation that Pawson and Tilley acknowledge. The idea of programmes as open systems is, for example, theoretically useful because they allow the evaluators to see the programme as part of a broader social and economic system. This idea does, however, make the boundaries of the programme wide and thus not very definitive. In our case, this meant that a wide range of articles could be considered in the review. Also, because a realist review can potentially include a range of studies with different paradigms, methods, etc., that test a number of hypotheses, it can be intellectually enormously challenging. There are no simple tick-box solutions for how findings are presented. Attempting to synthesise across more than 400 studies, testing 12 hypotheses underpinned by four theoretical/philosophical views was not always easy. Although this, in itself, is not an issue, the review team had to work through a considerable volume of data. However, this volume of data combined with answering more than one review question and also synthesising across many studies, considerably complicated our task.

Reconciling methodological values and commissioners' expectations

The commissioners of the review hoped the review would offer judgement on the effect that the Housing Subsidy Programme has had on the asset base of the poor and also on the effect that housing has had on poverty. However, this was a difficult task. The ability of the Housing Subsidy Programme to produce assets that the poor can use to help them escape poverty is contingent on so many factors that it was difficult to declare with certainty what effect the Programme had had in which context and for which category of beneficiaries. Perhaps, too, the expectation was too high. As Pawson et al.³ argue, realist reviews can deliver only a better understanding of a programme and not general truths. For programmes as extensive and important as the South African Housing Subsidy Programme, this kind of knowledge is both appropriate and useful. The lesson is that evaluators using the realist approach need to work at securing an understanding from the commissioners of government evaluations as to the kinds of answers the approach will generate. The commissioners moreover have to invest in unpacking and interrogating the evaluation findings to understand the implications for policy. In this case, the evaluator really walks alongside policymakers²⁷ and plays the role of what Pawson and Tilley refer to as 'alerting the policy community to the caveats and considerations that should inform decisions'²⁸.

Contested programme outcomes

Because different theoretical paradigms, analytical lenses and various academic schools of thought are involved in research on the Housing Subsidy Programme, the existing literature yields no clear-cut normative position. Although the realist approach offers some ways of dealing with this ambiguity, it nevertheless remained difficult to synthesise the body of evidence on any of the CMO configurations that were tested. Obviously, the normative positions in many of the research papers were determined by the ideological positions of the researchers. Although we attempted to factor this preference into the analysis, it did not assist in creating a better understanding. In fact, the ideological divide just became bigger. Obviously, the contested nature of housing research reflects the contested nature of housing itself. Yet, from an evaluation point of view, it remains difficult to reconcile different conclusions from the same reality. There is evidence of both market failure and of some housing asset generation. If the market has failed 80% of the people who traded, could it be a valid conclusion that market failure should necessarily lead to the abandonment of the programme?

Dealing with limitations in primary studies

Housing research in South Africa is heterogeneous and the research landscape is dominated by regional, qualitative case studies. Systematic evaluations, particularly of government programmes, are even more limited. In housing, very few studies can be classified as 'programme evaluations' and even fewer have established the effects of programmes with sufficient methodological rigour. Any form of review that discounts qualitative findings is sure to bypass the bulk of the research in the South African housing sector. We were able to use the realist review method's CMO configurations to identify some regularities and patterns across the different local case studies. However, case studies and other cross-sectional studies were not adequate to respond to the issue of asset creation as thoroughly as the contractors would have liked. The criticism that the quality of research matters in critical reviews remains important and the uneven nature of existing research in our review was problematic. This was not a weakness of the methodology itself but did point to a lack of investment in theory development in most research and to weaknesses in how housing research agendas are crafted. There is limited synergy between policy issues and the kind of research being done by academics and other partners.

Despite the challenges reflected above, the approach was useful for clarifying which factors will help the Programme to achieve the intended outcomes and also for pointing out what government should focus on to build assets for the urban poor. It was also useful for clarifying on what not to focus, for example, the focus on the poor prevents market access for poor households in a secondary market.

 

Conclusion and final reflections

We have shared our experience of using the realist review method to evaluate a government programme. We have explained some of the difficulties in responding to broad policy questions and the way in which this method helped us to assess a complex social programme and deal with research of an inconsistent quality (mostly small studies using qualitative methods). A realist review can help in explaining what change is happening, for whom and how, and in showing which aspects of a programme create enabling conditions for results. These attributes make it a useful framework for reviewing politically important programmes like the South African government's Housing Subsidy Programme. The government does not intend to abandon the Programme; it is a central element of the country's democracy. The review, in which we took part, was intended to strengthen elements of the Programme that are not functioning properly to enhance performance and help achieve results. The findings of the realist review alert government to those components of the Programme that need strengthening and help it to respond to a context that is complex and evolving.

This methodology has much potential in reviews and evaluations of other large, complex government interventions. However, we have pointed out that, in our case, the review was complicated by a number of issues. Firstly, the theory of change with which we were provided emphasised one pathway of change, namely a market-orientated approach to asset building while a substantial portion of asset building takes place outside market processes. Also, because this particular theory of change is contested, the findings from different studies and comments/inputs from different sector experts were sometimes irreconcilable. Secondly, too many review questions inhibited a focused review and the analysis of the relationships between context, mechanism and outcome was difficult. This was further complicated by the fact that there are many different ways in which the poor use housing to escape poverty. The latter circumstance necessitated the need to test a number of CMO configurations, which considerably complicated the synthesising and presentation of findings. Thirdly, the ideological difference between housing provision and housing research was a dominant factor in assessing the literature. In the end, it turned out that very few studies used asset generation as an important point of departure, which in its turn made reference to the review questions more difficult. Fourthly, a further challenge was that of reconciling the explanatory nature of the findings from the realist review with the commissioners' expectations of conclusive findings as regards the impact of housing on asset creation. Lastly, South African housing research is not always empirically sound and most studies tend not to address issues that are relevant to policy. As synthesis relies on existing research studies, this shortcoming created its own challenges.

However, from our experience, realism offers potential as an alternative to conventional review methods, not only in evaluation synthesis but also in primary programme evaluations. The challenges faced in this review however should not deter those who want to explore the use of realism in assessing housing programmes or any other complex programmes. We have highlighted areas in which evaluators will need to re-think to improve the application of the methodology in programme evaluations.

 

Acknowledgements

The Departments of Performance Monitoring and Evaluation and Human Settlements are acknowledged for funding the original review.

 

Authors' contributions

M.M.A. and L.M. wrote the manuscript. J.S.C. made comments on the manuscript. All authors were involved in the original review - M.M.A. as part of the contracting and L.M. and J.S.C. as service providers.

 

References

1.Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.         [ Links ]

2.Grant M, Booth A. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Info Libr. 2009;26(2):91-108. https://doi.org/10.1111/j.1471-1842.2009.00848.x        [ Links ]

3.Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review: A new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10(1):21-34. https://doi.org/10.1258/1355819054308530        [ Links ]

4.Ankem K. Evaluation of method in systematic reviews and meta-analyses published in LIS. Libr Inf Res. 2008;32:91-104.         [ Links ]

5.Pawson R. Simple principles for the evaluation of complex programmes. Cidades- Comunidades e Territórios. 2004;8:95-107.         [ Links ]

6.Marais L, Matebesi Z. Evidence-based policy development in South Africa: The case of provincial growth and development strategies. Urban Forum. 2003;24(3):357-371. https://doi.org/10.1007/s12132-012-9179-4        [ Links ]

7.South African Department of Housing. Breaking new ground. A comprehensive plan for the development of sustainable human settlements. Pretoria: Department of Housing; 2004.         [ Links ]

8.South African Department of Human Settlements. Housing delivery data provided by the Department of Human Settlements. Pretoria: Department of Human Settlements; 2017.         [ Links ]

9.Westhorp G. Realist impact evaluations: An introduction [document on the Internet]. c2015 [cited 2017 Jul 17]. Available from: www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/9138.pdf        [ Links ]

10.Wong G, Greenhalgh T, Pawson R. What is a realist review and what can it do for me? An introduction to realist synthesis [presentation on the Internet]. c2009 [cited 2017 Jul 17]. Available from: http://pram.mcgill.ca/i/Wong_G_JUNE09_what_is_a_realist_review_presentation.pdf        [ Links ]

11.Marchal B, Van Belle S, Westhorp G. Realist evaluation [webpage on the Internet]. c2015 [cited 2017 Jul 15]. Available from: http://betterevaluation.org/en/approach/realist_evaluation        [ Links ]

12.Maxwell JA. What is realism, and why should qualitative researchers care? In: Maxwell JA, editor. A realist approach for qualitative research. Thousand Oaks, CA: Sage; 2012. p. 3-13.         [ Links ]

13.Linsley P, Howard D, Owen S. The construction of context-mechanisms-outcomes in realistic evaluation. Nurse Res. 2015;22(3):28−34. https://doi.org/10.7748/nr.22.3.28.e1306        [ Links ]

14.Pritchett L, Sandefur J. Context matters for size: Why external validity claims and development practice do not mix. J Global Develop. 2013;4(2):161-197. https://doi.org/10.1515/jgd-2014-0004        [ Links ]

15.Tomlinson M. From 'quantity' to 'quality': Restructuring South Africa's housing policy ten years after. Int Develop Plan Rev. 2006;28(1):85-103. https://doi.org/10.3828/idpr.28.1.4        [ Links ]

16.Edgely A, Stickly T, Timmons S, Meal A. Critical realist review: Exploring the real, beyond the empirical. J Further Higher Educ. 2016;40(3):316-330. https://doi.org/10.1080/03        [ Links ]

17.Farrington D. Methodological quality standards for evaluation research. Ann Am Acad Polit Soc Sci. 2003;587(1):49-68. http://journals.sagepub.com/doi/abs/10.1177/0002716202250789        [ Links ]

18.Amisi M, Vawda A. Strengthening democratic governance in human settlements through evaluations. In: Podems D, editor. Democracy and evaluation: Exploring the reality. Charlotte, NC: Information Age Publishing; 2017. p. 122-138.         [ Links ]

19.Tissington K. A resource guide to housing in South Africa 1994-2010: Legislation, policy, programmes and practice. Johannesburg: Socio-Economic Rights Institute of South Africa; 2011. Available from: www.seri-sa.org/images/stories/SERI_Housing_Resource_Guide_Feb11.pdf        [ Links ]

20.Dawson H, McLaren D. Monitoring the right of access to adequate housing in South Africa: An analysis of the policy effort, resource allocation and expenditure and enjoyment of the right to housing. Studies in Poverty and Inequality Institute working paper 8 [document on the Internet]. c2014 [cited 2016 Nov 17]. Available from: http://spii.org.za/wp-content/uploads/2014/09/Working-Paper-8_Monitoring-the-right-to-adequate-housing-in-SA.pdf        [ Links ]

21.Huchzermeyer M. Housing for the poor? Negotiated housing policy in South Africa. Habitat Int. 2001;25:303−331. https://doi.org/10.1016/S0197-3975(00)00037-0. https://doi.org/10.1016/S0197-3975(00)00037-0        [ Links ]

22.Charlton S, Kihato C. Reaching the poor? An analysis of the influences on the evaluation of South Africa's housing programme. In: Pillay U, Tomlinson R, Du Toit J, editors. Democracy and development: Urban policy in South Africa. Cape Town: HSRC Press; 2005. p. 252-282.         [ Links ]

23.South African Department of Human Settlements and Department of Performance Monitoring and Evaluation. Synthesis evaluation of whether the provision of state subsidised housing addressed asset poverty for households and local municipalities? Pretoria: Department of Human Settlements and Department of Performance Monitoring and Evaluation; 2015.         [ Links ]

24.Shisaka Development Management Services. Housing subsidy assets: Exploring the performance of government subsidised housing in South Africa. Johannesburg: FinMark Trust; 2011.         [ Links ]

25.Sherraden M, Gilbert N. Assets and the poor: New American welfare policy. New York: M.E. Sharpe; 1991.         [ Links ]

26.Moser C, Felton A. Intergenerational asset accumulation and poverty reduction in Guayaquil, Ecuador 1974-2004. In: Moser C, editor. Reducing global poverty: The case for asset accumulation. Washington DC: Brookings Institution Press; 2007. p. 18-34.         [ Links ]

27.Rallis S, Rossman G. Dialogue for learning: Evaluator as critical friend. New Dir Eval. 2000;86:81-92. https://doi.org/10.1002/ev.1174        [ Links ]

28.Pawson R, Tilley N. Realist evaluation [document on the Internet]. c2004 [cited 2018 Feb. 16]. Available from: http://www.communitymatters.com.au/RE_chapter.pdf        [ Links ]

 

 

Correspondence:
Lochner Marais
MaraisJGL@ufs.ac.za

Received: 04 Sep. 2017
Revised: 16 Feb. 2018
Accepted: 13 Aug. 2018
Published: 27 Nov. 2018