ORIGINAL RESEARCH

 

Spirituality and impact evaluation design: The case of an addiction recovery faith-based organisation in Argentina

 

 

Severine Deneulin^{I, II}; Ann Mitchell^III

^IDepartment of Social and Policy Sciences, University of Bath, Bath, United Kingdom
^IIDepartment of Theology and Missiology, Stellenbosch University, Stellenbosch, South Africa
^IIIDepartment of Economics, Pontificia Universidad Católica, Chile, Argentina

Correspondence

 

 

---

ABSTRACT

The importance of the spiritual dimension in the lives of people living in conditions of poverty and social exclusion and the often-critical role of faith-based organisations has gained increasing relevance in development research and practice. A growing line of research focuses on how to integrate the faith dimension into the evaluation of social programmes and on quantifying the effects of faith. The objective of this article is to propose a framework for integrating a spiritual dimension into the design and practice of impact evaluation by using the concept of integral human development (IHD). Integral human development emerged within the Catholic social tradition, but is not specific to it. It is a perspective on human development that integrates the material and spiritual, recognises the interdependence between humans and their environment or territory and demands change at both the individual and collective levels. This framework is then applied to the design of an impact evaluation of a faith-based programme that accompanies people with drug and alcohol addictions in Argentina. The article highlights the following characteristics of an integral impact evaluation: the interaction between multiple well-being dimensions; the use of knowledge and methods of analysis from multiple disciplines; the importance of understanding the diverse pathways to improvements in well-being; the assessment of the spiritual dimension through changes in one's relationships with oneself, others and the environment; and the importance of assessing personal change within the context of social and community transformation.

Keywords: Religion; Impact evaluation; Integral human development; Capability approach; Youth; Addictions.

---

 

 

Introduction

At the midday meal of a neighbourhood addiction attention centre of an Argentine faith-based organisation (FBO), Jorge,¹ an 18-year-old young man, showed us proudly a picture of himself on his cell phone at age 16 at the height of his addiction to paco, or cocaine base paste.² The picture showed an emaciated boy with his eyes looking downward and a cap trying to hide his face. Jorge comes every day at noon to the centre, to take his place at a long table in a dining room and eat with others who suffer from addiction problems and with the volunteers and paid workers of the organisation. After the meal, everyone helps to clear up, clean the premises, or do household chores. There are also sessions with social workers or psychologists. And so the routine goes on every weekday from 12 to 5 pm. Jorge is now a smiling and more healthy-looking young man, eager to make social contacts with everyone. The staff, however, tells us that he is still at risk and can relapse into addiction at any time.

Drug use and addiction has escalated in Argentina during the past two decades (Epele 2011; SEDRONAR 2017). Substance abuse negatively affects not only the health, education and long-term opportunities of drug users and their families, but also the wider community by increasing violence and insecurity (Organización de Estados Americanos 2014). There is also a strong relationship between drug consumption and the territorial and sociocultural context of disadvantage in Latin America (Ullmann 2015).

One study on informal settlements in Greater Buenos Aires showed that 22% of youth had consumed drugs during the preceding month (Rival & Salvia 2016).

Within this context, in 2008 Catholic priests living in three of the City of Buenos Aires' informal settlements created 'neighbourhood centres' to provide an integral response to socially vulnerable people who suffer from addictions. The guiding principles of the neighbourhood centres include, in the words of its leaders, 'to welcome life as it comes', to create a community family and to understand that each person has a unique path and possibilities for addiction recovery. Today, the Hogar de Cristo ([Home of Christ] in Spanish) is a federation of 123 neighbourhood centres throughout Argentina.

As this special issue notes, the international development sector has increasingly recognised the role of such FBOs in addressing poverty and social exclusion, especially among youth living at the margins (Le Roux, Hankela & McDonald 2018). There is also an emerging literature on how to integrate the faith dimension into the evaluation of FBO programmes and on quantifying the effects of faith (Heinrich, Leege & Miller 2008; O'Neill 2017). The existing literature, however, falls short of taking into account the full theological underpinnings of incorporating faith into impact evaluation.

The objective of this article is to propose a framework for integrating a spiritual dimension into the design and practice of impact evaluation by using the concept of integral human development (IHD). The IHD concept emerged within the Catholic social tradition, but is not specific to it. It is an integrated perspective for social analysis which integrates the material and spiritual dimensions of life, recognises the interdependence between humans and their environment or territory and demands change at both the individual and collective levels.

Our proposed framework for an integrated perspective on impact evaluation is based on three sources of information and analysis. Firstly, we examine the concept of IHD as expressed in the Catholic social tradition, identify its defining characteristics and explain its relation to Amartya Sen's capability approach and how the approach serves to clarify and operationalise the IHD concept. Secondly, we review the literature on impact evaluation focusing, in particular, on the strategies that have been used to incorporate a spiritual dimension into programme evaluation. Our third source is qualitative empirical research with the Hogar de Cristo. During the 4-month process of designing the impact evaluation, we carried out interviews with organisation leaders, spoke with programme participants, observed neighbourhood centres' activities, read numerous documents describing the organisation's guiding principles and mode of operation and spoke with government officials at the national Argentine agency responsible for the prevention and treatment of persons with drug addictions (SEDRONAR).

For illustrative purposes, we apply the proposed framework for integral impact evaluation to the design of an evaluation of the Hogar de Cristo. Our research project's objective is to evaluate how the organisation contributes to improving the lives of participants, their families and communities, focussing, in particular, on understanding participants' perceptions about what changes have occurred in their own lives and why. The project aims to help the organisation develop a system for monitoring its activities and outcomes and gain insights that can contribute to improving the effectiveness of the organisation's activities.

The impact evaluation will be carried out in the neighbourhood centres located in the City of Buenos Aires,³ the locality with both the longest history and the largest number of participants.

The article is structured as follows. Section 'Integral human development' presents the IHD concept and its relation to the human development literature and Amartya Sen's capability approach. Section 'The spiritual dimension in impact evaluation' reviews the literature on the spiritual dimension in impact evaluation. Section 'Hogar de Cristo' describes the faith-based drug addiction recovery programme Hogar de Cristo. Section 'Integral impact evaluation' proposes a methodology for accounting for the spiritual dimension in impact evaluation and applies this framework to the design of an evaluation of the Hogar de Cristo.

 

Integral human development

Despite recent growth in the literature on religion and international development,⁴ little effort has been made to integrate a spiritual dimension into the very concept of development. Advances in this direction, however, have existed since at least the 1960s within theological circles and faith communities. A pioneering document, entitled Populorum Progresso (On the Progress of Peoples), which sets the ground for integrating spirituality into the concept of development and written in 1967 by Pope Paul VI, makes the first reference to the concept of 'integral human development'.⁵ It states:

The development We speak of here cannot be restricted to economic growth alone. To be authentic, it must be well rounded; it must foster the development of each [wo]man and of the whole [wo]man. (p. 14)⁶

The document discusses how recognising the 'wholeness' of human life implies also recognising the human person's openness to the Transcendent. An integral perspective on human development does not separate the material and spiritual dimensions of life, but rather integrates them. This spiritual dimension, the document highlights, is especially expressed in the 'values of love and friendship, of prayer and contemplation' (PP 20). In a document which furthers the concept of IHD, entitled Laudato Si': On Care for our Common Home published in June 2015, Pope Francis associates the spiritual dimension of life with 'openness to the gift of God's creation' (LS 85), for there is 'mystical meaning to be found in a leaf, in a mountain trail, in a dewdrop, in a poor person's face' (LS 233).

A first implication of integrating a spiritual dimension into development is recognising the uniqueness of each human being, with a special attention to those who live in or are vulnerable to poverty and social exclusion, what the Catholic social tradition calls the 'preferential option for the poor'. This also points to the need to take into account the territorial environment in which a person lives, for the human person is inseparable from the natural environment of which she is part (LS 6). Therefore, the promotion of the human person must be united with the care of the physical territory in which he or she lives, for the deterioration of the environment negatively affects the human person and vice versa (LS 48-49).

A second aspect of IHD, and of recognising the 'whole of [wo]man' (PP 14), is multidimensionality.

In a document entitled Caritas in Veritate (Charity in Truth) published in 2009, Pope Benedict XVI talks of 'authentic human development' (CV 11) as concerning 'the whole of the person in every single dimension'. This concern, he argues, 'requires a commitment to foster the interaction of the different levels of human knowledge in order to promote the authentic development of peoples' (CV 30). Pope Francis extends this multidimensional aspect by analysing how the relational, social, economic, cultural, political, ecological and spiritual dimensions interact in the context of climate change (LS 101-136). This is why he argues that no branch of knowledge and science, including different forms of wisdom, can be left out when developing actions to address poverty and ecological degradation (LS 63). He puts special emphasis on the relational dimension, which includes the relation to oneself, to others, to God and to the natural environment (LS 65-66), and the agency or freedom dimension, that is, our capacity to make decisions and how we choose to live out these relations.

A third aspect of IHD is the recognition that 'everything in the world is connected' (LS 16, 91, 117, 220, 240). This awareness of how everything is interconnected is the foundation of solidarity and responsibility towards each other and the environment. In a document entitled Sollicitudo Rei Sociales (On Social Concerns) published in 1987, John Paul II defined solidarity as a:

[F]irm and persevering determination to commit oneself to the common good; that is to say to the good of all and of each individual, because we are all really responsible for all. (SRS 38)

As everything is interdependent, the way we use our freedom to act has consequences for others and the whole of creation, including our own lives (LS 33, 205). Pope Francis describes interconnectedness as an invitation to 'develop a spirituality of global solidarity' (LS 240), for there is a relation between 'a sort of super development of a wasteful and consumerist kind which forms an unacceptable contrast with the ongoing situations of dehumanizing deprivation' (LS 109). The Catholic social tradition links this situation of injustice to the sum of individual actions which produce and consolidate certain structures, which become unresponsive to individual action (SRS 36-38), for example, a culture of over-consumption and waste is not transformed by isolated individual actions. This is why an IHD perspective demands change at both the personal and social levels, for social problems are tackled 'by community networks and not simply by the sum of individual good deeds' (LS 219).

Although the concept of IHD has essentially been developed from within the Catholic social tradition since 1967, it is not confined to it. We find traces of the key characteristics of IHD in the concept of development put forward by Amartya Sen (Sen 1988, 1999, 2017; United Nations Development Programme [UNDP] 2016). We highlight some ways in which Sen's capability approach to development incorporates the features outlined above and can serve as a useful tool to evaluate social programmes from an integrated perspective.

At the heart of Sen's capability approach to development (which conceptually grounds the human development framework advanced by the UNDP's Human Development Reports) lies the argument that progress should be assessed 'in terms of what people are able to be or able to do [i.e., their capabilities], rather than in terms of the means or resources they possess' (Sen 2017:357). There is progress when people are able to be and do certain things such as 'being alive, being well-nourished and in good health, moving about freely', or 'having self-respect and respect for others, taking part in the life of the community' (Sen 2017:357).

Sen has deliberately left open the choice of what constitutes valuable or relevant capabilities for assessing progress, making his approach a rather operational challenge (Alkire, Qizilbash & Comim 2008).

The consensus in the capability literature is to base the normative process for selecting dimensions on a variety of methods, such as participatory approaches, moral or political entitlements, theoretical arguments, empirical evidence or capability lists proposed by scholars, most notably by Martha Nussbaum (2011).⁷ The choice will depend on the purpose of the analysis as well as practical constraints such as data availability (Alkire et al. 2015), and even what counts as valuable will be context dependent (Robeyns 2017). Authors such as Sabine Alkire (2002) have considered capabilities related to the faith dimension, such as 'being in inner peace' and 'being in relation to a higher source of value'. From the IHD perspective, the spiritual aspect of life can be best captured through relational capabilities (to self, others and the territory) such as the ability to know oneself, having strong and supportive relationships with others and respecting the natural environment.

The capability approach also provides a framework for operationalising the principle of 'each person as an end', which it shares with IHD. As each person is of equal moral worth but unique, each will need different amounts of resources to achieve the same 'being' or 'doing' depending on his or her individual, social and environmental characteristics, what is called 'conversion factors' in the approach's jargon (Robeyns 2017:45-47; Sen 2017:26). This concern for human diversity reinforces the need for adopting multidimensional interventions (e.g. enabling persons with addiction to recover their health, job, social relationships and so on), applying different strategies in different social, economic and environmental contexts, and considering the multiple and diverse pathways to impact.

The IHD perspective emphasises the importance of transformative action at both the personal and community or collective levels. Sen's capability approach operationalises this by situating public reasoning and dialogue as the locus for personal and social transformation. For Sen, interaction with others, especially with those who are most disadvantaged and trying to see the world from their perspective, is central to changing people's priorities and views about what should be done (Sen 2017). He underlines the importance of a dialogue which is centred on the lives of those who experience suffering and on listening to their views and experiences. This dialogue, he argues, is the starting point for personal and policy transformation (Sen 2009).

Before examining further the implications of the IHD perspective for impact evaluation, we review the existing literature that seeks to integrate a spiritual dimension into impact evaluation.

 

The spiritual dimension in impact evaluation

During the past 20 years there has been an increased focus on impact evaluation and evidence-based policy design (White 2014). Impact evaluation differs from other types of programme evaluation in that it seeks to measure causal effects, that is, the changes in the well-being of beneficiaries that can be attributed to a particular intervention. While randomised control trials are considered by some to be the gold standard for impact evaluation (Duflo, Glennerster & Kremer 2008), they have received criticism for being unethical and for providing little information on the distribution of treatment effects, on how or why the effects occur and on the probable impact if implemented in other contexts (Barret & Carter 2010; Deaton & Cartwright 2018).

An array of alternative methods has been developed to obtain valuable information on program effects through nonexperimental techniques.⁸ Quasi-experimental methods, such as propensity score matching or regression discontinuity design, use statistical methods to measure causal effects when randomisation is not possible (Gertler et al. 2016). Participatory evaluations engage beneficiaries and other stakeholders in the design and application of programme assessment (Lilja & Bellon 2008). The realist approach uses diverse methods of data collection and analysis to gain understanding of how and why programs work within a specific social system and relies heavily on theory for understanding causation (Pawson & Tilley 1997). The Qualitative Impact Protocol (QuIP) method measures impact by asking intended beneficiaries what they think are the most important drivers of change in selected domains of their lives and well-being. Qualitative Impact Protocol is designed to promote transparency in data analysis and avoid the problem of confirmation bias (Copestake, Morsink & Remnant 2019). Case studies gain insight into programme effects by employing quantitative and qualitative data to intensely study a specific case within a particular context (Balbach 1999).

In recent years, new approaches have been put forth to integrate a spiritual dimension into impact evaluation. For example, Christian charities such as World Vision, Tearfund, the Salvation Army and Catholic Relief Services (CRS) have proposed frameworks for the design, implementation and evaluation of their organisations' activities. These approaches have several commonalities. Firstly, they take a holistic or integral view of development. Catholic Relief Services, for example, taking from Pope Paul VI's Populorum Progressio encyclical, promotes the 'good of every person and the whole person; an individual's cultural, economic, political, social and spiritual wholeness' (Heinrich et al. 2008:2). Tearfund (2016) uses a 'Light Wheel' as an analogy for its goal of pursuing 'holistic development'.⁹ The nine spokes of the wheel represent domains over which individuals and communities can live well and the wheel's rolling journey represents the interaction with the context. Secondly, all of these approaches are concerned not only with outcomes but also with development processes, understanding that people play a vital role in their own development. For example, World Vision's (2007) monitoring and evaluation framework for youth programming involves young people in defining indicators of success, collecting and reviewing data and learning and planning with local partners.

Faith or spirituality is often incorporated as an additional dimension within a multidimensional framework. For example, the 'living faith' spoke within Tearfund's Light Wheel symbolises the importance of faith within the community. One of the dimensions of quality of life that World Vision seeks to contribute to is 'transformed relationships', which encompasses a restored relationship with God: equitable, just, peaceful and productive relationships within households and communities and a responsible relationship with the environment. The International Joint Learning Initiative on Faith and Local Communities¹⁰ - an international effort to build evidence on the effectiveness of faith groups - shares knowledge and experience on the construction of spiritual metrics, such as measures of trust, hope and love, and spiritual growth. One line of research has focussed on quantifying the causal effects of faith (O'Neill 2017), such as a randomised control trial to measure the effects of teaching Christian values in the Philippines (Bryan, Choi & Karlan 2018).

Addiction recovery is an area in which research on the relationship between faith and programme effectiveness is most abundant, likely because of the long history of faith-based approaches to addiction recovery (White & Whiters 2005). A significant topic of this research concerns the assessment of the importance of individual religiosity versus religious emphasis of treatment programmes on addiction recovery (Alan, Bell & Carlson 2000).

The above-mentioned approaches have brought attention to the need to broaden our conception of development and incorporate spirituality into impact evaluation design. The existing literature, however, tends to focus mainly on adding faith as an additional domain, and does not sufficiently analyse the full implications for programme evaluation that can be drawn from an IHD perspective and the concrete experiences of FBOs embedded within communities.

In the next section, we seek to gain further insights on how to integrate a faith dimension into impact evaluation by examining the experience of an organisation which aims to have an impact on the lives of people suffering from drug addiction and on their families and communities, and which situates its programmes within the wider context of a faith community.

 

Hogar de Cristo

The Hogar de Cristo's stated goal is to provide an integral response in situations of social vulnerability and problematic consumption of psychoactive substances.¹¹ Although the neighbourhood centres emerged in response to drug addiction (primarily to the highly addictive paste base cocaine), it soon became clear that substance abuse was a symptom of a much broader problem of social exclusion in neighbourhoods afflicted by precarious housing, inadequate access to public services, violence, health risks, lack of jobs and the inadequacy of government response. The situation demanded a response that encompassed not only the treatment of addiction but also the satisfaction of diverse related needs from housing and food to health or childcare services.

Following the faith-based conviction that each person is unique and sacred, the organisation does not define a uniform formula for recovery, but rather seeks to respect the possibilities of each person, aligning the forms of assistance with the specific needs and pace of each individual. Through the process of helping each person to satisfy her or his basic needs and finding solutions for concrete problems, the team of psychologists, social workers, volunteers and participants form relationships and a sense of belonging to the community, which they call the familia grande [or big family]. As in a family, the aim is to make each person feel welcome, supported and loved.

The neighbourhood centres' activities include the provision of meals and showers, group meetings, workshops ranging from theatre and cinema to football, job training programs and childcare services. Once an individual makes a commitment to begin addiction recovery, if the case cannot be treated through the centre's daytime programmes, the organisation facilitates access to one of its two treatment farms located in the peri-urban area of Buenos Aires, or seeks public funding for treatment in one of the city's private residential addiction treatment centres. After completing therapeutic treatment, many participants live in one of the organisation's homes, where they initiate a process of reincorporation into school or job training programs and eventually work.

Consistent with a growing body of literature on the ecology of alcohol and drug addiction (White 2009), the neighbourhood centres are rooted in the community and the territorial links are evident at all stages of the recovery process. The team of professionals and volunteers do not wait for people to approach the centre, but rather conduct outreach activities, offering hot coffee under a tent on a busy street corner or hanging out with groups of drug consumers in the neighbourhood's alleyways. Many of the participants' family members attend the centres' support groups and form part of the community. One collaborator explained that it is by working immersed within the community and in collaboration with other organisations and networks that the neighbourhood centres are able to provide an integral response to persons with addictions.

Rapid growth over the past 10 years in the network of neighbourhood centres and in the number of people served by each centre (ranging from about 50 to 1000) has created a need to monitor the organisation's activities through data collection and analysis. There is also an increasing demand for evidence on the programme's effectiveness from the national drug prevention and assistance agency, which helps finance the organisation's activities.

 

Integral impact evaluation

In this section, we propose a framework for incorporating a spiritual dimension into impact evaluation. On the basis of the defining characteristics of IHD, their operationalisation through the capability approach and further insights which emerge from the Hogar de Cristo's experience within the community, we derive the following components of an integral impact evaluation: (1) consideration of the whole person in the multiple dimensions of her life; (2) integration of perspectives and methods from different disciplines; (3) recognition of human diversity and multiple pathways to life improvement; (4) significance of relations to self, others and the territory as part of our relation to God; and (5) importance of dialogue and communal change. For each component, we suggest how it can be applied to the design of an evaluation of the Hogar de Cristo.

The whole person in her multiple dimensions

Seen from an IHD perspective, the aim of development is to foster the development of each person and the whole person in her multiple dimensions of life. An integral evaluation, therefore, will measure the effects of a social intervention on multiple life dimensions,¹² integrating material and spiritual, human and territorial and personal and collective realms of life. It requires not only an assessment of the effects on the intervention's direct objectives (say, the effects on nutrition of a community kitchen), but also on a broader set of outcomes (say, on opportunities for social interaction or for eating together as a family). The selection of the dimensions of analysis will require a close understanding of the local context, programme activities and the development of a 'theory of change', a process commonly used in impact evaluation to understand how activities produce intermediate results and final programme impacts. An initial mapping of these processes can serve to identify the multiple outcome dimensions and examine the expected interaction between them (e.g. how the capability 'to walk safely on the streets at night' affects the capability 'to attend school in the evening' or how lacking the capability 'to be in inner peace with oneself' affects levels of violence). As the IHD perspective recognises the uniqueness of each person, the diverse pathways to change cannot be fully understood in advance, but rather must be studied throughout the evaluation process, a topic which will be discussed further in point (3). An integral impact evaluation also places special emphasis on the analysis of the effects on the relational dimension, as explained in detail in point (4).

Interviews with organisation leaders, observation of neighbourhood centres' activities and a review of the broad literature on addiction recovery programmes (Kelly, Magill & Stout 2009; Machado 2005, among others) were the sources of information we used to identify the following five possible capability dimensions to be used in the evaluation of the Hogar de Cristo: (1) basic needs; (2) health; (3) relations to self, others and the territory; (4) education and (5) work. The basic needs dimension refers to the capability to satisfy material needs such as food, hygiene and shelter. The Hogar de Cristo's activities often begin by addressing concrete problems and needs that keep participants from beginning the addiction recovery process. Health, a vital human functioning, is the outcome dimension that tends to be the main focus of most evaluations of drug treatment programmes. Progress in this dimension may include reductions in physical and psychiatric illness and changes in drug and alcohol use and treatment continuity. The relational dimension will analyse progress in relationships with oneself, with others and with the territory (cf. point 4 for further elaboration). All of these could be seen as manifestations of one's relationship with God, expressed through love and friendship and openness to God's gift of creation, as described in the section 'Integral human development'.

According to neighbourhood centre leaders, the dimensions of education (reincorporation into formal education institutions or job training programs) and work are those that are impacted at the final stages of the addiction recovery process. Finally, it is important to emphasise that while these are the dimensions of analysis identified during the evaluation design phase, dimensions of analysis may evolve as the project advances.

Multidisciplinary

The IHD perspective calls for all branches of knowledge to be brought to bear in addressing poverty and ecological degradation. The interconnected nature of these problems demands actions that transcend traditional disciplinary boundaries, integrate fields of study and foster the sharing of knowledge. So too, an integral impact evaluation may employ methods and gain insight from multiple disciplines. Rigorous quantitative techniques employed in the field of economics may be combined with case studies or qualitative analysis more commonly employed in sociology or anthropology. Depending on the type of intervention being evaluated, the researcher may benefit from drawing on theories and empirical methods of analysis from engineering (say, for the evaluation of a housing programme) or medicine (for a programme that provides access to drinking water).

The nature of the Hogar de Cristo programme precludes the possibility of constructing a control group. The community centres work with anyone within the community in situation of social vulnerability and experiencing problematic consumption of psychoactive substances and they could not be expected to refuse or delay treatment. Non-experimental methods for constructing a control group, such as propensity score matching, are also not an option because of the scarcity of comparable data on persons with addictions who do not receive treatment or participate in alternative addiction recovery programmes. Moreover, as each centre is rooted within the community, some aspects of the programme vary across different locations and contexts.

The evaluation method will consist of the combination of three strategies. Firstly, we will work with the organisation to develop a monitoring system that regularly collects data on the programme's activities and participants in each of the selected capability dimensions. This data will be used to study the socioeconomic characteristics of the participants, measure the change over time in selected outcome indicators and analyse the diversity of timing and stages of the recovery process. Secondly, qualitative data collected through semi-structured interviews with programme participants will be the principal source of data for evaluating the causal effects of the programme. Following key elements of the QuIP evaluation method (Copestake et al. 2019a), the interviews will focus on the participants' life trajectories and on identifying the perceived causes of changes in the selected dimensions, both programme-related aspects and other factors. So as to reduce confirmation bias (i.e. that participants respond what they believe the interviewer wants to hear), no explicit reference to programme evaluation will be made when explaining to participants the objectives of the research project.¹³ We will also draw on techniques traditionally used in qualitative methods such as theoretical sampling and conducting a sufficient number of interviews so as to allow one to reach saturation points and explore a diversity of subjects and experiences. The main focus will be on understanding how and why changes occur and identifying both recurrent categories and a range of trajectories. Thirdly, geographically referenced data on drug use and addiction, the location of treatment services and neighbourhood characteristics will be used to conduct spatial analysis of the relationship between addiction, treatment, and positive and negative neighbourhood attributes. Experts from multiple disciplines (in particular, economics, sociology, development studies, medicine and psychology) will participate in project design and fieldwork.

Diverse pathways

The IHD principle of 'each person as an end' implies that an integral impact evaluation will have to be concerned not only with the measurement of average treatment effects, but, more importantly, with understanding the diversity of experiences. Each person's experience of a social programme is unique and therefore each will have different possibilities for attaining different outcomes. As highlighted in 'Integral human development' section, this is what the capability approach literature refers to as 'conversion factors'. In the absence of childcare, a young mother with an addiction will not be able to 'convert' a group therapy session into progress towards recovery in the same way as a woman without children. The focus of an integral impact evaluation is not only on quantifying treatment effects or identifying key programme elements and participant characteristics that help produce outcomes, but also on documenting and understanding the diversity of pathways.

The fact that the Hogar de Cristo tailors its activities to the specific needs and possibilities of each person and does not have strict rules for exiting the programme (e.g. the organisation does not demand abstinence as a pre-condition for participation) makes it particularly important for each person's unique characteristics and experiences to be taken into account when evaluating the ways in which the programme is more or less effective. Although the Hogar de Cristo's ultimate goal is full recovery from addiction and reintegration into the community, a single metric (such as length of abstinence) would not adequately capture the programme's impact. From an integral impact evaluation perspective, the provision of a safe and supportive home to a participant's young children or the offer of love and care to a person in her final days of life could be considered to be programme contributions, even in cases in which a person ultimately dies from her addiction. What is important is to gain greater understanding of the diverse processes of recovery and how the organisation aids or hinders these processes. Although in the interviews with participants special attention will be given to the testimonies concerning the capability dimensions selected during the evaluation design phase, openness to unexpected pieces of information, perspectives and interpretations of reality will be essential to understanding the diversity of experiences.

Relations to self, others and the territory

As discussed in the 'Integral human development' section, relationships are a fundamental component of the spiritual dimension of human life. It is manifested, in particular, through relations to self, others and the environment, and how people choose to live out these relations, whether they choose to relate in a way that allows themselves and others to flourish and the natural or built-in environment to be protected and cared for. An integral impact evaluation will have to give an account of the quality of these relations and how they have changed as a result of participation in a social programme and an account of how people have used their freedom and chosen to act within these relations.

In the interviews with participants in the Hogar de Cristo, open-ended questions will be used to elicit information on the changes participants have experienced in their relationship with themselves (e.g. self-forgiveness, feelings of acceptance, greater self-knowledge or self-confidence), with families and the community (e.g. integration into the neighbourhood centre community, contacts and time spent with family members, ability to care for children, ability to communicate and share feelings, development of habits of coexistence and positive relationships within the community) and with the physical environment (e.g. greater respect for the property of others and green spaces such as discarding of needles or other residual waste of addiction in safe bins and not consuming drugs on children's playgrounds), and on the reasons behind these changes. We will also seek information about the participants' perceptions of how these changes are connected to faith and their relationship with God.

Incorporating a spiritual dimension into impact evaluation also implies accounting for the mutual transformations that might emerge from the relationships between providers and participants. In interviews, leaders of the Hogar de Cristo spoke of how their own lives had been transformed by the shared life experiences within the community. One person explained 'our spirituality means seeing each person as a gift and allowing ourselves to be challenged and transformed by their lives'. In this sense, an integral impact evaluation blurs the traditional distinction between 'provider' and 'beneficiary' as all members of the community become mutual givers and receivers. We will aim to capture evidence on this through interviews with programme leaders during the initial phase of field research.

Dialogue and personal-communal change within context

From an IHD perspective, interaction with others and dialogue is a privileged place for transformation to take place, at both the personal and community levels. An integral impact evaluation will therefore need to assess the extent to which an intervention has fostered the creation of spaces for dialogue and interaction, and how it has impacted the community and the wider context in which people relate and make choices about their lives, such as the choice to consume drugs.

In the case of the evaluation of the Hogar de Cristo, the qualitative analysis will focus on understanding not only how the intervention impacts the lives of people with addictions, but also the effects on the conditions that lead to addiction. Relevant questions include: How does the Hogar de Cristo's presence in the neighbourhood and prevention activities contribute to reducing drug consumption in the community or change the local residents' sense of self-worth and confidence in themselves? How does the creation of spaces for dialogue (sessions with psychologists and social workers, group meetings, workshops) become the locus for personal change and generate changes in the five dimensions described in point (1)? The spatial analysis of the relationship between drug use and access to treatment will provide additional insight into the effects at the community level.

 

Conclusion

Over the last two decades, there has been an increased interest in development research and practice in the role of religion in shaping development outcomes. This article has sought to contribute to that literature by exploring how to incorporate a spiritual dimension into the evaluation of social programmes. We have done so by employing the concept of integral human development. We argued how Amartya Sen's capability approach could serve to clarify and operationalise the concept, and we used a faith-based drug addiction recovery programme in Argentina as an example of how a spiritual dimension could be incorporated into impact evaluation. The proposed framework for integral impact evaluation has five distinctive characteristics. Firstly, it measures the effects of an intervention on multiple dimensions of life and examines the interconnection between dimensions. Secondly, it integrates knowledge and methods of analysis from multiple disciplines. Thirdly, it seeks to understand the diverse pathways that lead to programme experiences and outcomes. Fourthly, it argues that the spiritual dimension of life can be evaluated by considering changes in relationships with oneself, others and the territory, and by examining how people choose to live out these relations. Fifthly, it is concerned with transformations that occur at both the personal and community levels, changes that emerge through the creation of opportunities and space for dialogue.

In the coming months, we will use the integral impact evaluation framework outlined here to conduct an evaluation of the Hogar de Cristo's programmes. We hope that the proposed framework could be used by other organisations too, whether faith-based or not, to evaluate the impact of their work in a way that better reflects the idea that no human is fully human without this openness to something beyond himself or herself, whether manifested in the wonder of a beautiful landscape or the mystery of a loved one's face.

 

Acknowledgement

The authors would like to thank James Copestake and Jimena Macció for comments on an earlier draft and the staff of Hogar de Cristo for their generosity and guidance.

Competing interest

The authors have declared that no competing interests exist.

S.D. focused on the conceptual components of the article and A.M. focused on the research on impact evaluation and on the faith-based organisation, Hogar de Cristo.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Data availability statement

Data sharing is not applicable to this article as no new data were created or analysed in this study.

Disclaimer

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.

Ethical consideration

The authors carried out interviews only with organisation leaders and therefore an ethical clearance was not required.

 

References

Alan, J.R., Bell, D.C. & Carlson, J.W., 2000, 'Individual religiosity, moral community and drug user treatment', Journal of the Scientific Study of Religion 39(2), 240-246. https://doi.org/10.1111/0021-8294.00019        [ Links ]

Alkire, S., 2002, 'Dimensions of human development', World Development 30(2), 181-205. https://doi.org/10.1016/S0305-750X(01)00109-7        [ Links ]

Alkire, S., Qizilbash, M. & Comim, F. (eds.), 2008, The capability approach: Concepts, measures and applications, Cambridge University Press, Cambridge.         [ Links ]

Alkire, S., Foster, J., Seth, S., Santos, M.E., Roche, J.M. & Ballon, P., 2015, Multidimensional poverty analysis and measurement, Oxford University Press, Oxford.         [ Links ]

Balbach, E., 1999, Using case studies to do program evaluation, California Department of Health Services, Sacramento, CA.         [ Links ]

Barrett, C. & Carter, M., 2010, 'The power and pitfalls of experiments in development economics', Applied Economic Perspectives and Policy 32(4), 515-548. https://doi.org/10.1093/aepp/ppq023        [ Links ]

Bryan, G., Choi, J. & Karlan, D., 2018, Randomizing religion. The impact of Protestant evangelization on economic outcomes, NBER Working paper 24278, National Bureau of Economic Research, Washington, DC, viewed n.d. from https://www.nber.org/papers/w24278.         [ Links ]

Copestake, J., Morsink, M. & Remnant, F., 2019, Attributing development impact, Practical Action, Warwickshire, Rugby.         [ Links ]

Copestake, J., James, M., Morsink, M. & Flowers, C., 2019a, 'Faith-based rural poverty reduction in Uganda', in J. Copestake, M. Morsink & F. Remnant (eds.), Attributing development impact, pp. 117-139, Practical Action, Rugby.         [ Links ]

Deaton, A. & Cartwright, N., 2018, 'Understanding and misunderstanding randomized controlled trials', Social Science & Medicine 210(August 2018), 2-21. https://doi.org/10.1016/j.socscimed.2017.12.005        [ Links ]

Deneulin, S., 2018, 'Integral human development through the lens of Amartya Sen's capability approach', Working Paper 427, Kellogg Institute for International Studies, University of Notre Dame, Indiana.         [ Links ]

Deneulin, S. & Zampini-Davies, A., 2017, 'Engaging development with religion', World Development 99(November), 110-121. https://doi.org/10.1016/j.worlddev.2017.07.014        [ Links ]

Duflo, E., Glennerster, R. & Kremer, M., 2008, 'Using randomization in development economics: A toolkit', in T. Schultz & J. Strauss (eds.), Handbook of development economics, pp. 3895-3962, vol. 4, North Holland, Amsterdam and New York.         [ Links ]

Epele, M.E., 2011, 'New toxics, new poverty: A social understanding of the freebase Cocaine/Paco in Buenos Aires, Argentina', Substance Use & Misuse 46(12), 1468-1477. https://doi.org/10.3109/10826084.2011.576745        [ Links ]

Gertler, P., Martinez, J., Premand, P., Rawlings, L.B. & Vermeersch, C., 2016, Impact evaluation in practice, 2nd edn., Inter-American Development Bank and World Bank, Washington, DC.         [ Links ]

Heinrich, G., Leege, D. & Miller, C., 2008, A user's guide to integral human development, Catholic Relief Services, Baltimore, ME.         [ Links ]

Keleher, L., 2018, 'Integral human development', in L. Keleher & J. Drydyk (eds.), Routledge handbook of development ethics, Routledge, London.         [ Links ]

Kelly, J.F., Magill, M. & Stout, R.L., 2009, 'How do people recover from alcohol dependence?', Addiction Research and Theory 17(3), 236-259. https://doi.org/10.1080/16066350902770458        [ Links ]

Le Roux, E., Hankela, E. & McDonald, Z., 2018, 'Social justice required: Youth at the margins, churches and social cohesion in South Africa', HTS Tehologiese Studies/Theological Studies 74(3), e1-e8. https://doi.org/10.4102/hts.v74i3.5046        [ Links ]

Lilja, N. & Bellon, M., 2008, 'Some common questions about participatory research: A review of the literature', Development in Practice 18(4-5), 479-488. https://doi.org/10.1080/09614520802181210        [ Links ]

Machado, M.P., 2005, 'Substance abuse treatment, what do we know? An economist's perspective', The European Journal of Health Economics 6(1), 53-64. https://doi.org/10.1007/s10198-004-0253-2        [ Links ]

Mitchell, A. & Maccio, J., 2018, 'Evaluating the effects of housing interventions on multidimensional poverty: The case of TECHO-Argentina', OPHI Working Paper 120, University of Oxford, viewed 11 March 2019, from https://www.ophi.org.uk/wp-content/uploads/OPHIWP120.pdf        [ Links ]

Nussbaum, M., 2011, Creating capabilities, Harvard University Press, Cambridge.         [ Links ]

O'Neill, D., 2017, 'What works? Evidence on the role of faith in poverty reduction', Christian Journal of Global Health 4(1), 55-59. https://doi.org/10.15566/cjgh.v4i1.158        [ Links ]

Organización de Estados Americanos, 2014, Consumo de pasta base de cocaína en América del Sur. Comisión Interamericana para el Control del Abuso de Drogas, Observatorio Interamericano de Drogas, Montevideo, Uruguay.         [ Links ]

Olivier, J., 2016, 'Hoist by our own petard: Backing slowly out of religion and development advocacy', HTS Theologiese Studies/Theological Studies 72(4), a3564. https://doi.org/10.4102/hts.v72i4.3564        [ Links ]

Pawson, R. & Tilley, N., 1997, Realistic evaluation, Sage Publications, London.         [ Links ]

Pope Benedict XVI, 2009, Caritas in Veritate: On integral human development, viewed 11 March 2019, from http://w2.vatican.va/content/benedict-xvi/en/encyclicals/documents/hf_ben-xvi_enc_20090629_caritas-in-veritate.html.         [ Links ]

Pope Francis, 2015, Laudato Si': On care for our common home, viewed 11 March 2019, from https://w2.vatican.va/content/francesco/en/encyclicals/documents/papa-francesco_20150524_enciclica-laudato-si.html.         [ Links ]

Pope John Paul II, 1987, Sollicitudo Rei Socialis: On the twentieth anniversary of Populorum Progressio, viewed 11 March 2019, from http://w2.vatican.va/content/john-paul-ii/en/encyclicals/documents/hf_jp-ii_enc_30121987_sollicitudo-rei-socialis.html.         [ Links ]

Pope Paul VI, 1967, Populorum Progressio: On the development of peoples, viewed 11 March 2019, from http://w2.vatican.va/content/paul-vi/en/encyclicals/documents/hf_p-vi_enc_26031967_populorum.html.         [ Links ]

Rival, J.M. & Salvia, A., 2016, 'Estudio diagnóstico sobre las condiciones de vida, consumos problemáticos y seguridad ciudadana de jóvenes en villas y asentamientos del Conurbano Bonaerense', Working paper, Observatorio de la Deuda Social Argentina, Buenos Aires.         [ Links ]

Robeyns, I., 2017, Wellbeing, freedom and social justice, Open Book Publishers, Cambridge.         [ Links ]

SEDRONAR, 2017, Estudio Nacional en población de 12 a 65 años, sobre Consumo de Sustancias Psicoactivas. Informe de Resultados N° 1, Buenos Aires.         [ Links ]

Sen, A., 1988, 'The concept of development', in H. Chenery & T.N. Srinivasan (eds.), Handbook of development economics, pp. 9-25, Elsevier, Amsterdam.         [ Links ]

Sen, A., 1999, Development as freedom, Oxford University Press, Oxford.         [ Links ]

Sen, A. 2009, The idea of justice, Allen Lane, London.         [ Links ]

Sen, A., 2017, Collective choice and social welfare, Allen Lane, London.         [ Links ]

Swart, I. & Nell, E., 2016, 'Religion and development: The rise of a bibliography', HTS Teologiese Studies / Theological Studies 72(4), a3862. https://doi.org/10.4102/hts.v72i4.3862.         [ Links ]

Tearfund, 2016, An introductory guide to the The LIGHT Wheel toolkit, viewed October 2018, from https://learn.tearfund.org/en/resources/impact_and_effectiveness/the_light_wheel/.         [ Links ]

Tomalin, E., 2013, Religions and development, Routledge, London.         [ Links ]

Ullmann, H., 2015, 'La salud y las juventudes latinoamericanas y caribeñas', in D. Trucco & H. Ullmann (eds.), Juventud: Realidades y retos, CEPAL, Santiago.         [ Links ]

United Nations Development Programme (UNDP), 2016, Human Development Report 2016: Human development for everyone, viewed 11 March 2019, from http://hdr.undp.org/en/content/human-development-report-2016-human-development-everyone        [ Links ]

White, H., 2014, 'Current challenges in impact evaluation', European Journal of Development Research 26(1), 18-30. https://doi.org/10.1057/ejdr.2013.45        [ Links ]

White, W., 2009, 'The mobilization of community resources to support long-term addiction recovery', Journal of Substance Abuse Treatment 36(2),146-158. https://doi.org/10.1016/j.jsat.2008.10.006        [ Links ]

White, W. & Whiters, D., 2005, 'Faith-based recovery', Counselor 6(5), 58-62.         [ Links ]

World Vision International, 2007, Learning through evaluation with accountability & planning, Washington, DC, viewed October 2018, from https://www.wvi.org/sites/default/files/LEAP%202nd%20Edition%20with%20Update.pdf.         [ Links ]

 

 

Correspondence:
Severine Deneulin
sd251@bath.ac.uk

Received: 01 Apr. 2019
Accepted: 06 June 2019
Published: 14 Aug. 2019

 

 

1. Jorge is a pseudonym.
2. Paco, short for pasta de cocaina, is obtained from the residuals of the chemical process of cocaine production. It is the cheapest drug on the market but has the most immediate, debilitating and long-term health effects.
3. The work is intended to start in June 2019.
4. For a review of the literature, see Deneulin and Zampini (2017), Olivier (2016), Swart and Nell (2016), Tomalin (2013).
5. See Deneulin (2018) and Keleher (2018) for further discussions on the concept.
6. Documents of the Catholic social tradition are referenced by the initials of their title followed by the paragraph.
7. See Robeyns (2017), Alkire (2002) and Alkire et al. (2015) on the process of selecting capability dimensions.
8. A list of alternative approaches is available at https://www.betterevaluation.org/en/approaches.
9. For a combination of the 'Light Wheel' and the Qualitative Impact Protocol, see Copestake et al. (2019).
10. See https://jliflc.com.
11. The information presented in this section is based on project development fieldwork as well as multiple organisation documents available at https://hogardecristo.org.ar/.
12. For a discussion on how to evaluate impact multidimensionally, see Mitchell and Maccio (2018).
13. Although this point may raise concerns about whether it is ethical to not fully disclose to participants the purpose of the investigation, it is necessary to obtain more credible information. At a later stage in the project we plan to carry out participatory activities which will give participants the opportunity to share their opinions about the programme and the research findings.