On-line version ISSN 0038-2337
S. Afr. j. occup. ther. vol.44 n.1 Pretoria Jan. 2014
B Occ Ther (UP), B Occ Ther (Hons) (Medunsa), PG Dip Vocational Rehabilitation (UP), Dip Higher Education and Training Practices (UP), M Occ Ther (UP), PhD (UP); Associate Professor, Department of Occupational Therapy School of Therapeutic Sciences, University of the Witwatersrand
Many occupational therapists in South Africa and the United Kingdom are using the Vona du Toit Model of Creative Ability with its associated assessments and outcome measures in their practice settings. Although there appears to be strong clinical confidence in the use of these instruments that apply the levels of creative ability in the scoring system, little evidence to date has been published on the validity of the levels.
The aim of this study was to investigate three instruments based on the levels of creative ability: the Creative Participation Assessment (CPA), the Functional Levels Outcome Measure (FLOM) and the Activity Participation Outcome Measure (APOM), for evidence that the levels indeed represent increasing amounts of ability and that the scoring of the three instruments follow a linear or hierarchical pattern.
A secondary data analysis was done using the threshold ordering of the Rasch Measurement Model to indicate whether the levels of creative ability exist. Results showed that all three instruments indeed represent increasing amounts of ability in a person and that the levels of creative ability exist. Although these findings are significant, it is the first in a series of analyses and the remaining assumptions in the Rasch Measurement Model still need to be tested.
Key words: Vona du Toit Model of Creative Ability; outcome measures; Rasch Measurement Model; threshold ordering
All health care professionals are expected to demonstrate the effectiveness of their services, not only to their clients but also to the funders of the service and society at large. It is the responsibility of a profession to provide evidence of the noticeable value and quality of service delivery1. Demonstrating evidence of change after intervention requires valid and reliable measurement instruments. Measurement in occupational therapy gathers vital information about individuals, groups or populations. It is part of evaluation in the occupational therapy process (Occupational Therapy Practice Framework - OTPF II)2. During evaluation, clinicians make use of different measurement instruments and assessment tools to observe and measure the factors that support or prevent occupational performance in individuals. Some of these instruments have been developed before the awareness of the importance of validity and reliability properties.
Evaluation is core to the process of occupational therapy but not all variables are easily measured. Occupational therapists, more often than not, deal with abstract constructs when they attempt to understand individuals as occupational beings and what their needs and expectations are for occupational therapy services. Assessment tools aim to describe the influence of illness, injury, disability, poverty, delays in development and environmental factors on individuals' participation in activities and occupations. It is also done to predict performance, for example school readiness or return to work. There are also tools used for screening purposes and outcome measures to evaluate change after intervention. When considering the reasons why occupational therapists do assessments and what they have to measure, it is clear that they often have to measure latent variables, those things that are abstract and not linear in nature. Schofield3 and Laver-Fawcett4 further referred to the fact that defining and measuring outcomes are complicated by the dynamic and multifaceted nature of occupational performance as well as the complex and subjective experience of individual cases. This brings numerous challenges to accurate measurement.
Despite the complexities of the variables that occupational therapists need to measure, measurement principles still need to be applied in the use of assessment tools. Stevens' influential theory of scales of measurement becomes essential in the choice of tools. According to Stevens' theory, there are four levels of measurement: nominal (level one), ordinal (level 2), interval (level three) and ratio (level four) scaling5. The nominal level classifies data into categories, for example males or females. This level is not measurement per se. The second level is the ordinal scale with numbers assigned to observations in a hierarchical order. Lickert scales are examples of ordinal scaling. The third and fourth levels of scaling, interval and ratio scales, also provide a hierarchical order but the difference is that the distances between the points on the scale are equal e.g. the temperature points on a thermometer (level three) or the numbers on a ruler (level four).
Most of the assessments in occupational therapy are on an ordinal scale, which limits the interpretation of the data. Scores may not be summed and averaged as the distance between points are unequal, and mathematical calculations are not allowed on this level. It can only be used to describe observations. The misuse of ordinal raw scores is often seen in the calculation of changed scores, means and effect sizes. Grimby, Tennant and Tesio6 voiced their concerns about this malpractice and urged researchers to subject their assessments and outcome measures to Rasch analysis. This type of analysis has the unique ability to transform ordinal data (level two) to linear or interval type of data (level three and four), provided the data fit the Rasch model7,8. The constructs investigated by the Rasch model are often latent traits (not measured directly) and therefore this model falls in the family of latent trait models 9. The expectation of the Rasch model is that persons with higher ability will pass more of the difficult items in a test and those with lower ability will only pass easier items. The model puts the measures of persons and items on the same scale. Rasch, a Danish mathematician developed a formula to calculate the probability (or the odds ratio) of a person to pass a certain item. These odds ratios make up the scale for persons and items at the same time. The scale is arranged in equal intervals (according to the odds ratios in the responses of the sample) and is called the logit scale in Rasch terminology. The required response structure is a probabilistic Guttman pattern because persons with lower ability are expected to pass fewer items than those with higher ability10. Detailed descriptions of the Rasch process of analysis are available in the literature7,8,9,10.
Theoretical frameworks and outcome measures
Occupational therapists develop preferences for the theoretical frameworks and practice models that they use in the occupational therapy process. Examples include the Model of Human Occupation, the Canadian Model of Occupational Performance, the Person-environment-occupation model or the Kawa Model to name a few11. Each of these models often have a series of screening and assessment tools for instance self-report questionnaires, checklists or objective measures.
Many occupational therapy clinicians use the Vona du Toit Model of Creative Ability (VdTMoCA) with its different assessment and outcome measurement tools. The Creative Participation Assessment (CPA) is used to determine the level of participation in day-to-day living. It was developed by van der Reyden12 and its psychometric properties were investigated by Casteleijn13. The CPA showed good inter-rater reliability provided that the therapists had a good understanding of the underlying theoretical concepts of the VdTMoCA and characteristics of each level of creative ability. Content and construct validity were also found to be adequate13. De Witt compiled a grid to assess creative ability according to occupational performance areas, namely personal management, social participation, work, and leisure. Descriptions of the actions in each occupational performance area for each level are provided12. This method is used to identify the level of an individual's creative ability and does not have a specific scoring system but it allows the therapist to indicate the specific level of creative ability in each performance area that a person is currently able to achieve. The specific phase within each level (therapist-directed, patient-directed or transitional) is also indicated for each area of occupational performance12. This assessment has not been subjected to psychometric investigation.
Zietsman developed the Functional Levels Outcome Measure (FLOM) that is used to measure change in functioning after intervention and Casteleijn et al14 investigated the psychometric properties of this measure. Its psychometric properties have been investigated and the Rasch analyses showed that the FLOM has excellent internal construct validity with good item functioning. Inter-rater reliability was also good provided the therapists were familiar with the characteristics of the levels of creative ability14. This outcome measure has been successfully implemented in several long-term care centers in South Africa and is used by occupational therapy technicians/assistants14.
Another outcomes measure is the Activity Participation Outcome Measure (APOM) developed by Casteleijn that also tracks changes after intervention in mental healthcare settings. The APOM has demonstrated good inter-rater reliability with good construct and content validity15.
Vona du Toit contributed significantly to the establishment of a sound theoretical foundation for occupational therapy when she developed the Model of Creative Ability16,17. The main focus of the VdTMoCA is on participation in everyday activities16. A person needs motivation to participate: this motivation is the inner force that initiates or directs all behaviour and results in the creation of a tangible or intangible product17. Motivation governs action and the actions are the manifestations of motivation. Through observations of the actions of a person, one is able to infer the level of motivation17. Du Toit then developed a hierarchy of levels of motivation with corresponding levels of action. This resulted in nine consecutive levels of motivation and action, starting with Tone and ending with the Competitive Contribution level (Table I)16,17,19.
The ability to create is central to the philosophy of the VdT-MoCA. This ability to create is indicative of the level of psychical development and for this reason, the assumptions of developmental theories such as Piaget's stages of cognitive development are well aligned with Du Toit's thinking in the levels of creative ability.
Levels of creative ability are seen in a developing child where a new born baby functions on the level of tone. At the age of approximately five months, the baby moves to the level of self-differentiation demonstrating actions that are incidental. Explorative actions are then seen from the age of two years which is the level of self presentation. When a child is ready to enter school, they show characteristics (e.g. norm awareness) of the level of passive participation. During adolescence, imitative actions are observed which imply the level of imitative participation. Active participation is the next level where there are actions of original contributions with authentic actions. Competitive participation, contribution and competitive contribution are the last three levels of creative ability that Du Toit described17. These levels are found in people who contribute towards their communities and society at large and who have transcended the self to change circumstances for the better. A typical example of the highest level of creative ability namely competitive contribution is Nelson Mandela. His contributions to society are well known.
Grobler18 argued that the higher levels of creative ability may be used to identify the chasm between management and leadership skills. According to her study, people with leadership skills are more likely to be on the levels from competitive participation to competitive contribution.
Not all people have the capacity to develop to the highest level of creative ability. Each individual has a creative potential or capacity which is influenced by many factors for instance genetics, temperament, intellectual functioning, resilience, and environmental barriers to name a few16.
Children move through the levels of creative ability in a sequential manner as they develop while adults regress in a sequential pattern to lower levels due to challenges in their lives, (illness, injury, economic changes, and unemployment). When they overcome these challenges, they may progress sequentially to higher levels again.
Occupational therapists who use the VdTMoCA usually assess the level of creative ability, design and implement an intervention programme, and measure the change after intervention. During intervention, specific principles are used to facilitate growth in a person to progress to higher levels of creative ability12,16,17.
It is generally the first six levels that are seen in occupational therapy services. Persons achieving levels seven to nine usually possess the knowledge and skills to overcome everyday challenges, achieve the norms for participation and reach out to influence and contribute towards communities and populations19.
Assessment of creative ability
The different assessment tools and outcome measures mentioned above apply the levels of creative ability in the scoring of a person's ability to participate in everyday activities12,15,16,17.
Occupational therapists in South Africa and the United Kingdom believe in the levels of creative ability and find the VdTMoCA extremely useful as a practice model19. The fact that a clinician can describe participation in everyday activities or occupation in levels opens up possibilities of measuring baseline performance or functioning, tracking the progress of clients and calculating the effect of the service at the end of the intervention. The levels are thus useful in outcome measurement 15,19.
The levels of creative ability also assist clinicians in assessing and describing the nature of a client's participation and engagement in activities and occupationsl2,l6. Once the level has been determined, the intervention can be directed at the correct level and intensity. Specific intervention principles are available for each level 12,14,16,17.
The characteristics mentioned above make the VdTMoCA a popular practice model in SA and the UK l9. However, one basic important question has not been answered: do the levels of creative ability exist? What evidence do we have that there are different levels of creative ability and that they follow a linear and hierarchical pattern? Several assessment and outcome measures are being used by occupational therapists that are based on the levels of creative ability without evidence of these characteristics.
The aim of the study was to determine if the levels of creative ability as measured by three different measurement instruments, follow a linear or hierarchical pattern.
A secondary data analysis was done using data that were previously collected by clinicians in South Africa using measurement instruments whose scoring systems are based on the levels of creative ability.
Data were collected with the three measurement instruments as part of the routine use of assessment or outcome measurement in the case of the FLOM and the APOM while the data for the CPA were collected as part of a research project in 2001 as well as routine assessments during 2007 and 2008. The three instruments are briefly described below.
The Creative Participation Assessment (CPA)12,13. This assessment uses a 7-point scale with l2 items. It covers levels from Tone to Competitive Participation and is a one-page table with 8 columns and l4 rows. The first column presents the l2 items namely Action, Volition, Handle tools and materials, Relate to people, Handle situations, Task concept, Product, Assistance/supervision required, Behaviour, Norm awareness, Anxiety and emotional responses, Initiative and effort. Columns 2 to 7 represent the levels of creative ability from Tone to Competitive Participation. There is a description of the expected response in each cell of the table e.g. the description for the item Action under the level of Tone reads undirected and unplanned. A therapist will select the description of the item that fits her client the best and mark or tick that cell. In the last row, the number of ticks for each level will be counted and written down. The level with the most number of ticks will be the level of that client12,l3.
The Functional Levels Outcome Measure (FLOM)l4. This outcome measure has l0 items and covers the first five levels: Tone to Imitative Participation. The items are Mental illness, Orientation, Self care, Appearance, Continency (of urine and faeces), Social behavior, Activity participation, Domestic skills, Responsibility and Employment potential. The FLOM is a 5-page document with descriptions (in the form of a question) for each level and each item. A therapist needs to answer yes or no to each description and the level with the most yes marks will be the level of the clientl4. The last page is a summary of the items and the score that the client obtained in the form of a line graph.
The Activity Participation Outcome Measure (APOM)'5- This outcome measure has eight domains namely Process skills, Communication /interaction skills, Life skills, Role performance, Balanced lifestyle, Motivation, Self-esteem and Affect. Several items (53 in total) represent each domain. It covers the first six levels from Tone to Active Participation. The scale has l8 points as it includes the phases within each level. For each item a description for the first six levels of creative ability are given and a therapist decides which description best fits her patient. As soon as the level has been decided, the therapist will also determine the phase within the level, that is therapist-directed, patient-directed or transitional phase. These phases within the level are based on the amount of assistance a patient needs within the level. The APOM is a web-based application and therapists need to complete a one-day training workshop before they may use the APOMl5.
Table II describes the samples that were subjected to measurement by the three instruments.
The data collection using the CPA took place in a pubic mental health care facility. Two occupational therapists trained in the VdTMoCA and the use of the CPA assessed the level of 52 patients (males and females) diagnosed with schizophrenia. The average of the two assessors for the 12 items on the CPA were taken as the final score. These data were collected for a previous study in 2001 and were re-used in this study13. An additional 156 records from occupational therapists who used the CPA routinely to determine the level of creative ability in their clients between 2007 and 2008, were used. After a preliminary Rasch analysis was done on this sample, the logit scale in the threshold map was extremely compressed between -77 and 68 with a very high standard deviation. The logit scale is indicated by the odds ratio of a person to fall into a certain category. When the sample does not represent all the categories on the scale, the logits may be extremely compressed. Since the sample only included people with mental disorders, none of them obtained scores higher than five (imitative participation). Categories on level six (active participation) and level seven (competitive participation) were thus not represented. The author then collected data from people employed in the open labour market (without mental disorders) with successful participation in their respective occupations. Thirty four additional assessments were completed.
The collection of data using the FLOM 14 was done as routine assessments in a public-private partnership institution that provides long-term care for mental healthcare users. The FLOM is routinely completed for each patient on a monthly basis to track progress and enable reports to be written for the Review Board as stipulated in the Mental Health Care Act of 2002. Five hundred and twenty FLOMs were completed between January and March 2013. A selection of 306 FLOMs was used for this study. Incomplete assessments (all items not assessed) were discarded. These FLOMs were completed by six occupational therapy technicians and assistants trained in the use of the FLOM who use it as routine assessments14.
The data collection for the APOM15 also occurred during routine outcome measurement in different public and private mental health care institutions and clinics that provide care of acute, sub-acute and long-term adult mental healthcare users. All diagnostic groups were included. Occupational therapists trained in the use of the APOM completed the assessments between August 2012 and February 2013.
Permission to use the secondary data was obtained from the relevant authorities with three separate ethical clearance numbers.
Preparation of the data for analysis
The raw data of each instrument were recorded on an excel spreadsheet. The data from the CPA were at the level of creative ability as assessed by occupational therapists. The data of the FLOM and the APOM were the baseline assessments (before intervention commenced). This article only reports on the threshold ordering of the categories of the instruments and therefore no personal factors (e.g. gender, age, condition) were included for the analysis, only their score (level of creative ability) was used.
The data on the excel spreadsheet were entered into the RUMM2030 system, a software package for Rasch analysis.
The Rasch Measurement Model6,7,8,9 was used to test the hierarchical nature of the levels of creative ability as scored with the three instruments described above. This model has the ability to convert ordinal type of measurement into interval (or linear) measures. Since the levels of creative ability are ordinal measurements, the Rasch Model was the obvious choice.
The aim of this investigation was to discover if the points on the rating scale indicate an increasing amount of the ability of a person to participate in everyday activities. When scales measure the abilities of people, they should clearly indicate that people with higher abilities would pass more items on the scale than those with less ability. This measurement is only possible if the categories or points in the scale are ordered from low to high. The thresholds are those distances between two points on a scale. When the thresholds of an instrument are ordered according to the Rasch Model, each point or category is represented on a line as indicated in an example analysis in Figure 1a and 1b. Figure 1a indicates that only three items were ordered on a scale of 1 - 7. The analysis indicated that the rest of the items could not be measured on a 1 - 7 point scale. When this happens, one has the option to collapse points or categories (e.g. collapse point 2 and 3 as one category) as indicated in figure 1b. When the scale was redesigned with only three thresholds for most items, and in some items like bladder and bowel and stairs, only two thresholds, all items appeared linear. With the example below, it became clear that certain items like eating, grooming, bathing etc. cannot be measured on a scale of 1 to 7 as originally designed but that a 1 to 3 scale is a more valid scale to measure a person's ability for specific items. This process is called threshold ordering in the Rasch Model7.
Thresholds ordering or scalability was done with data collected with the CPA, FLOM and APOM using the RUMM2030 software
Figure 2 shows that the levels of the CPA are ordered in a linear fashion. No collapsing of items was necessary. The logit scale ranges between -15 and 15.
Figure 3 shows similar results for the FLOM and threshold are all ordered. The logit scale ranges between -8 and 11.
Figure 4 also indicates that the thresholds in the APOM are ordered. The logit scale ranges from -14 to 11.
All the scales showed perfect threshold ordering of categories of the scales, indicating that the levels of creative ability indeed exist.
In all three of the instruments, no collapsing of categories was necessary, showing that the categories of all the instruments resemble a linear or hierarchical arrangement of scores. The logit scale in all three instruments displayed a relatively lengthy scale. This could be that the majority of the sample is arranged between the levels of self-presentation (level 3) and passive participation (level 4). With less representation on the higher ends of the scale, it is expected that a longer logit scale will be present.
The categories or levels of creative ability in three different instruments all showed hierarchical ordering. This indicates that the levels are measuring increasing amounts of the construct under measurement, in this case creative ability. These findings confirm the clinical belief of occupational therapists that the levels of creative ability indeed exist.
The ordered thresholds hold great significance for occupational therapists using any of the three instruments. The significance lies in the fact that mathematical calculations may be done: for instance scores of the individual items of the instruments may be summed to obtain a total score, means and averages may be calculated as well as effect sizes (difference between final and baseline score). However, this significance must be viewed in light of the rest of the requirements of the Rasch Model. Threshold ordering is but one of 10 requirements8 before an instrument may be used as a measure with linear or interval scores. The remaining analyses such as item functioning, local dependency between items, unidimensionality and item-person fit still need to be reported on these three instruments.
The APOM is the only instrument that integrates the phases of each level (therapist-directed, patient-directed and transitional) in the scoring. It results in a lengthy scale with 18 categories but becomes essential when small changes in activity participation need to be detected. Even with this lengthy scale, thresholds were still ordered. It reiterates the fact that the levels of creative ability are accurate and valid and resemble linear measurement.
Although the three different instruments use the levels of creative ability as the basis for the scoring, each tool uses different terminology. The CPA uses the term Creative Participation, the FLOM uses Functional Levels and the APOM Activity Participation. The reason for the different terminology could be the different interpretations of the basic assumptions of the VdTMoCA but also shows the flexibility of the model. Different items are included in the three instruments, which allows for variety but the scoring seems to be accurate.
Only the first six levels have been tested, as these are the levels that are most commonly seen in clients seeking occupational therapy services. Data for the higher levels were not available and currently no assessments for the levels above active participation have been described or published to date.
The analysis of threshold ordering that was done is only the first of several assumptions to be tested before one can claim that the instruments are true measures with interval characteristics. The rest of the analysis of the three instruments are under investigation and will be reported in the near future.
There is sound evidence that the levels of creative ability exist and that clinicians can use the assessments and outcome measures in the VdTMoCA with confidence and assurance that it gives a valid indication of the limitations of participation in everyday activities.
1. Baum C, Christiansen C. Outcomes: The result of interventions in occupational therapy practice. In: Christiansen C, Baum C, BassHaugen J, editors. Occupational Therapy: Performance, Participation and Well-being. 3rd ed. New York: Slack Incorporated, 2005: 523-534. [ Links ]
2. American Occupational Therapy Association. Occupational Therapy Practice Framework: Domain and Process (2nd ed). American Journal of Occupational Therapy, 2008; 62: 625 - 683. [ Links ]
3. Schofield P Measuring outcome in psychiatric rehabilitation. British Journal of Occupational Therapy, 2006; 69(10): 481-83. [ Links ]
4. Laver-Fawcett A. Principles of Assessment and Outcome Measurement for Occupational Therapists and Physiotherapists. Theory, Skills and Application. Chichester: John Wiley, 2007. [ Links ]
5. Stevens SS. On the Theory of Scales of Measurement. Science. 1946; June: 677-680. [ Links ]
6. Grimby G, Tennant A, Tesio L. The use of raw scores from ordinal scales: time to end malpractice? Journal of Rehabilitation Medicine, 2012; 44: 97-98. [ Links ]
7. Bond TG, Fox CM. Applying the Rasch Model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum Associates, 2007. [ Links ]
8. Tennant A, Conaghan PG. The Rasch Measurement Model in Rheumatology: What is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Rheumatology, 2007; 57(8):1358-1362. [ Links ]
9. Iramaneerat C, Smith EV Jr, Smith RM An introduction to Rasch measurement. In: Osborne JW. Best Practices in Quantitative Methods. Los Angeles: Sage Publications, Inc., 2008: 50 - 69. [ Links ]
10. Hagquist C, Bruce M, Gustavsson JP Using the Rasch model in nursing research: An introduction and illustrative example. International Journal of Nursing Studies, 2009; 46: 380 - 393. [ Links ]
11. Turpin M, Iwama M. Using Occupational Therapy Models in Practice - A Field Guide. London: Churchill, Livingstone Elsevier, 2011. [ Links ]
12. De Witt P. Creative ability, a model for psychiatric occupational therapy. In: RB Crouch RB, Alers VM, editors. Occupational Therapy in Psychiatry and Mental Health. 4th edn. London: Whurr Publisher, 2005: 1 - 32. [ Links ]
13. Casteleijn,D, Smit C. The psychometric properties of the Creative Participation Assessment. South African Journal of Occupational Therapy, 2002; 32(1): 6-11. [ Links ]
14. Casteleijn D, Zietsman K, Crawford S, Grahan-Parker L, Karadimos L, Michell S. The psychometric properties of the Functional Levels Outcome Measure (FLOM): Construct validity, internal consistency and inter-rater reliability. Research project in partial fulfilment of the requirements for BSc IV OT (WITS). Unpublished document, 2013. [ Links ]
15. Casteleijn JMF. Development of an outcome measure for occupational therapists in mental health care practice. University of Pretoria, South Africa. Unpublished doctoral thesis, 2011. Available from: http://upetd.up.ac.za/thesis/available/etd-02102011-143303/ [ Links ]
16. Van der Reyden, D. Vona du Toit memorial lecture: Creative participation, 20 years later. South African Journal of Occupational Therapy, 1989; 19(1): 28-36. [ Links ]
17. Du Toit V Patient volition and action in occupational therapy. 3rd edn. Pretoria: Vona & Marie du Toit Foundation, 2004. [ Links ]
18. Grobler A. Growth in the higher levels of creative ability. Presentation delivered at the International Creative Ability Conference, London 21 May 2010. [ Links ]
19. Casteleijn D. The use of core concepts and terminology in South Africa. World Federation of Occupational Therapists Bulletin 2012; 65: 20 - 27. [ Links ]
This research was supported by the National Research Foundation Thuthuka Funding (Post PhD track), grant number TTK20II08I4000024469.