versão On-line ISSN 1012-277X
S. Afr. J. Ind. Eng. vol.23 no.1 Pretoria Jan. 2012
I.H. Abdul RazakI; S. KamaruddinII, *; I. Abdul AzidIII
IMechanical and Manufacturing Section Universiti Kuala Lumpur Malaysia, France Institute, Malaysia. firstname.lastname@example.org
IISchool of Mechanical Engineering Universiti Sains Malaysia, Engineering Campus, Malaysia. email@example.com
IIISchool of Mechanical Engineering Universiti Sains Malaysia, Engineering Campus, Malaysia. firstname.lastname@example.org
This paper provides an understanding of performance evaluation as a valuable tool in determining the strengths and weaknesses of a maintenance worker. The Maintenance Workforce Competency Model is introduced to embrace these key facets. In this paper, the development process of the model is presented as a tool to gauge or quantify the maintenance workforce's competency, by considering their intrinsic personal traits. The developed model aims to represent the workforce's competency level in carrying out their tasks, and whether or not they have performed their assigned tasks.
Hierdie artikel verskaf insig tot prestasie-evaluering as 'n nuttige metode om die sterk- en swakpunte van 'n instandhoudingswerker te bepaal. Die "Maintenance Workforce Competency Model" word aangewend om hierdie fasette te bepaal. Die proses wat gevolg is om die model te ontwikkel word voorgehou om sodoende die instandhoudingsarbeidsmag se bekwaamheid te kwantifiseer deur te kyk na die intrinsieke persoonlikheidseienskappe van die individue. Die model wat ontwikkel is poog om die arbeidsmag se vlak van bekwaamheid in die uitvoering van hulle pligte te bepaal en om vas te stel of hulle die aangewese pligte uitgevoer het al dan nie.
In the challenging and competitive industrial arena, companies strive for world-class competitiveness. Since maintenance makes up a large portion of total production cost - between 15 and 40 percent  - this also puts maintenance under increasing pressure to reduce the cost and waste of daily production. Apart from influencing production efficiency, on-time deliveries, capacity, and total plant cost effectiveness, maintenance has a major impact on product quality, which is dependent on equipment conditions. Thus maintenance is expected to make a long-term contribution to a company's profitability by intensifying production efficiency, extending equipment life, and improving equipment reliability and availability.
Maintenance, in general, can be described as the combination of a set of technical and administrative actions to retain or restore equipment or a system in a state to perform its designated functions [2, 3]. Maintenance is also seen as a system carried out in parallel with the production system. It also has a key role in achieving organisational goals and objectives. According to Duffuaa et al. , a maintenance system can be viewed as a simple input-output model. The inputs include labour, management, tools, spare parts, materials, equipment, etc. These desired resources should be optimised, thus maximising the output of the maintenance system and keeping the equipment reliable and well- configured, in order to achieve the planned operations of a plant.
In modern industries, in order to fulfil their operational goals, organisations have adopted a large amount of complex equipment. Extensive maintenance systems and management become more crucial. Widespread mechanisation and automation, for example, have reduced the number of production personnel and increased the capital investment in equipment and civil structures. As a result, the number of maintenance personnel and maintenance expenses in the total operational costs are escalating .
The roles and responsibilities of people in maintenance are not only limited to diagnosis and error recovery, but also include recording maintenance data, handling and operating equipment, training the maintenance personnel, and planning and scheduling maintenance. As people become involved in specific systems, their abilities and limitations are revealed in their performance of assigned tasks. Since they are essential to the operation of such systems, it is important to study the effects of human performance on the maintenance system.
The work presented in this paper demonstrates the importance of human involvement in the maintenance system, and the crucial need to assess and evaluate each maintenance worker's performance. The goal is to develop a model as a tool to gauge actual performance for effective assessment and evaluation of the maintenance worker.
2. THE NEED FOR HUMAN PERFORMANCE EVALUATION
It is well known that a successful maintenance system can be partly achieved through excellent equipment performance and reliability. However, the other fundamental factor is having skilful workers to operate the equipment and manage the overall maintenance system. The role of qualified technicians, for example, is essential in having proper maintenance. Their performance may, in fact, directly or indirectly influence the maintenance quality. This is acknowledged by Duffuaa et al. , who claim that much maintenance ineffectiveness can be traced back to the lack of skilled technical workers, resulting in various errors. According to Mason , human error in maintenance can have an impact on safety and overall performance in a number of ways. Poor repairs, for example, can increase the amount of breakdown, which in turn can increase the risk associated with equipment failure and personal accidents.
In addition, unlike production work that is more routine and does not require much information to be performed, maintenance work presents different levels of information processing, as well as problem-solving and decision-making. Because maintenance work is mostly non-repetitive and has more variability, establishing and sustaining a high level of technical employee performance is crucial, which means that the workers need to be trained. However, the question is: How much training and development is needed, and in what areas?
Owing to cost and time constraints, most companies cannot afford to send all their maintenance workers on formal training programmes to improve or enhance their performance level. As an alternative, training is only provided to the persons who most require specialised types of training. To that end, a properly developed and implemented maintenance competency assessment is a valuable tool in determining the strengths and weaknesses of individual maintenance workers in order to design an intensive training programme, and thus answer the research question.
Another problem most organisations encounter is that it is difficult to upgrade their maintenance workers' technical skills and knowledge, because many of the available training programmes are redundant or do not take the workers' current skill levels into consideration (Smith, in ). Furthermore, owing to the subjective nature of workers, there is less concern among researchers to discuss and develop the best assessment tool for measuring maintenance worker performance.
In the light of these problems, this paper will focus on efforts to eliminate them. The aim is to develop a model to help effectively assess and evaluate the performance of an individual maintenance worker. First, the issues of human performance and its measurement are described in the following subsections.
2.1 Human performance and human performance measurement
The importance of human involvement in the maintenance system, and in maintenance management, has been recognised in the past, especially since an aggressive approach, Total Productive Maintenance (TPM), was introduced in 1971 . In general, TPM is an improvement strategy that builds a close relationship between maintenance and production. It puts a strong emphasis on overall equipment operation and product quality with the active participation of every employee in the organisation. Besides developing a system of productive maintenance for the entire life of the equipment, this approach also focuses on the root causes of failure by taking advantage of the abilities and skills of all individuals in the organisation [8, 9].
Since then, the importance of human involvement in maintenance has been proven in many cases. This research is being studied in some depth in a number of industries, and it becomes increasingly clear that human factors in maintenance operations are of growing interest in most industries . With an increasing awareness that maintenance workers add value to the effectiveness of the maintenance system in any industry, more effort should be given to the development of these individuals, and the measurement of their performance becomes essential because, according to Kumar , it is difficult to plan, control, and improve human performance without any formal measurement. This was supported by Parida , who claimed that performance cannot be managed without measurement, because measurement indicates the present status of performance.
In fact, numerous predictive models that were developed for human performance measurement have been constructed and used to generate performance prediction . Performance measurement, whether using modelling methods or not, is applied to evaluate the actual human performance and to improve overall operational system performance.
The development and application of models for evaluating human performance, or Human Performance Models (HPMs), has been around for many years, using various techniques and purposes. Not only can HPMs represent either individual or aggregate human performance , they can also be used to symbolise how humans interact with the system. According to Young , HPMs can be used to support training, mission analysis, and simulation-based acquisition. In addition, Zinser & Henneman  agree that the modelling approach will contribute to a better understanding of human performance in the task being studied.
In the present work a modelling approach is chosen to develop a new model of human performance measurement, since it will contribute to a better understanding of human performance in maintenance work. The individual worker's competency, which includes their knowledge, skill, and attitude, is the main focus in this assessment. The primary target is to close or eliminate the workforce performance gap in the most cost-effective way. Thus the emphasis of this paper is on: (i) the introduction of the proposed Maintenance Workforce Competency Model, and (ii) the development process of this model.
3. THE WORKFORCE COMPETENCY MODEL: QUANTIFYING MAINTENANCE WORKFORCE PERFORMANCE
It is found that identifying, analysing, and evaluating the contributing factors or root causes that lead to inappropriate human actions have been the main concern in most literature . A good example is the Human Reliability Analysis (HRA) model, developed for nuclear power industries. However, very few studies have tried to quantify the effects of those factors on the performance of technical workers, especially for maintenance work.
As a first step in the present research, the 'Maintenance Workforce Competency Model' is developed by considering the same concept, which is identifying, analysing, evaluating the contributing factors, and quantifying the effects of those factors on the performance of maintenance workers. To be more specific, three main steps were proposed by Suwignjo et al.  for their Quantitative Model for Performance Measurement System (QMPMS), an initial reference towards developing the Maintenance Workforce Competency Model. The steps are illustrated in Figure 1.
In the development stage of the Maintenance Workforce Competency Model, data is one of the most important requirements for quantification purposes. Referring to the concept used for the model, subjective evaluation would be the primary source for gathering the required data; yet there is a need to quantify those subjective data. In this situation, expert judgment is the best approach to adopt. Thus, one of the expert judgment techniques that has similarities with the QMPMS framework, the Success Likelihood Index Methodology (SLIM), is adopted in developing the framework of the Maintenance Workforce Competency Model. SLIM is a widely-recognised HRA method that was originally developed for the nuclear power industry to quantify operator actions in the plant response model of a probabilistic risk assessment . It was used to quantify operator actions with the assumption that the human error rate in a particular situation depends on the combined effects of a relatively small set of performance shaping factors (PSFs) that influence the operators' ability to perform the action successfully. Figure 2 illustrates the general framework of SLIM.
The SLIM framework has six main stages (Figure 2). Park & Lee  divided these stages into three phases: Preliminary, Expert Judgment, and Calibration Parts for their newly proposed AHP-SLIM model. Adopting this idea, the Maintenance Workforce Competency Model has also been developed in three fundamental phases. However, in order to use this framework, a few modifications were carried out, because the main purpose of the application of the original SLIM method is to quantify the probability of an error associated with a task, sub-task, or task steps. The development of the Maintenance Workforce Competency Model, conversely, is to quantify the performance of an individual maintenance worker, but not only by focusing on human error. In response to the concepts identified above, the Maintenance Workforce Competency Model has been developed by integrating the QMPMS steps into the SLIM framework.
In addition to modifying the original SLIM framework, the Maintenance Workforce Competency Model was also extended to be applicable to more than one performance indicator. Evaluating the performance of maintenance workers should not be limited to only one indicator, because there are many people who have knowledge without skills . In addition, an attitude problem may also detract from a person's performance. For example, tardiness and poor attendance can negatively impact the individual's productivity and job satisfaction .
The structure of the proposed Maintenance Workforce Competency Model is shown in Figure 3. The purpose of this model is to evaluate a maintenance worker's capability in performing given tasks, while considering a number of performance indicators as the model's parameters. By using this model, a maintenance person's competency level, focused on their skills, knowledge, and attitude, can be gauged. The basic concept of the Maintenance Workforce Competency Model is that the competence of a person carrying out a given responsibility depends on the combined effects of a set of intrinsic personal traits or performance-shaping factors (PSFs) that influence a maintenance worker's ability to accomplish the tasks.
The model has three fundamental phases; Preliminary Part, Expert Judgment Part, and Quantification Part. The purpose of the Preliminary Part is to find and choose the most appropriate human performance indicators (PIs) as the model's parameters. The second phase, Expert Judgment Part, involves a number of people making subjective decisions in identifying a set of PSFs and providing numerical feedback on weighting (w) and rating (r) scores for the PSFs. The final phase, Quantification Part, involves mathematical formulation to quantify the competency level for each maintenance worker by multiplying the scores of the selected performance indicators (y). The result of the developed mathematical formulae is expected to represent the percentage of competency level for the individual worker (φ). A quantitative approach is adopted in this model in order to reduce the influence of personal judgmental to the minimum. The overall process in building up the Maintenance Workforce Competency Model, and the details of the development process for each of the phases, are discussed in the following sections.
3.1 Phase I: Preliminary Part
The Preliminary Part is the first phase of the Maintenance Workforce Competency Model. In this initial stage, the most appropriate human performance indicators (PIs) that can be used to gauge the performance of individual maintenance workers is identified. The PIs are proposed as the model parameters, incorporating a number of important aspects of human or workforce performance measurement in maintenance work areas. As mentioned in the previous section, the PIs also need to be linked to elements that relate to the total competency of a maintenance worker, including their skill, knowledge, and attitude, in order to compare their actual performance with the organisation's performance requirements.
In addition, a review of relevant literature and observation in a manufacturing company are carried out in order to ascertain that the selected PIs, which have rarely been discussed in the literature, are suitable for the real industrial application. As a guideline, Kumar  suggests that the SMART test, developed by the USA's Department of Energy in its handbook, can be used effectively to test the attributes of the indicators. According to the SMART test, the selected indicators should be:
- Specific: focused and intelligible definition to avoid misinterpretation. They must also be easily understood by persons who are involved in the evaluation.
- Measurable: can be quantified and compared with other data. These should allow for meaningful statistical analysis.
- Attainable: achievable, reasonable, and credible under expected conditions.
- Realistic: fit into the organisation's constraints and are cost-effective. In this study, the selected PIs should be related to the strategic goals and objectives of the organisation's maintenance management. It means that the organisation's expectations and needs from the maintenance department must be clearly identified.
- Timely effectiveness: this criterion is to balance the time taken to collect information with its usefulness. Wherever possible, the PIs should be based on information that is already available, and should be linked to existing data collection activities (e.g., the organisation's maintenance reporting system).
According to the SMART criteria, performance analysis (PA) is proposed as a tool to identify the most suitable PIs. In general, PA is used to figure out requirements and to identify the needs in order to help assessors find and do the right things in the right way. Therefore, for the purpose of evaluating the performance of the maintenance workforce, the PA should be carried out in a systematic way, aimed at thoroughly investigating the current organisation's overall performance requirement in maintenance management. From the analysis, the desired maintenance workforce performance will be identified, and the indicators that satisfy the SMART criteria will be determined. Thus in this phase it is proposed that organisational maintenance system analysis and situational analysis be carried out before the most appropriate performance indicator (PI) can be selected.
3.1.1 Organisational maintenance system analysis
An analysis of an organisation's maintenance system needs to be carried out in order to identify its stated maintenance policies, goals, and strategies. Apart from having a clearer understanding of the business's goals and objectives, its organisational strategic planning and performance standards (especially for the maintenance department) are also reviewed and analysed. This analysis may help to find indicators that satisfy the 'attainable' and 'realistic' criteria in SMART.
3.1.2 Situational analysis
A situational analysis is carried out to analyse the resources provided by the organisation in order to ascertain that their stated target maintenance policies, goals, and strategies can be achieved. This analysis should observe the current situation in the organisation, including how the maintenance system is operated and managed. For this purpose, interviewing the related personnel, evaluating the organisation's maintenance performance reporting system and standard operating procedure, and reviewing the published training materials are part of this analysis. It may help to find the indicators that are 'specific' and 'measurable', as well as 'timely effective' when the information, and the data available from the organisation's reporting system, are being referred to.
3.1.3 Performance indicator selection
Knowing the organisation's requirements for maintenance performance and the actual situation in the organisation, the critical performance of each person in the maintenance department can be investigated. From this critical performance analysis, the most appropriate performance indicators (PIs) that should be used to gauge the workers' performance are determined. It should be stressed here that the chosen PIs must be linked to the company's vision and goals and to individual performance standards. If this cannot be done, the measurement will not have the required attributes. The indicators are then proposed as the parameters for the Maintenance Workforce Competency Model that will be used in the second phase of the model's development - the Expert Judgment Part.
3.2 Phase II: Expert Judgment Part
The second phase of the Maintenance Workforce Competency Model (illustrated in Figure 3), the Expert Judgment Part, is a critical phase of the model, where the main content for assessment purposes is structured. The overall idea of this second phase involves a number of people making qualitative and quantitative decisions. According to the literature, a few expert judgment techniques have been developed and applied within many disciplines, and these are of primary interest when there is a limited availability of required data. Examples of fields that have contributed to probability elicitation by experts are decision analysis, psychology, risk analysis, Bayesian statistics, mathematics, and philosophy .
There are a few risks when employing the expert judgment techniques, such as inconsistencies of judgments and biases in making estimations. However, the primary advantage of using these techniques is that experts can expose the unique strengths and weaknesses of their own organisation. This is because the experts - who are usually the managers, engineers, or highly skilled workers - have the most knowledge and experience about the organisation's system and operation, and about its possible problems. Thus these people can provide a meaningful evaluation . Referring to Figure 3, the Expert Judgment Part consists of the following stages: PSFs Elicitation, PSFs Weights, PSFs Rating, and Indicator Scoring. However, before these phases are carried out, an expert selection procedure needs to be performed. Each of these stages will be elaborated below.
3.2.1 Expert selection
The selection and formation of a group of experts is a critical task that should be carried out structurally and in detail. It is because the development process of the Maintenance Workforce Competency Model - and the results it generates - are highly dependent on these personnel. 'Experts' are individuals who have adequate background and experience in the organisation's maintenance system and operation; who are regarded by others as those who are more knowledgeable about the maintenance section; and who are recognised by their peers and subordinates as qualified to address the technical problems. In order to find the experts who also have the ability to make the required evaluations, the following selection criteria are used (partly obtained from DiMittia et al. ):
- Actively participated in maintenance activities and acquired professional experience and knowledge in the maintenance field, with a minimum of five years' experience;
- Actively involved in the organisation's maintenance management as staff of the company, and possesses the best knowledge and understanding of the organisation's maintenance system;
- Familiar with the organisation's maintenance workforces;
- Capable of dedicating the required time to perform the evaluation, and committed to participate as required.
Based on these selection criteria, a questionnaire is developed. Apart from clearly stating the required professional background and experience, the questionnaires may also provide significant validation for the experts being selected. This selection process is vital in order to identify the best experts who fulfil all the stated selection criteria. Through this selection procedure, the experts' level of expertise in maintenance work, as well as their ability to provide the evaluation required by this study, can be measured.
On how many experts should participate, previous studies that utilise expert judgments, such as DiMittia et al.  and Park & Lee , do not state optimum numbers. Yet the selections are usually based on the number of experts available in the study scope. This is stated by Hokstad et al., , who claim that the quality of the experts in assessment and judgment is more important than the quantity. Acknowledging this proposition for the present research, the number of experts to be selected will not be stressed. But fulfilling all the stated criteria is vital.
Referring back to the Expert Judgment Part (Figure 3), the tasks to be performed by the experts - PSF elicitation, PSF weight, and PSF rating - are carried out by adopting a weighted scoring method. Also known as 'weighting and scoring', this is a form of multi- attribute or multi-criteria analysis. In general, the weighted scoring method involves identification of all the factors or attributes that are relevant to the project, the allocation of weights to compare each of the factors to reflect their relative importance, and the allocation of scores to each factor, which may be used to indicate performance of the items or elements to be evaluated . The important reason for choosing the weighted scoring method is that its basic steps are also used in the SLIM framework, which is the main reference for building the Maintenance Workforce Competency Model.
For the application of the weighted scoring method in this research, experts who have been selected at this stage (N persons) will quantitatively assess and evaluate the individual maintenance workers in the PSF Elicitation, PSFs Weight and PSF Rating stages, before quantifying the performance of those workers in terms of each indicator in the Indicator Scoring stage. All four stages are discussed below.
3.2.2 PSFs elicitation
Performance-shaping factor (PSF) elicitation is the first task to be performed by the experts (see Figure 3). As mentioned earlier, PSFs are parameters that influence the ability of a person to accomplish a given task successfully, or factors that may enhance or degrade the person's performance. The identified factors, in essence, might represent current or anticipated performance problems to be solved, or an opportunity for performance improvement.
PSF elicitation in the model development phase is the process of identifying and selecting those important factors that influence the maintenance workforce's performance. These factors are proposed as variables of the Maintenance Workforce Competency Model that examine maintenance worker in terms of each of the identified performance indicators (PIs). This means that a PI will contain one set of PSFs, which will be the same with another set of PSFs in other PIs. This is in order to narrow down the variables used for evaluation. Figure 4 shows the PSF elicitation process, which consists of collecting, screening, and ranking processes. It is carried out in an orderly fashion to ensure that the selected PSFs are completely appropriate as the variables for performance measurement.
As illustrated in Figure 4, the first draft of internal PSFs is constructed by collecting as many PSF items as possible from observation and various studies, as well as from relevant publications and books. The main references are previous studies that applied Human Reliability Analysis (HRA) methods, especially the work carried out by Kim & Jung  and DiMittia et al. . However, only internal or individual PSFs are considered in this research, with an assumption that any external factor (equipment condition, workload, etc.) and the stress factors are the same for every person. Internal PSFs are those that contribute to the personal traits, including the attributes, attitude, and abilities of a worker. These are appropriate to the purpose of the Maintenance Workforce Competency Model, which is to evaluate the performance of individual maintenance personnel.
The first draft of PSFs is then listed and screened by omitting those items that have little or no relation with maintenance tasks. Factors that have similar terms are also grouped together. This second draft of the PSF items (after the screening process) is then reviewed and commented on by the experts. From their feedback, questionnaires (with a five-point itemised rating scale) consisting of n PSF items that might possibly influence maintenance workforce performing their work are provided. (The itemised rating scale is one where a category of responses is offered, out of which the respondent picks the one that is most relevant for answering the question under consideration .)
The five-point itemised rating scale is chosen for the developed questionnaire to elicit the best PSFs for the following reasons:
- To ensure consistency and allow for easy completion and data coding
- To prevent the occurrence of central tendency error
- To provide standardised data for statistical analysis with a greater chance of being reliable.
A questionnaire form is developed for the N experts to rate the PSFs items' level of influence. A sample of the questionnaire form is given in Figure 5.
For each PSF item in the questionnaire form (Figure 5), experts are required to assign a rating by circling the score according to its level of influence, with the score of 5 indicating the highest influence on each of the identified Performance Indicators (PIs). Results from the Ν experts are then gathered to be summarised, and the mean value of each PSF (µPSFα) is calculated using equation (1):
where µPSFα is the mean value of one PSF item, G is the score given for the PSFa, and Ν is the number of experts. According to the µ value, the PSF items in each PI are then ranked and analysed using Pareto Analysis, resulting in a final set of the most relevant PSFs. Pareto Analysis is the basic tool in statistical analysis that represents data in an ordered and prioritised manner. By using discrete data categories, Pareto Analysis can also be used to distribute items from general to specific classifications. In the current stage of the model, the PSF items are ranked, using Pareto Analysis, according to the highest to the lowest given µ values. Pareto Analysis is carried out to select the top PSF items that contribute most to the performance of maintenance workers for each PI.
According to Park & Lee , the number of appropriate PSFs is around six, which is similar to the SLIM application. The final set of PSFs (n items) that have the highest µ value, also known as the factors that highly influence maintenance workforce performance, are then used in Phase II and Phase III in the Maintenance Workforce Competency Model (see Figure 3).
3.2.3 PSFs weight
The weight of a PSF is the relative importance or degree of influence of that PSF compared with the other PSFs. It relates to the degree to which a change in the numerical rating of the PSF scale might change the worker's ability to accomplish an action . Numerous tools and methods have been applied in various studies to assign the factor's weights, including the pairwise comparison tool, questionnaires, distributing points among the identified factors, and the analytical hierarchy process (AHP), a well-known multi-criteria decision-making tool .
At this stage of the Expert Judgment Part, the paired comparison (PC) method is applied to assign the weights of the PSFs relative to each other. As one of the tools for streamlined decision-making, the PC method is useful when there is no numerical data to work from. The basic principle of this method is a comparison of each option, one by one, in a rational and consistent way. The fact that people are generally better at making relative comparison rather than absolute judgments (Park 1987, in ) is a great advantage in adopting the PC method at this stage. This tool is also simple and fast to use, and suitable for industrial application. Weights for PSFs by applying the PC method are assigned by the steps shown in Figure 6.
For the first step of the PC method, a matrix box (shown in Figure 7, with separate matrix boxes for different performance indicators) is constructed and distributed to each expert for assigning the PSF weights. Each cell in the matrix represents the intersection (or pairing) of two items. For example, for six PSFs, the top box represents PSF 1 paired with PSF 2. For each pair, experts determine the more important PSF that has a higher influence on workforce performance, and write it in the cell, with the score of the one with relative importance written in brackets (as shown in the first cell in Figure 7).
The score of the item of relative importance is found by referring to arbitrary scales on a scale of 1 to 9 (Table 1) that are typically used in the AHP method , and whose validity and reliability have been well established. Saaty  stresses that the advantage of using AHP is that it enables people to provide accurate judgment. The scales also provide a useful mechanism for checking the consistency of the evaluation measures and alternatives suggested by the experts, thus reducing bias in their evaluations. This scale may help the experts to assign values for a comparative judgment of each pair of PSF. For example, if PSF 2 is strongly more important than PSF 1, a score of five is written in the top matrix box (in brackets), as shown in Figure 7.
The weight of each PSF (wPSFα) given by the experts is then computed by equation (2), epresenting the mean PSF weight value of the Ν experts.
where wis the weighted score given by each expert.
At the end of this stage, the level of importance of each PSF for every different PI is identified, demonstrating the specific factors that are most important or contribute highly to each performance indicator.
3.2.4 PSFs rating
Referring back to the Expert Judgment Part in Figure 3, the next stage to be performed by the experts after completing the PSF weights determination is the PSF rating. The rating of PSF is a measure of its quality . A rating system based on rating scales is often used in decision-making or in prioritising a set of quantitative alternatives. In general, three types of rating scales are commonly used in business research: the graphical rating scale, the itemised rating scale, and the Likert scale . The rating scale, however, must be agreed upon by the experts or the evaluators, who must also have a common understanding of what the high, medium, and low scores represent.
At the current stage, the performance of individual maintenance workers is evaluated by the experts by rating the identified PSFs in each performance indicator (PI). The itemised rating scale is chosen to develop the rating scale because there are various categories that need to be considered (a different category for each PSF), and the description of scales for a PSF category is different from another PSF. An example of one PSF rating scale (say, 'motivation' factor) is shown in Table 2.
PSF rating scales range from 1 to 5, with the value of 5 being the best performance. This has been developed as a guide for the experts, and the sample form for PSF rating elicitation is shown in Figure 8. This range of scales is selected for several reasons:
- It provides standardised data, and allows easy completion and data coding;
- It ensures consistency among the evaluators;
- It provides a balanced scale, with the middle value being moderate;
- A five-point scale is as good as any other: increasing from 5- to 7- or 9-point scales does not improve the reliability of the rating (Elmore & Beggs, 1975; in ).
The rating score of every PSF (rPSF ) for each maintenance worker being evaluated is then computed with equation (3), representing the mean rating score of Ν experts.
where r is the rating given by each expert. Using the weighting and rating values generated, the score of each performance indicator can be calculated by following the weighted scoring method (Section 3.2.1), described in the next section.
3.2.5 Indicator scoring
The final stage of the Expert Judgment Part of the Maintenance Workforce Competency Model (Figure 3) is the interpretation of the indicator score. At this stage, the actual performance of a maintenance worker for each indicator is calculated (in percentage form) using the results of the weighting and rating steps. For a selected performance indicator (γ), the weight of each PSF is normalised by dividing the weight (mean of Νexperts; WPSFα) by the sum of all PSF weights for that PI (as in equation (4)), where the total weight for each PI must be equal to one (Σwx = 1; i = 1, 2, η, η is the number of PSFs). The resulting quotient is termed the PSF η-weight (wx).
Again, for the same PI (γ1), the rating score (mean of Ν experts; rPSFa) given for each PSF is converted into a percentage (as in equation (5)) in order to yield a percentage output for the final result of the performance level. Referring to Eq. (5), the rating score, rPSFα, is divided by five, being the highest score for the developed rating scale (Figure 8).
At the end of this stage, the result of multiplying the η-weight (wx) over rating (rx) for the η PSF is then summed up, to yield the percentage of a worker's actual performance level for indicator γ1 as shown in (6).
The scores for the other performance indicators are also calculated by following the same procedure (Figure 3, the Maintenance Workforce Competency Model). The third or final phase of the model, the Quantification Part, is carried out after completing this stage.
3.4 Phase III: Quantification Part
The Quantification Part is the final phase of the Maintenance Workforce Competency Model (Figure 3). At this phase, the actual competency level (φ) of each maintenance worker is calculated by multiplying all the parameter scores obtained in the previous stage, as given by (7).
where γ represents the scores of each PI. It is hypothesised that the higher the φ value, the greater the competence of the maintenance worker. Results from the model analysis may help an organisation to develop a training plan to address the identified needs for each maintenance worker, design a curriculum to meet those training goals, and deliver training to the targeted group of workers, to enhance their competency level and thus to increase the organisation's targeted outcomes.
Once this final phase is completed, the actual level of competency of each maintenance worker can be quantified with a proven numerical value.
This paper presents the development process of the Maintenance Workforce Competency Model as a tool to quantify the individual maintenance worker's performance. The basic concept of the model is that the competency of a person performing a given task depends on the combined effects of a relatively small set of intrinsic personal traits, called performance shaping factors (PSFs), that influence the workforce's ability to perform their tasks. The Maintenance Workforce Competency Model has three fundamental phases: Preliminary Part, Expert Judgment Part, and Quantification Part. This model integrates qualitative and quantitative techniques to generate its output, which is the competency of individual maintenance workers expressed in numerical form. The quantification aspect is able to give an accurate and precise evaluation that also allows for validity and reliability analysis, increasing the model's robustness.
The developed model can also be considered as a modular framework that allows specific companies to decide any parameters or factors to be considered. Apart from providing a unique performance measurement tool that suits a company's specific needs, the purpose of performance evaluation also can be defined from various perspectives. Moreover, the model contains a number of decision support approaches as tools for accurate assessment and judgment. The tools, including performance analysis, SMART test, Pareto analysis, and the paired comparison matrix with AHP arbitrary scales and mathematical formulations, aid in data collection and analysis, and in support for quality decisions or judgments. In future, this research could aim to validate the developed Maintenance Workforce Competency Model in a real case study in order to test its credibility and plausibility.
The authors wish to thank Universiti Sains Malaysia and the Government of Malaysia for offering the grant that has made it possible for this research to be carried out.
 Al-Najjar, B. & Alsyouf, I. 2003. Selecting the most efficient maintenance approach using fuzzy multiple criteria decision making. International Journal Production Economics, 84, 85-100. [ Links ]
 Duffuaa, S.O., Raouf, A. & Chambell, J.D. 1999. Planning and control of maintenance systems. Canada: John Wiley & Sons, Inc. [ Links ]
 Dhillon, B.S. 2006. Maintainability, maintenance, and reliability for engineers. Boca Raton: Taylor & Francis Group. [ Links ]
 Dekker, R. 1996. Applications of maintenance optimization models: A review and analysis. Reliability Engineering and System Safety, 51(3), pp 229-240. [ Links ]
 Mason, S. 2000. Improving maintenance by reducing human error. Conference Transaction on Railtex - International railway technology conference. Birmingham: John Wiley & Sons, 57-62. [ Links ]
 Higgins, L.R. & Mobley, R.K. 2002. Maintenance engineering handbook. New York: McGraw- Hill. [ Links ]
 Rodrigues, M. & Hatakeyama, K. 2006. Analysis of the fall of TPM in companies. Journal of Material Processing Technology, 179, 276-279. [ Links ]
 Ben-Daya, M. 2000. You may need RCM to enhance TPM implementation. Journal of Quality in Maintenance Engineering, 6(2), 82-85. [ Links ]
 Cua, K.O., McKone, K.E. & Schroeder, R.G. 2001. Relationships between implementation of TQM, JIT, and TPM and manufacturing performance. Journal of Operations Management, 19, 675-694. [ Links ]
 Kumar, U. 2006. Development and implementation of maintenance performance measurement system: Issues and challenges. WCEM, 127, 1-6. [ Links ]
 Parida, A. 2007. Study and analysis of maintenance performance indicators (MPIs) for LKAB: A case study. Journal of Quality in Maintenance Engineering, 13(4), 325-337. [ Links ]
 Glenn, F., Neville, K., Stokes, J. & Ryder, J. 2004. Validation and calibration of human performance models to support simulation-based acquisition. IEEE Proceedings of the Simulation Conference, 2, 1533-1540. [ Links ]
 Young, M.J. 2003. Human performance model validation: One size does not fit all. Summer Computer Simulation Conference, 732-735. [ Links ]
 Zinser, K. & Henneman, R.L. 1988. Development and evaluation of a model of human performance in a large-scale system. IEEE Transaction on System, Man, and Cybernetics, 18(3), 367-375. [ Links ]
 Rademeyer, A., du Plessis, Y. & Kepner, C.H. 2009. Human performance variation analysis: A process for human performance problem solving. South African Journal of Human Resource Management, 7(1), 1-9. [ Links ]
 Suwingnjo, P., Bititci, U.S. & Carrie, A.S. 2000. Quantitative models for performance measurement system. International Journal of Production Economics, 64, 231-241. [ Links ]
 Chien, S.H., Dykes, A.A., Stetkar, J.W. & Bley, D.C. 1988. Quantification of human error rates using SLIM-based approach. IEEE Fourth Conference on Human Factors and Power Plants, June 1988, 297-302. [ Links ]
 Park, K.S. & Lee, J.I. 2008. A new method for estimating human probabilities: AHP-SLIM. Reliability Engineering and System Safety, 93, 578-587. [ Links ]
 Levitt, J. 1997. The handbook of maintenance management. New York: Industrial Press Inc. [ Links ]
 Foust, M.S., Elicker, J.D. & Levy P.E. 2006. Development and validation of a measure of an individual's lateness attitude. Journal of Vocational Behavior, 69, 119-133. [ Links ]
 Hora, S. & Jensen, M. 2002. Expert judgment elicitation. SSI Report, September 2002, 19, 1 13. [ Links ]
 Ha, J.S., Seong, P.H., Lee, M.S. & Hong, J.H. 2007. Development of human performance measures for human factors validation in the advanced MCR of APR-1400. IEEE Transactions on Nuclear Science, 54(6), 2687-2700. [ Links ]
 DiMittia, D.G., Khan, F.I. & Amyotte, P.R. 2005. Determination of human error probabilities for offshore platform musters. Journal of Loss Prevention in the Process Industries, 18, 488 501. [ Links ]
 Hokstad, P., Oien, K. & Reinertsen, R. 1998. Recommendations on the use of expert judgment in safety and reliability engineering studies. Two offshore case studies. Reliability Engineering and System Safety, 61, 65-76. [ Links ]
 Kim, J.W. & Jung, W. 2003. A taxonomy of performance influencing factors for human reliability analysis of emergency tasks. Journal of Loss Prevention in the Process Industries, 16, 479-495. [ Links ]
 Sekaran, U. 1992. Research methods for business: A skill building approach, 2nd ed. Canada: John Wiley & Sons, Inc. [ Links ]
 Saaty, T.L. 1980. The analytic hierarchy process: Planning, priority setting, resource allocation. Maidenhead: McGraw-Hill. [ Links ]
* Corresponding author.