COMMUNICATION

 

The Research Focus Question: Part 6: Finding the flaws, explaining the errors, and suggesting solutions

 

 

LM Sykes^I; WG Evans^II; PD Brand^III; AMP Harris^IV

^IBSc, BDS, MDent (Pros). Department of Prosthodontics, University of Pretoria
^IIBDS, Dip Orth. Department of Orthodontics, University of Witwatersrand
^IIIBChD (Stell), MSc (Odont)(UP), Adv Dip Aesthet Med (FPD). Department of Odontology, University of Pretoria
^IVBChD, Hons BSc (Epid), Dip Ter Educ, MChD, FFD (SA) Ortho, PhD. Department of Orthodontics, University of the Western Cape

Correspondence

 

 

INTRODUCTION

Plato talked about the paradox of doing research by stating "If you know what you're searching for, why do you search for it? If you don't know what you're searching for, what are you searching for?"¹ This statement reflects one of the biggest difficulties researchers have, that is, in the formulation of a flawlessly focused research question. Failure to precisely define that question is also one of the most common errors seen by members of any Dental Scientific Research Committee. During the initial planning stages of any study, some form of protocol is needed as a blueprint for the investigation. This consists of various sections, which are all inter-related and thus need to tie up with each other. After selecting a topic, one should be able to present the main research question / hypothesis as one short statement. This is the Aim of the study. The Objectives then expand on the main Aim in the form of a "To do" list, itemizing the sequence of steps that will be followed.² The Materials and Method is arguably the most crucial section to scrutinize when deciding on the value, relevance and feasibility of the project. At this stage, six key questions need to all be answered in the affirmative to validate the investigation: Is the method reliable and repeatable? Is it scientifically sound? Is it ethically justified? Is the procedure valid? Will the results be of benefit to patients, society or the scientific community? And will the design answer the research question?²

While being aware that the scope and number of topics in dental research is vast, this paper will present examples of problematic research study designs. The flaws will be identified and explained by assessing the investigation in terms of the six questions above. Possible solutions will be suggested to try to improve the study. The examples also serve to illustrate that research need not be technically involved and complicated. There is a wealth of useful information that can be gathered from relatively straightforward studies that are within the reach of clinicians. Such projects can offer valuable clinical advice.

 

CASE SCENARIOS

Case 1: Non-adherence to manufacturer's instructions.

Aim: To test the flexural strength of endodontic files after repeated use and autoclave thermocycling.

Design evaluation: The study aimed to test the flexural strength of a sample of endodontic files after they had been exposed to a varying number of autoclave cycles. However, the manufacturer's instructions for the test files clearly stated that they should only be used once. The researchers justified the investigation by stating that "all clinicians use files more than once".

Reliability and repeatability: The results will be unreliable and indeed of no relevance because these files are being tested on a characteristic for which they were not designed.

Ethically justified: Results will be misleading and the manufacturers may challenge the researcher if negative information is published, for they had clearly stated that the files were meant for single usage.

Validity: By not adhering to recommended handling guidelines the results of this study will be invalid, as these files are not designed to be sterilized.

Benefit of results: These results could be conflicting to clinicians wanting to use this product, as they might presume that the results were actually advocating multiple use up to a certain point.

Answers the research question: Yes and No. The aim has been addressed, however the results may not be reliable or valid, and thus are of no use clinically.

Solution: Strict adherence to all manufacturer's recommendations is essential when testing materials. The researchers should rather have investigated files that are designed for multiple usage and noted the point at which the files broke. This could be valuable clinical information for practitioners who could then institute some form of marking procedure for files after each use and ensure they were discarded before reaching the stage where there was a risk of fracture.

Case 2: Secondary use of data.

Aim: To determine the number and type of post-operative complications following wisdom tooth extraction under general anesthesia.

Design evaluation: The study was retrospective and involved collection of patient record files to determine the number and types of complications encountered after extraction of third molar teeth at a particular institution during the preceding five years

Reliability and repeatability: Many files were missing, data were entered by a variety of clinicians and students, files were incomplete, and not all patients with complications returned for follow-up visits. These results were thus not reliable. Validity: The findings are invalid as they do not reflect the full number or type of complications.

Benefit of results: The data collected may identify some of the common post-operative problems, but will not have sufficient details as to the full extent of the nature of the complications. It will not add any relevant knowledge or be of help to clinicians.

Answers the research question: No.

Solution: Rephrasing the question to investigate the nature of post-operative complications that result in patients returning for follow-up treatment after third molar surgery. Acknowledging in the study the limitations of having missing files, incomplete data and non-standardization of entries. The study could provide further ethical benefits to patients by looking at possible ways to improve recall attendance and monitoring of complications. Justification would be strengthened by also addressing the logistical and managerial issues in the department, by looking for ways to standardize and improve record keeping and file storage.

Case 3: Participant bias.

Aim: To determine the effectiveness of sterility procedures in general dental practices.

Design evaluation: The investigators called a number of dentists and asked permission to visit to conduct a study on their sterility procedures. The investigators were correct in gaining informed participant consent - however, this alerted the clinicians to the impending visit and could have prompted them to alter their behaviour.

Reliability and repeatability: The results will be unreliable as there is no way of knowing if the prior warning led to a brief improvement and more stringent practices, and thus the true nature of sterility procedures in practice may never be known.

Validity: Results may not be a true reflection of routine practices.

Benefit of results: Without knowing the true nature of sterility procedures, the investigators cannot determine if there is a real problem and need for improvement, and have no justification to institute any interventions or educational programmes for private practitioners.

Answers the research question: No.

Solution: Deception studies are undesirable and seldom approved, so the researchers could not have ethically conducted this study without the dentist's knowledge and consent. However, they could have arrived at the surgeries unannounced and gained permission to carry out their investigations at that time. They would have had to guarantee total anonymity and confidentiality of all findings. This raises a different ethical concern. What if they did discover that the practices were substandard and patient's health and welfare was at risk? They had a moral obligation to provide feedback and warn the practitioner of their findings, but were ethically bound to their assurances of anonymity. Post-survey advice would necessitate having some form of contact information, which could only be used if there was complete trust and guarantees of confidentiality from the researchers. In addition, all practitioners could be provided with a written copy of recommended guidelines before conducting the study, so that everyone received the same information and education regardless of the study findings.

Case 4: Researcher bias.

Aim: To determine the durability of a new restorative material.

Design evaluation: The clinician was given a new restorative material to "test" out on patients, and in return was promised a year's supply for free to use.

Reliability and repeatability: The results will be unreliable and unrepeatable as there is no standardization of the types of cavities in which the material was used. There was also no mention of follow up visits to monitor the durability of the material.

Scientifically sound: In material testing there needs to be a clear description of the exact procedures to be followed. In a clinical situation this would entail defining the inclusion criteria for teeth to be filled, such as mentioning cavity site, size, tooth type and position in the arch. There also needs to be specification of conditions that warrant exclusion, if any exist, and justification for their omission.

Ethically justified: The patients may not have been informed that a test material is being used in their mouths. In addition they are being charged for the service and the material while the dentist has received this for free.³ One also has to consider the possibility that the material may not last. This would result in inconvenience and wasted time and costs for many patients who would have to return to have the fillings replaced. How would the dentist explain the failures to them?

Benefit of results: These results could be misleading to other clinicians wanting to use this product as there was no long term follow-up reports. Results would be based purely on personal preference and ease of handling.

Answers the research question: No.

Solution: Firstly, the clinician should have established if there were peer-reviewed scientific trials recommending the use of this material before agreeing to take part in the study and exposing patients to the new product. The trials could then be conducted on a statistically determined random sample of patients, all having different sized lesions. Ideally a third person should evaluate the restorations at the subsequent recall visits. Patients should also be made aware that a new material was being evaluated.

Case 5: Mis-interpretation of data, leading to statistical "lies".

Aim: To determine the incidence of smoking amongst medical students.

Design evaluation: The study looked at final year medical students at one university, but managed to interview only 40 subjects on that particular day (out of a class of over 200). The data collection was accurate, however its interpretation and presentation was totally misleading. One question asked respondents to state their race. There was one Indian male student in the class that day, who also happened to smoke. In the research write up, it was reported that 100% of Indian medical students smoke!

Reliability and repeatability: The results may be unreliable if the sample size was too small and not representative of the entire class.

Scientifically sound: Statistical analysis cannot be performed if the sample is too small or non-representative. Results need to be analyzed and interpreted with caution. Technically, one out of one is 100%, but when presented as a conclusion that is very misleading. Beware of how easy it is to make statistics lie. "They say that 50 % of marriages will fail. Thus statistically, either you or your partner will get divorced".

Ethically justified: Results will be misleading and could cast an aspersion that all Indian medics are smokers. This is poor science automatically equated to unethical study in that it is a waste of time and resources for all involved.⁴

Validity: Poor sampling and misinterpretation of statistics is dangerous and results are misleading, making these findings invalid. This also makes the study unethical and futile.

Benefit of results: The results may still be an indication of the proportion of smokers that could be expected in the whole class.

Solution: Unless there is a compelling reason for race to be investigated it should be omitted from routine research studies. This is because of the sensitivities associated with race, as well as the blurred ancestries of many people and ill-defined classification system. In South Africa, when race is investigated as a variable, research participants are asked to report it as "self-identified" race. The researchers should also have indicated how they planned to use this information.

Case 6: Asking leading questions to (subconsciously) arrive at the desired answers. (The same applies with regards to posing intentionally misleading questions).

Aim: To determine the effectiveness of an oral hygiene intervention programme.

Design evaluation: The investigators wished to evaluate if their community oral hygiene instruction programme had led to improvement in the oral hygiene habits of the children. They conducted the study by means of a questionnaire to be filled in by the scholars. Examples of questions were: Do you clean your teeth twice a day? Do you use a tooth brush and tooth paste to clean your teeth? Do you use dental floss to clean your teeth?

Reliability and repeatability: The results may not be reliable as the children are being presented with the correct answers, and most would know that it is "right" to answer "yes".

Scientifically sound: This questionnaire will not reveal the actual practices, whether the intervention has helped change habits, or if the programme has resulted in improved oral health.

Ethically justified: Although a beneficial oral hygiene instruction programme had been implemented, the follow up research was purely for the investigators to gain information about its effectiveness. Any form of non-therapeutic research is difficult to justify ethically (see note below).

Validity: The findings will be invalid in that they will probably not reflect the actual daily habits.

Benefit of results: This design will not establish whether the oral hygiene programme had been effective. At best it will display if the children know what they are supposed to be doing.

Answers the research question: No.

Solution: The manner in which questions are posed can subconsciously lead respondents to answer what they "think" is correct or what they perceive the researcher wants to hear. It would have been better to have more open ended questions such as: How often do you clean your teeth? What do you use to clean your teeth? The scientific value of the study could also have been improved by having a two pronged investigation. Initially all assenting children could have had a simple clinical examination wherein their DMFT scores were recorded, followed by the intervention, which in this case was oral hygiene instruction. At a pre-determined later date their scores could have been re-evaluated to statistically determine whether there had been any improvement. At this second visit the questionnaire would be handed out, and answers linked to the clinical findings. Ethical note: To justify non-therapeutic research, especially in minors, at both screening sessions, those children found to be in need of treatment should have been attended to, or at least referred to the appropriate centres for care.⁴

Case 7: Trying to establish scientific facts based on subjective observations.

Aim: To compare the buccal corridor and smile aesthetics in extraction versus non-extraction orthodontic cases.

Design evaluation: The researchers planned on using retrospective dental records and photographs of orthodontic patients treated with either extractions and banding or non-extractions and banding. Previous studies had reported that the buccal corridor dimensions changed after orthodontic treatment. The researchers thus wanted to assess aesthetics by means of measuring buccal corridor dimensions on pre and post-operative photographs to see which treatment modality had the better outcome (according to their evaluation).

Scientifically sound: There was no standardization of the nature, or degree of malocclusion of each patient before treatment. The smile assessment was based on personal preferences and was a highly subjective evaluation.

Validity: There is no scientific basis for the assessment, and as such it cannot be used as a predictor for future treatment procedures.

Benefit of results: These results cannot be used as a guide to orthodontists as only personal opinions are reflected. The notion of "beauty" is also highly individual and strongly influenced by cultural norms and identities, as well as by current media trends.

Answers the research question: No. The results are subjective opinions.

Solution: A subjective analysis can never be used as a basis for future clinical treatment decision making. At best, this researcher could have measured the dimensions of the buccal corridor before and after treatment and reported on if and how this changed for each type of orthodontic protocol. If there was a constant finding of the corridor getting larger / smaller, that may help clinicians plan future cases depending on which outcome was desired.

Case 8: Lack of anonymity.

Aim: To establish registrar's perceptions of their learning environment.

Design evaluation: A survey was conducted to gather information on how dental registrars perceived the learning environment at each of the four Universities. The questionnaires were anonymous in order to try to elicit the most honest feedback. However opening questions included the following demographic data: University:

Department: Age: Race: Sex: Year of study. Considering how many 28 year old, black females are in the second year of study in orthodontics at the University of Pretoria, one has to question the anonymity?

Ethically justified: Anonymity cannot be guaranteed and respondents may be victimized if their superiors gained access to the results.

Validity: If the respondents felt the slightest intimation that their identities may be revealed they may not respond in a totally honest manner.

Answers the research question: No. The true feelings may not be revealed and thus the real problems will remain unidentified.

Solution: In almost all research, anonymity is desired by participants and should be guaranteed by investigators. Irrelevant data collection that jeopardizes this anonymity is not ethical and will influence the honesty and thus validity of feedback. The questionnaire should have been structured so that there was no possible way for any of the respondents to be identified in order to gain their trust and foster open and meaningful dialogue.

Case 9: Clinical trials using incorrect methods or outdated materials.

Aim: To test the solubility of gutta percha cones with two different solutes.

Design evaluation: The researchers were testing to see which solute was the most effective in softening gutta percha cones, specifically during endodontic re-treatment. In order to cut costs, they conducted the study using old stock that was no longer in use. However, it was later discovered that the material had long passed its expiry date.

Reliability and repeatability: The results will be unreliable and unrepeatable as the material had expired and ideal properties may have changed. The degree of alteration, and its effect on the solubility are unknown.

Scientifically sound: No study can be sound if the product is not used as stated by the manufacturers. This includes adhering to all manufacturer's directions, such as indications for use; recommended mixing ratios; correct clinical manipulation; and adherence to expiry dates.

Ethically justified: Results will be misleading, clinicians may have clinical failures if they follow the study advice, and manufacturers may challenge the research results.

Validity: By not adhering to recommended handling guidelines the results of this study will be invalid.

Benefit of results: These results could be misleading to clinicians when deciding on which solute to use.

Answers the research question: No.

Solution: In all cases where materials and products are tested the results will be invalid if the material is not handled as advocated by the manufacturer. Reporting results based on erroneous experimental designs or execution can mislead clinicians, and even open the researcher up to litigation by the manufacturers. Ensure that research is always conducted according to set standards, using only approved materials and in keeping with recommended guidelines.

 

CONCLUSIONS

Science and ethics in research are closely linked in a continuous circle. As seen by the examples in this paper, poor science equates to unethical research. If the original study is unethical, then it would be even more unacceptable to replicate it. Research that cannot be tested, repeated, validated or refuted, is invalid and consequently unusable. Studies which cannot be implemented are thus worthless and as such are poor science.

 

References

1. Shields P, Rangarjan N. A Playbook for Research Methods: Integrating Conceptual Frameworks and Project Management. Stillwater, editor. Stillwater; 2013.         [ Links ]

2. Sykes LM, Gani F, Vally Z. Statistical terms Part 1:The meaning of the MEAN, and other statistical terms commonly used in medical research. SADJ. 2016;71(6):274-8.         [ Links ]

3. American Dental Association. Section 5 - Principle of veracity ("truthfulness") Accessed on: 24-10-2016. Accessed at: http://www.ada.org/en/about-the-ada/principles-of-ethics-code-of-professional-conduct/veracity.         [ Links ]

4. Sykes LM, Brandt PD, Evans WG. The ethics of research: Part 4: Safeguarding the scientist, protecting the participants. SADJ. 2016; 71(9):422-4.         [ Links ]

 

 

Correspondence:
Leanne M Sykes
Department of Prosthodontics, University of Pretoria
E-mail: leanne.sykes@up.ac.za