What is a definition of evidence-based practice?
integrating the use of best evidence in treating patients which is a balance of clinical expertise, patient feedback and external clinical evidence (research)
Welcomes clinical skill and experience
If there is a better way to practice therapists should find it
Blend old ways with new knowledge
Incorporates patient feedback into best practice
What are the driving forces behind the use of evidence-based practice?
Unexplained variation in practice
Health care costs
Medical errors
Harm now seen in previously approved treatments
Technology trends and advancement in outcome research
What are 3 principle sources of evidence (knowledge) that go into best evidence?
Clinical wisdom – experience from clinical practice
Research wisdom – what gained from research
Patient wisdom – what learned from patient and family
What are the characteristics of a clinician that embraces EBP?
Those practitioners who question their own practice
Those who are “humble about what they know”
Those who always want to improve their own practice
What are the steps in the EBP Process?
Develop clinical questions
Finding of the evidence to answer those clinical questions
Appraisal of this evidence
Application of evidence to their practice
Evaluation of clinical outcomes based upon the application of evidence
What are the myths associated with EBP & what is the reality of those myths?
MYTH - EBP already exists
REALITY - Many practitioners spend little or no time reviewing current medical findings & being introspective about their own practice
MYTH - EBP is impossible to put into place
REALITY - Little work is needed to begin asking clinical questions
MYTH - EBP is cookie cutter health care
REALITY - It is a balance of the 3 factors: practitioner knowledge & practice, clinical research and patient involvement. Guidelines are developed to enhance care and to standardize best clinical practice but allows for appropriate patient based variation in care.
MYTH - EBP is strictly a cost cutting mechanism
REALITY - EBP is designed to eliminate unneeded & ineffective interventions and in that way it is cost cutting. The emphasis is on practice based upon best clinical evidence not the least expensive form of practice.
How do we learn best?
We learn best from intermittent reinforcement – when periodically we receive information
What is confirmation bias?
“confirmation bias” - the tendency we have to see only what confirms our belief, and disregard, or rationalize, contradictory evidence
Good studies designed to eliminate what factor?
A good experimental study is one that is designed to lessen or eliminated the influence of experimenter bias on the outcome of the study
The greater the attempt to eliminated this bias, the higher the quality of the study.
How do we identify a research article?
It is asking a question
It has a methods section
It has a results section
It discusses those results in the context of other studies
What three questions need we answer from all research?
Do these results apply to my patients?
Are the results just by chance or real?
Is there a cause & effect relationship between the intervention and outcome?
What are the levels of evidence from the literature?
Editorials/Opinion
Non-experimental research
Experimental Research
What are some problems associated with good research?
Costly & time consuming
Experimental studies & meta-analyses sometimes hard to generalize because of subgroup composition, effect size and quality of outcome measure
Presentation of evidence sometimes hard to translate into practice
Problems from a clinicians viewpoint:
Lack of time & resources for searching for evidence
Availability of articles to answer you clinical questions
Time to apply the results to your patients
Experimental Research
An active intervention is involved
Those that use more than one group of subjects
One of the groups is a control group – do not have the active intervention
Subjects are randomized to groups – that is not assigned to groups because of their characteristics but have an equal chance of being in a group when they enter a study
Editorials/Opinions
Not necessarily factual; show a lot bias
Non-experimental research
studies that use single subjects or single groups or do not have an active intervention
Systematic review
when articles are judged for quality with higher quality articles given more weight in conclusions
Meta-analysis
mathematically assigned weights
combination of articles to strengthen results
Clinical Guidelines
when panels of experts review experimental research, systematic reviews and meta-analyses to state what “best practice” is
What is a theory?
organized set of constructs or concepts that is proposed to describe and explain systematically a phenomenon
What are the two basic hypotheses?
Null hypothesis – no difference between groups
Experimental (research/alternative) hypothesis – difference between groups
What are 3 key clinical (patient-oriented) questions that need to be asked about each article?
Do these results apply to my patients?
Are the results just by chance or real?
Is there a cause & effect relationship between the intervention and outcome?
What are the main types of questions that are asked?
“Background” questions ask for general knowledge about a condition or thing.
“Foreground” questions ask for specific knowledge to inform clinical decisions or actions.
What is the PICO method for formulating clinical questions?
Patient - Describe as accurately as possible the patient or group of patients of interest
Intervention – Indicate the main intervention or therapy you wish to consider (in other forms of research you can also investigate cause or prognosis)
Comparison (optional) – List any alternative treatment which you wish to compare with the primary treatment you have listed under intervention (some include placebo here – some not)
Outcome – List the particular outcome measure you are studying including a time frame over which you expect the change in the outcome measure to occur
Clinical questions can be asked about what 6 content areas?
- interventions
- clinical measures - reliability, validity, responsiveness
- diagnostic tests - accuracy, risk, cost, etc
- prognostic factors - predict future health and disease
- Self-Reported Outcomes - End point of therapeutic intervention with an analysis of how these outcomes can be measured and the relevance of these outcomes to the patient’s activities
- Clinical Prediction Rules - Clinical prediction rules are systematically derived indications of what types of patients may most benefit from interventions
What is a hypothesis?
a specific prediction of an outcome based upon previous literature or theory/conceptual framework
What is a Boolean operator and why is it valuable in using research databases?
Speeds up a search:
able to combine terms
able to eliminate directions you do not want to go
AND-Narrow search and retrieve records containing all of the words it separates.
OR-Broaden search and retrieve records containing any of the words it separates.
NOT-Narrow search and retrieve records that do not contain the term following it.
() -Group words or phrases when combining Boolean phrases and to show the order in which relationships should be considered
What are search limits and how can they be valuable in searching a database?
Narrow the search to certain categories of articles to enhance the efficiency of the search process
What are the benefits and limits of PubMed (Medline)?
Free
Comprehensive – but many journals is specific areas of physical therapy
Linked to online journal and full text versions of article
Rigorous standards for determining which journals will be included (indexed)
Medical Subject Headings (MeSH) – systematic way to retreave information OR general keyboard search
Search history available
Clinical Queries – evidence-based practice database on the home page
Related articles function
Can personalize it “My NCBI”
Tutorials
What are the types of journal articles and what are the advantages and disadvantages of each?
Peer- reviewed journals – critically reviewed before publication so more sure of quality
Non-peer reviewed journals/magazines – no prepublication review so unsure of quality
Monographs - Less critical review but reviewed at several levels however information rapidly out-of-date (can be out of date at publication)
Conference proceedings – often appearing as monographs related to single topic
What are the types of sources for websites and what are their functions?
News & Current Events – general & specialty sites newspapers & magazines
Business & marketing – corporate, selling a product
Informational – government CDC & NIH & online encyclopedias – wikis & professional organizations
Advocacy - attempt to influence public opinion, online presence of advocacy organizations disease oriented in the health sphere & professional organizations
Personal – hobbists, bloggers, specialty topics, clubs
Others – entertainment
What are the characteristics that you are looking for in a quality website and what questions you would ask about such a site?
relevant to question, updated, reliable source, useful?
Authority – Who authored site & what are their credentials?
Accuracy – Information match other resources?
Objectivity – Is the information balanced or have one point of view?
Currency – How recent information?
Scope – How broad or narrow?
Commercialism – Selling something?
What are the elements of an evidence matrix?
Complete Citation Purpose Statement/Hypothesis Design Type Sample Information Intervention/Independent Variable/Grouping Outcome/Dependent Variable/Measures
What are the benefits and limits of CINAHL?
Cumulative Index of Nursing & Allied Health Literature (CINAHL):
Focuses on Nursing & Allied Health – includes journals not covered in Medline
Somewhat less comprehensive
Can use limits & returns additional limits
Easy to view search history
Advanced and basic search options
Statistical Analysis
Identification of Confounding Variables
Threats to Internal Validity
What are the benefits and limits of PEDro?
Physiotherapy Evidence Database (PEDro) Characteristics
Citations of > 16,000 RCTs, systematic reviews, & clinical practice guidelines
Combines citations and ratings of articles
Exclusively PT
An ability to search by:
Therapeutic approach
Clinical Problem
Body part
PT subspecialty
Can use Boolean operators
Tutorials
What are the benefits and limits of Cochrane Library?
Databases of systematic reviews & meta-analysis
Limited to clinical questions reviewed
Limited to interventions having randomized clinical trials
Few trials on other areas than interventions
Subscription fees but Armstrong has access
Peer Reviewed Articles
Reviewed by a panel of 2-3 reviewers and an area editor – most often articles are returned to the author for revisions and corrections
Research articles – characterized by having intro methods, results & discussion & peer reviewed
Short reports – research articles with same parts but often unlabeled
Editorials (PR)
– invited presentations to discuss issues in the field
Systematic Reviews (PR)
– address overall questions & compares articles with discussion of limitations, article quality
Regular reviews (PR)
– summarize results of research articles but not as critical as systematic reviews
Clinical Guidelines (PR)
– in journals from professional organizations
– statement of that organizations opinion on treatments or other issues
Book reviews, Article Reviews & Technology articles (PR)
– short reviews/critiques of books, articles from other journals & new equipment/software
What are the 5 general features of a research design?
- Philosophical Approach - Quantitative vs Qualitative
- Design Format
- Number of Groups
- Degree of Control of Study Methods
- Time Elements
What characterizes methodological studies about clinical measures?
non-experimental
tend to be cross sectional; may be longitudinal
measure or devise instruments
Level 5
Collection of data is the transition to this level
In vitro (test-tube) & animal research
But question clinical applicability
Philosophical Approach - Quantitative vs Qualitative
Quantitative – evidence arises from large numbers, independence of investigators & subjects, statistical analysis & probability
Qualitative – evidence arises from patterns and themes developed from the experiences of individuals or small numbers of individuals & close interaction between investigators and subjects collection of past data
True Experimental (Design Format)
– Active interventional with at least 2 groups with one group being a control group & randomization to groups
Quasi-experimental (Design Format)
– Active intervention but with one or more of these experimental characteristics missing
Pre-experimental (Design Format)
– Active intervention but study is really descriptive or observational
Non-experimental (Design Format)
- No active intervention and study is descriptive or observational
Number of Groups
Single – within group designs
Multiple – between group designs
No Groups – single case studies or single subject experimental designs
Degree of Control of Study Methods
Recruitment, assignment, communication & management of subjects
Calibration and use of equipment
Maintenance of the environment of study
Administration of testing and /or training
Collection and recording of data
Communication among investigators and subjects
Cross-sectional
– analysis at single time or limited time frame (most experimental studies)
Prospective
Looking forward & collection of new data Allows control of intervention Allows control of measurement Can determine groups Can randomize to groups Can determine cause and effect if true experimental design
Longitudinal
– following over a long period of time (many observational studies
Retrospective
– looking back & collection of past data All data already collected – just find it But unable to control intervention But unable to control data collection But unable to randomize groups
Level 6
Lowest Level –
Editorials and commentaries
Often are not specifically based on the analysis of data
Level 4 – Clinical data
Collection of data from human/patient populations represents the transition to this level Studies included in this level are: Case studies Case series Single group designs
Level 4– Comparison group
Transition to this level occurs where there is more than one group so comparisons between groups can be made
Designs in this category include observational retrospective designs:
Case control design
Retrospective cohort design
Case Control Design
Comparison of individuals with a disease to those without the disease (control)
Investigate exposures of each to find potential risk factors for the development of the health problem (disease)
Retrospective cohort design
Comparison
present health of group exposed to a condition in the past to those not exposed
Looking at risks of developing health problems (disease) from exposure
Level 3
– Prospective studies
Transition to this level occurs where there is a transition from the retrospective to prospective designs
Designs in this category include observational prospective designs such as prospective cohort design
Prospective cohort design
Comparison
future health of group exposed to a condition in the past to those not exposed
Looking at risks of developing health problems (disease) from exposure
Level 2
– Studies with and active intervention
Transition to this level occurs where there is a transition from observational to interventional designs
Designs in this category include designs such as:
Pre-experimental designs
Quasi-experimental designs
Pre-experimental designs
One group with active intervention
Reliability studies
Quasi-experimental designs
Most commonly lacking random assignment to groups
Two groups with assignment based upon some characteristics
Characteristics matched control groups
What are the factors that determine internal validity?
Temporal precedence - the intervention (“cause”) precedes the change in outcome measure “effect” in time
Co-variation - the “cause” and the “effect” are related
Non-spuriousness - there are no plausible alternative explanations for the observed relationship between the “cause” and “effect”
Internal Validity
Characteristic of research studies that reflects the degree to which a causal conclusion is appropriate based upon the methodology used in the study
The degree to which the study minimizes “bias” or systematic error
What are Threats to Internal Validity
When experimental precautions or quality of design of studies are not satisfactory to assure the reduction of bias or or systematic error we have THREATS to internal validity
These threats can be corrected or moderated by procedural components of the studies
Procedural Corrections to Eliminate Threats to Internal Validity
Use of control group
Subjects assigned to get intervention or not (control group)
Difference between responses of the two groups is the true effect of the intervention
Random assignment to groups
Each subject has an equal chance of being assigned to one group or the other
Should equalize effect of any inter-subject differences
Blinding
Making sure that participants in a study do not know the assignment of subjects to groups
Making sure that the subjects to not know their assignment – blinding
When investigators doing the measurements also do not know group assignment – double blinding
Examples of Threats to Internal Validity
Selection (assignment) bias
History
Maturation
Repeated Testing (Testing Effect)
Instrumentation
Regression toward the Mean (Statistical Regression)
Experimental Mortality/Differential Attrition
Experimenter Bias/Compensatory Equalization
Diffusion/Imitation of Treatment
Compensatory Rivalry OR Resentful Demoralization
Placebo Effect
Selection (assignment) bias
Pre-test differences in characteristics of subjects (age, weight, hearing capability, strength, etc.) between groups exist
These differences may interact with the intervention in changing the outcome or by themselves effect the outcome
CORRECTION: Control group with random assignment to groups in theory all characteristics have an equal chance of assignment to each group – however with small numbers of subjects randomization of characteristics is not assured
CORRECTION 2: Stratified random assignment can be used where characteristics are identified and random assignment is made from each of these characteristics. For example divided into men & women and random assignment is made from first men and then women to assure equal distribution between groups
History
Events in the subject’s life outside of the study could affect the outcomes having nothing to do with the intervention
CORRECTION: Random assignment to groups should equally distribute these factors between groups
Maturation
Natural changes in the subject’s outcome variables or response to the intervention can occur during the study
Children age
Diseases wax and wane
Subjects naturally have compensatory behaviors to correct problems
Fatigue
CORRECTION – use of a control group which will exhibit those changes
Repeated Testing (Testing Effect)
Repeated testing may condition subjects to the testing procedure and improve their performance
Repeated testing may give subjects knowledge that can lead to improved performance
Used in all pre-test & post-test designs
CORRECTION: use of a control group ore repeated exposure prior to beginning the study
Instrument change (instrumentation)
Physical measurement device even though calibrated prior to the study may vary in that calibration during the study
Test with any subjectivity the observer (one taking the measurement) may change the way that they assess a change during the study
Change in observer
Observer may become more attuned to changes
Observer may unconsciously changed the criteria they use to make judgments
CORRECTION – use of a control group because any changes would be seen in both control and intervention groups
Regression toward the Mean (Statistical Regression)
If any single subject repeats a measurement they will have a variety of scores depending upon external factors
Repeat enough times see a pattern of scores above and below a central value (the mean)
CORRECTION – control group with random assignment so upward & downward regression are equally distributed between the two groups
Experimental Mortality/Differential Attrition
Inference is only made on those who both start and complete the study
Subjects are ethically allowed to withdraw from a study at any time and for any reason – part of the “informed consent” process of participation
When people leave a study it is called “experimental mortality”
When they leave one group more than the other it is called “differential attrition”
When the characteristics of the subjects assigned to each group changes it is a threat to internal validity
Experimenter Bias/Compensatory Equalization
Investigators may inadvertently affect the outcome by unconsciously behaving differently to members of control and experimental groups
CORRECTION: Blinding is used so that the investigators have no knowledge of the group to which a participant belongs
Diffusion/Imitation of Treatment
When subjects in the control group see the intervention and imitate that intervention – so intervention spreads (diffuses) from the treatment group to the control group
CORRECTION: Isolate control and intervention groups from one another and blind subjects to the intervention
Compensatory Rivalry OR Resentful Demoralization
When subjects in the control group perceive that they are not receiving the intervention and they either:
Work harder to prove that they can show the change (compensatory rivalry)
Give up and do nothing (resentful demoralization)
CORRECTION: Isolate control and intervention groups from one another and blind subjects to the intervention
What is external validity? Why is it important?
Definition – extent to which the sample are generalizable to the population
Study must be qualified by the limitations of the external validity of the study
How do you determine the population from the sample?
Definition – total members of a defined class of people, objects, places or events selected because they are relevant to the research question
Must be defined at the beginning of the study – often stated in title of the study but always in the research question
What is sampling and why is it done?
Sampling is the process of selecting a sample of subjects from a target population
Investigator will attempt to use results from the sample to generalize to the whole population - inference
What is random sampling and how can it be done?
Every population member has an equal chance to be selected
What are the problems with random sampling?
It is difficult or impossible to identify all members of any population
Even if identified, it is difficult contact all members
Even if contacted, subject have the right to refuse to participate and others may be unable to participate
What is cluster sampling?
Random groupings of individuals selected rather than random individuals
Groups are randomly selected then all individuals in the group are part of the sample
It is important that each group (cluster) does not differ greatly from each other or from the population from which the groups are chosen
- What concerns do you have about cluster sampling?
It is important that each group (cluster) does not differ greatly from each other or from the population from which the groups are chosen
- What is convenience sampling?
Only those individuals who can be easily identified or located are included
Most common form of sampling
- What concerns do you have about non-probability sampling?
Sampling error
Differences occur between the sample and the population with certain characteristics that are due to random differences
Occur through no fault of the researcher
Systematic error
Due to a systematic difference between the sample and population rather than random differences
Response rate error – when those who participate in the study have specific characteristics that make them different from the population
Coverage error – when researchers mistakenly restrict their sampling framework to a subset of the population of interest
What is purposive sampling?
(“snowball) sampling –
Selection of individuals with specific characteristics
Often done when population members are hard to identify
What is sample size and why is it hy is sample size important?
Number of participants required for the study
Major decision when planning research
Decided in advance, otherwise it may look as if data collected until hypothesis was supported
What are the factors that have to be taken into consideration when determining sample size – power analysis?
Practical considerations – limited factors Availability of subjects Cost in terms of time and money Representation and generalization Making sure all characteristics of the population represented in the sample More varied characteristics more subject needed When making group comparisons: Variability of the sample Size of the expected difference Population size Types of subjects Purpose of the research Ethical considerations
Why is it important to determine sample size before you begin a study?
Decided in advance, otherwise it may look as if data collected until hypothesis was supported
How do you calculate the number of subject you need based upon the size of the expected difference and upon variability of the sample?
Use the formula n= 16(s2/d2)
What are inclusion criteria and what is their value?
Those characteristics makes a person part of the population in terms of clinical findings, demographics, or other factors that need to be studied
Must reflect the characteristics of the population – if not threatens external validity
What are exclusion criteria and what is their value?
Factors that preclude someone from being in a study
Includes those factors that may introduce confounding variables
If too exclusion criteria too numerous, unreasonable or restricts the sample too much compared to the population can also threaten external validity
What are independent variables and what are examples?
Grouping Variable
Manipulated in experimental studies
What are the characteristics of independent variables and what are examples?
A variable is a characteristic or feature that varies, or changes within a study
The opposite of variable is constant: something that doesn’t change
The characteristic or feature that varies need to be of interest in the study to be considered variables
What are active independent variables?
Specifically designed intervention is actively imposed on volunteer subjects and controlled by the investigators
This is the highest level of independent variables, met by true experimental studies.
It has the advantage of having a consistent intervention, where all subjects receive the same treatment under the same conditions, dose, and setting
What are the levels of independent variables?
Variation of the same independent variable are considered levels
Example – different intensities of the same form of exercise
All the members of one group will have the same level of the independent variable
Different groups will have a different level of that same variable
What are independent independent variables?
Independent (between groups) grouping variables have different subjects in the different groups to be compared
The people who are in the experimental group are different (independent) people than those who are in the control group
Allows for between group or independent comparisons
What is the importance of random assignment to groups?
Assignment to groups can raise bias – bias of the characteristics of the subjects & bias of the investigator
Random assignment eliminates experimenter or subject preference bias in determining group assignment
All variables that are randomly assigned are active, since the investigators must have active control over the variable to randomly assign them to subjects
What are multiple independent variables?
Single studies frequently have several independent variables.
Permits the assessment of multiple variables simultaneously, and permits the assessment of their interaction.
The most common usage – 2 group, Pre-test/Post-test Design
Combination of active intervention variable (treatment and control) along with a pre-test & post-test assessment of the outcome measure (dependent variable)
What are attribute independent variables?
Groups were defined by a characteristic of the subjects or chosen by the subjects.
Groups are compared, but the grouping variable cannot be chosen and manipulated by the investigators because it is a characteristic of the subjects themselves, or it is unethical to manipulate
What are repeated measures independent variables?
Repeated measures are “within groups” variables; have several types
2 group, Pre-test/Post-test Design
Combination of active intervention variable (treatment and control) along with a pre-test & post-test assessment of the outcome measure (dependent variable)
Compare the intervention to the control (independent, independent variable)
Compare the effect of time (pre-post) on the outcome variable (repeated measure, independent variable)
Used when the hypothesis is that the intervention or treatment will produce improvement in symptoms
Investigators would set up the study so that the groups are equivalent at the start of the study
Groups are equivalent because subjects are randomly assigned to groups
Confirm that the groups are not different at the start of the study by statistically comparing the pre-test scores of the two groups
Then after the intervention
Post-test would be given to all subjects in both groups, and the change of both groups between the pre and post tests would be compared
Since active intervention, a control group & randomly assignment to groups, the investigators could conclude that it was the intervention that was imposed that caused any differences in this pre-post change
If not a true experimental design at least could conclude association between intervention and change observed
Time of Testing – Pre-test/Post-test
Pre-test before the intervention & post-test following the intervention - on all subjects
The measurements from the pre-test are taken from the same individuals (“within groups”) as the post-test
Studies can have multiple repeated measures, measuring the dependent variable at “baseline”, 3 month, 6 month, 1 year, 5 year follow-up
The effect of the independent variable is the change that occurs within the group over the testing times
Time of testing variables are always independent variables, and always active
Investigators must have control over the testing schedule to have the multiple tests.
Matched or Paired Subjects in the groups
Before subjects are assigned to groups, they are matched with another subject based on their age, gender, smoking status, or other variable thought to be a possible confounding variabl; one member of each pair is randomly assigned to the control group, and the other to the experimental group
In this case, the effect of the independent variable is considered “within groups” because of the matching of variability between the groups
This is a useful strategy because it reduces the “noise” or error in the study, so actual differences can be detected if they are present