Establishing the Validity and Reliability of
a Research Instrument
Construct validity
– Overall validity encompassing other elements
– Do measurements:
• A. Represent all dimensions of the concept
• B. Distinguish concept from other similar concepts
– Tied to meaning analysis of the concept
• Specifies the dimensions and indicators to be tested
– Assessing construct validity:
– A. Convergent validity
– B. Discriminant validity
اسلاید 254
Convergent validity:
– عبارتست از همبستگی نسبتاً قوی میان سؤالات مربوط به یك حیطه با همان حیطه
– To assess we examine construct loadings and average variance
extracted (AVE)
– Rule of Thumb: Standardized loadings estimates should be .5 or
higher, and ideally .7 or higher.
– AVE should be .5 or greater to suggest adequate convergent
validity
اسلاید 256
Discriminant validity:
– Ability of measure of a concept to discriminate that
concept from other closely related concepts
– Measures of different constructs should not correlate
highly with each other.
– E.g., Measuring Self-esteem construct and Locus-of-control
construct as distinct concepts. Might be correlated but not too
highly or this is an issue.
اسلاید 257
Rule of Thumb: According to Fornell and Larcker (1981),
if the square root of the AVE estimate for each construct is
greater than the correlation between that and all of the
other constructs in the model, then discriminant validity is
demonstrated.
اسلاید 259
The concept of reliability:
Consistency" or "repeatability" of your measures:
-Reproducibility over time, over different indicators, used by different
interviewers.
Estimates of Reliability
– Statistical coefficients that tell use how consistently we
measured something.
اسلاید 261
Factors affecting the reliability of a research instrument
– The wording of questions
– The physical setting (In the case of an instrument being used in an interview…)
– The respondent’s mood
– The interviewer’s mood
– The nature of interaction (In an interview situation, the interaction between
the interviewer and the interviewee)
– The regression effect of an instrument (When a research instrument is
used to measure attitudes towards an issue, some respondents, after having
expressed their opinion, may feel that they have been either too negative or too
positive towards the issue. The second time they may express their opinion
differently, thereby affecting reliability.)
اسلاید 262
Methods of determining the reliability of an instrument in
quantitative research
1. External consistency procedures:
Test/retest or repeatability test: This is a commonly used method for
establishing the reliability of a research tool.
An instrument is administered once, and then again, under the same or
similar conditions.
- Advantage: it permits the instrument to be compared with itself
- problem of recall
Parallel forms of the same test: In this procedure you construct two
instruments that are intended to measure the same phenomenon. The two
instruments are then administered to two similar populations.
اسلاید 263
Methods of determining the reliability of an instrument in quantitative
research
2- Internal consistency procedures (The split-half technique,
Cronbach's Alpha (a), Average Inter-item Correlation, Kuder-
Richardson)
در یك پرسشنامه هر خرده مقیاس باید صفت مشابه ای را اندازه بگیرند و سوال دیگری وارد
این مجموعه نگردد.
In split-half reliability we randomly divide all items that purport to
measure the same construct into two sets. We administer the
entire instrument to a sample of people and calculate the total
score for each randomly divided half.
Cronbach's Alpha (): Correlation of each item on the measure all
other items on the measure.
اسلاید 264
Reliability is a necessary condition for validity -
consistency as an indicator
Reliability is not a sufficient condition for validity
- consistency does not = accuracy
اسلاید 265
Validity and reliability in qualitative research
– Trochim and Donnelly (2007) compare the criteria proposed by
Guba and Lincoln in the following table with validity and
reliability as defined in quantitative research:
اسلاید 266
Quality of Qualitative Research
Credibility
– establishing that the results are credible from the perspective of the
participant
Transferability
– degree to which results can be generalized to other contexts
Dependability
– description by the researcher of changes within the context that the
research occurs and how these might affect conclusions
Confirmability
– degree to which others can confirm or corroborate the results
اسلاید 2Validity and Reliability in Qualitative
Research
Qualitative validity means that the
researcher checks for the accuracy of the
findings by employing certain procedures,
while qualitative reliability indicates that the
researcher’s approach is consistent across
different researchers and different projects
(Gibbs, 2007).67
اسلاید 268
Validity is one of the strengths of qualitative research and
is based on determining whether the findings are accurate
from the standpoint of the researcher, the participant, or the
readers of an account (Creswell & Miller, 2000).
Terms abound in the qualitative literature that address
validity, such as trustworthiness, authenticity, and
credibility (Creswell & Miller, 2000), and it is a muchdiscussed
topic (Lincoln, Lynham, & Guba, 2011).
اسلاید 269
Internal Validity in qualitative research
In qualitative research, equates to credibility
The right setting and informants
Accurate reflection of situation, informant
perceptions
Multiple approaches lead to similar results
Multiple researchers yield similar
interpretations
Peer review/Informant review
اسلاید 270
Triangulate different data sources of information by examining
evidence from the sources and using it to build a coherent
justification for themes.
Use a rich, thick description to convey the findings. This
description may transport readers to the setting and give the
discussion an element of shared experiences. When qualitative
researchers provide detailed descriptions of the setting, for
example, or offer many perspectives about a theme, the results
become more realistic and richer. This procedure can add to
the validity of the findings.
اسلاید 271
Use an external auditor to review the entire project.
Spend prolonged time in the field. In this way, the researcher
develops an in-depth understanding of the phenomenon under
study and can convey detail about the site and the people that
lends credibility to the narrative account.
Self-reflection, acknowledgement of own biases: Clarify the
bias the researcher brings to the study
اسلاید 272
External Validity in qualitative research
(Qualitative generalization)
In qualitative research, equates to transferability
Transferability is responsibility of reader,
not researcher
Provide dense description
Use nominated informant sample
Provide detailed demographic and
situational description
اسلاید 273
External Validity in qualitative research
(Qualitative generalization)
External validity is problematic in qualitative
research because In qualitative research, a single
case or small nonrandom sample is selected
precisely because the researcher wishes to
understand the particular in depth, not to find out
what is generally true of the many” (Merriam,
1998, p. 208).
اسلاید 274
Researcher is the instrument”- how to test for reliability?
How do qualitative researchers check to determine if their
approaches are reliable (i.e., consistent or stable)?
Yin (2009) suggested that qualitative researchers need to
document the procedures of their case studies and to
document as many of the steps of the procedures as
possible. He also recommended setting up a detailed case
study protocol and database, so that others can follow the
procedures.
اسلاید 275
Gibbs (2007) suggested several qualitative reliability
procedures:
Check transcripts to make sure that they do not contain obvious
mistakes made during transcription.
Make sure that there is not a drift in the definition of codes, a
shift in the meaning of the codes during the process of coding.
For team research, coordinate the communication among the
coders by regular documented meetings and by sharing the
analysis.
Cross-check codes developed by different researchers by
comparing results that are independently derived.
اسلاید 276
STEP IV Selecting a Sample
The differences between sampling in
quantitative and qualitative research
- In quantitative research you attempt to select a sample in such a way that it is
unbiased and represents the population from where it is selected.
- In qualitative research, number considerations may influence the selection of a
sample such as: the ease in accessing the potential respondents; your judgement
that the person has extensive knowledge about an episode, an event or a situation
of interest to you; how typical the case is of a category of individuals or simply
that it is totally different from the others.
اسلاید 277
The differences between sampling in
quantitative and qualitative research
- - The determination of sample size: In quantitative research you are guided by a
predetermined sample size. In qualitative research you do not have a
predetermined sample size but during the data collection phase you wait to reach a
point of data saturation
-- Considerable importance is placed on the sample size in quantitative research.
The sample size in qualitative research does not play any significant role as the
purpose is to study only one or a few cases in order to identify the spread of
diversity and not its magnitude.
اسلاید 278
In quantitative research, randomization is used to avoid
bias in the selection of a sample and is selected in such a
way that it represents the study population. In qualitative
research no such attempt is made in selecting a sample.
You purposely select ‘information-rich’ respondents who
will provide you with the information you need
اسلاید 279
Assumptions of quantitative sampling
We want to generalize to the
population.
Random events are
predictable.
We can compare random Therefore…
events to our results.
Probability sampling is the
best approach.
اسلاید280
Assumptions of qualitative sampling
Social actors are not
predictable like objects.
Randomized events are
irrelevant to social life.
Probability sampling is
expensive and inefficient.
Therefore…
Non-probability sampling is
the best approach.
اسلاید 281.
Sampling in quantitative research
Sampling is the process of selecting a few (a sample) from a bigger
group (the sampling population) to become the basis for estimating or
predicting the prevalence of an unknown piece of information,
situation or outcome regarding the bigger group.
اسلاید 283
Simple Random Sample
1. Get a list or sampling frame”
a. This is the hard part! It must not
systematically exclude anyone.
b. Remember the famous sampling mistake?
2. Generate random numbers
3. Select one person per random number
اسلاید 287
Selecting Random Samples
1. Simple Random sampling
Advantages
1. Easy to conduct
2. High probability of achieving a representative sample
3. Meets assumptions of many statistical procedures
Disadvantages
1. Identification of all members of the population can be
difficult
2. Contacting all members of the sample can be difficult
اسلاید 288
Systematic Random Sample
1. Select a random number, which will be known as
k
2. Get a list of people, or observe a flow of people
(e.g., pedestrians on a corner)
3. Select every kth person
– If every Kth person on the list is, say, rich” or senior” or some
other consistent pattern, avoid this method
اسلاید 290
Systematic sampling
Example, to select a sample of 25 dorm rooms in your college dorm,
makes a list of all the room numbers in the dorm. For example there
are 100 rooms, divide the total number of rooms (100) by the number
of rooms you want in the sample (25). The answer is 4. This means
that you are going to select every fourth dorm room from the list. First
of all, we have to determine the random starting point. This step can be
done by picking any point on the table of random numbers, and read
across or down until you come to a number between 1 and 4. This is
your random starting point. For instance, your random starting point is
"3". This means you select dorm room 3 as your first room, and then
every fourth room down the list (3, 7, 11, 15, 19, etc.) until you have
25 rooms selected.
اسلاید 292
Stratified Random Sample
طبقه ای
1. Separate your population into groups or strata”
according to some criterion, such as geographic
location, grade level, age, or income.
2. Do either a simple random sample or systematic
random sample from there
a. Note you must know easily what the strata” are
before attempting this
b. If your sampling frame is sorted by, say, school
district, then you’re able to use this method
اسلاید294
Selecting Random Samples
Stratified random sampling (continued)
– Advantages
• More accurate sample
• Can be used for both proportional and non-proportional
samples
• Representation of subgroups in the sample
– Disadvantages
• Identification of all members of the population can be
difficult
• Identifying members of all subgroups can be difficult
اسلاید 295
Cluster sampling
خوشه ای
1. Get a list of clusters,” e.g., branches of a
company
2. Randomly sample clusters from that list
3. Have a list of, say, 10 branches
4. Randomly sample people within those branches
a. This method is complex and expensive!
اسلاید 298
Nonrandom sampling methods.
There are four commonly used nonrandom
designs, each based on a different
consideration, which are commonly used in
both qualitative and quantitative research.
اسلاید 299
1- The Convenience Sample
Find some people that are easy to find
the process of including whoever happens to be
available at the time
– Examples:
• Man on the street”
• Agricultural student in the library
• Volunteer samples
• Patient coming to OP
…called accidental” or haphazard (hapˈhazərd)” sampling
اسلاید 300
disadvantages…
…difficulty in determining how much of the
effect (dependent variable) results from
the cause (independent variable)
درباره این سایت