Virtual
Sampling:
Methodology
Creig R.
Kronstedt, Ph.D.
Programs
in Management for Adults
Cardinal
Stritch College
Milwaukee,
WI
Society
for Judgment and Decision-Making Conference
November
3, 1996
Chicago,
IL
Establishing accurate behavioral decision-making
norms requires the elimination of personal revelation biases. The present
research elicited responses to a set of ethical behavior decisions from adult
business students at Cardinal Stritch College, Milwaukee, WI. It then asked
participants to evaluate the ethicality of business associates, business
competitors, and finally of themselves. Regression Analysis of the decisions
accounted for seventy percent of the variability of perceptions of the
ethicality of others, but was only able to account for fifty percent of the
variability of participant's own perceived ethicality.
The analysis suggests that using research participants as
observers of behavior may provide more validity than asking them to make
personal decisions that may be confounded by a personal response bias. The
`social desirability' of a particular response may affect a respondent's
willingness to share his or her personal feelings as well as the accuracy of
responses (Brookhouse, Guion, and Doherty, 1986; Phillips and Clancy, 1972).
An alternative methodology is proposed which treats the
participant as an observer and reporter of the behavior of those around him or
her. The participants' responses may then be viewed as a `virtual sample' of
the part of the population to which the participant is exposed with the mean of
the sample being the individual him or herself. Just as a large number of
sample means form a normal distribution of their own, these `virtual samples'
would constitute a normal distribution of `virtual sample means' which may
provide a more accurate assessment of the population being sampled than
treating the participants as part of a single sample from the population. The
methodology proposed is offered as a technique to expand and enhance validity
and reliability of decision making research.
Developing instruments and procedures to accurately evaluate
the decision making process is perhaps the primary challenge for research in
decision making. Verbal report is notoriously questionable since participants
may be unwilling to report their personal thoughts or may report them
inaccurately (Randall and Fernandes, 1991). Though this may seem to be the only
way to ascertain such information, its validity (notwithstanding efforts to
establish validity through various means) is dubious (Reidenbach and Robin,
1988,1990); (Reidenbach, Robin, and Dawson, 1991); and (Henthome, Robin, and
Reidenbach, 1992).
The present research attempted to elicit participant
judgments of a set of ethical behavior questions and then asked participants to
give their perception of the ethical levels of business associates, business
competitors, and finally of themselves. Participation was completely voluntary
and participants were assured that results would be kept confidential. While
demographic information such as gender, age, socio-economic status, etc.,
might have been informative, it was not germane to the purpose of the study. It
was believed that gathering such information might have made the respondents
less sure of confidentiality and therefore less willing to give accurate
information. This should not be construed as a criticism of research which does
gather such information, but rather as a cautionary note that gathering such
information when it is not necessary to the focus of a study, may skew the
results.
It was believed that asking participants to evaluate their
associates and competitors might be more accurate method of assessing the
decision processes of individuals because as objective observers the
participants may be more precise and free of personal response bias. Responses
framed as objective decisions also provide a baseline with which to compare
personal judgments of appropriate ethical behavior.
Having participants make judgments regarding a series of
ethical situations provides an ethical profile rather than just an overall
measure of ethicality. This profile was used to develop linear regression
models to predict the overall measure of ethicality of the individual as well
as for the participant's perception of the ethicality of business
associates/competitors. The extent to which the regression models of the
associates/competitors, and individuals match, is an indication of the validity
of the three measures of overall ethicality. A strong match, suggests that a
sampling of participants personal decisions accurately reflect the decision
making processes of the general population. A weak match suggests that personal
response bias skews and confounds the decision making results. The degree with
which the objective decisions predict the ethical level of competitors and/or
associates vs ones personal ethical level is an indication of which comparison,
provides the more valid assessment. If asking a participant to act as an
observer provides a stronger prediction than asking the participant to make a
personal decision, it may be more valid as a tool for measuring the ethics of
the population An objective methodology, if validated, provides a less
intimidating, and perhaps more accurate investigative device.
Decision Making is not an outgrowth of the individual in
isolation, but rather of an interaction of the individual within his or her
cultural grouping. Therefore any evaluating of the decisions of individuals is
really a measure of the degree to which they have assimilated the value system
of their peer group.
Since we are generally attempting to establish cultural
decision making norms it makes sense to eliminate to whatever degree possible,
the biases or half truths of personal revelation. It is, after all, the
behavior of the population as a whole which statistical analysis measures.
Applying such normative data to a particular individual is of questionable
validity.
Methodology
Participants
Participants were a cross-section of 180 students at
the associate's, bachelor's, and master's levels in the Programs in Management
for Adults at Cardinal Stritch College, Milwaukee, WI. While the respondents
were students in a college business program, they are also working
professionals with managerial experience. They ranged in age from mid twenties
to early fifties with the average age being in the mid-thirties. They
were almost equally divided between men and women. The results, therefore, should
better reflect the behavior of the adult population of the business community
as a whole.
Anonymity
As indicated above, researchers assurance of anonymity may
not be completely believed. Participants were told that they were not being
asked demographic information because the researcher did not want to be able to
identify individuals. Participants were asked to turn in their response sheets
face‑down and were informed that if they did not want to participate they
should feel free to turn in a blank survey.
Survey
The first nine survey questions were chosen because they
call for objective judgments of ethical behavior. A ten point scale was chosen
because individuals are more familiar with a decimally based number system and
0-10 relates more directly to a percentage level. Narrower scales such as
0-4 or 0-5 compress the range of choice and may make differences
harder to discern. In addition, pilot testing suggested that almost all
subjects consider these selected behaviors to be at least somewhat unethical. A
scale with ethical and unethical as end points tended to result in a ceiling
effect at the unethical end of the scale.
To increase objectivity, the participants were not asked
whether they had engaged in any of the behaviors, but rather, to make a decision
about how unethical they believed the behaviors were. Their responses were then
used as predictors of their assessment of competitors ethical behavior, of
associates ethical behavior, and finally of their own ethical behavior.
The survey questions were as follows:
HOW
UNETHICAL DO YOU CONSIDER THE FOLLOWING BEHAVIORS?
1. Giving or accepting gifts/favors for preferential
treatment (Quid Pro Quo).
somewhat
unethical very
unethical
1 2 3 4 5 6 7 8 9 10
2. Using company materials, supplies, or services for
personal use.
somewhat
unethical very
unethical
1 2 3 4 5 6 7 8 9 10
3. Taking extra personal time, doing personal business on company time, taking longer than necessary to do a job, or calling in sick to take a day off.
somewhat unethical very
unethical
1 2 3 4 5 6 7 8 9 10
4. Padding an expense account.
somewhat unethical very
unethical
1 2 3 4 5 6 7 8 9
10
5. Violating or allowing a subordinate to violate company
rules.
somewhat unethical very unethical
1 2 3 4 5 6 7 8 9 10
6. Claiming credit for someone else's work or passing blame
for errors onto an innocent coworker.
somewhat unethical very
unethical
1 2 3 4 5 6 7 8 9 10
7. Divulging confidential information from company records.
somewhat unethical very
unethical
1 2 3 4 5 6 7 8 9 10
8. Falsifying company reports (concerning time, quality, or
quantity).
somewhat unethical very
unethical
1 2 3 4 5 6 7 8 9
10
9. Discriminating against another employee on the basis of
gender, race, religion, or sexual preference.
somewhat unethical very
unethical
1 2 3 4 5 6 7 8 9
10
The last three questions of the survey were placed at the
bottom of the survey because it was expected that after considering a variety
of behaviors, the participants would have a mind set of what ethical issues
they were being asked to evaluate. These questions on perceived ethical
behavior moved from the more distant to the more personal level.
On average, how ethical are business competitors with whom
you have contact?
ethical unethical
1 2 3 4 5 6 7 8 9 10
On average, how ethical are business associates with whom
you have contact?
ethical unethical
1 2 3 4 5 6 7 8 9 10
On average, how ethical do you consider yourself?
ethical unethical
1 2 3 4 5 6 7 8 9 10
Surveys typically use Likert type scales though there is
little agreement as to whether the scale should be 0-4, 0-5, 0-7,
or 0-10; or even whether the scale should be agree-disagree; or
strongly agree, agree, disagree, or strongly disagree; etc. (Likert, 1932). What
the scales mean to each of the respondents or whether a "0" or
"4", or " 5" means the same thing to each respondent is
never established. One of the difficulties of doing an analysis of behavior is
that no matter how the researcher tries to establish a range of parameters
(such as a 0-10 scale) the scale may be interpreted differently by each
respondent. One solution is to standardize the responses of each respondent
using the mean and standard deviation of the respondents raw ratings (Hair,
etal, 1995, pp.434-435). Using thus technique, at least, ensures that the
researcher will be able to measure the distance between individual responses
and the "mean" response for a given subject, and the scores of all
subjects may be compared with minimal bias. The standardized scores of each
individual may then be compared to the standardized scores of each of the other
respondents.
Clearly, the list of ethical behaviors does not encompass
all possible ethical or unethical behavior, nor was that intended. The
participants were not a true random sampling, they were a quasi-random
convenience sampling. Since students in the program do represent
demographically, the make-up of the business community of Milwaukee and
the surrounding area, one may draw some general conclusions with regard to
community ethics. However, the primary intent of the research was to determine
the degree to which participants were able to make objective assessments of
ethics, with regard to themselves and others. For these purposes, the data may
have much more general application.
Regression analysis is a valuable tool for determining the
extent to which a set of variables (such as the first nine ethical situations)
can predict another dependent variable (such as overall ethicality) (Neter,
Wasserman, and Kutner, 1990,pp. 225-236). Three separate multiple
regression analyses were performed to determine the extent to which their
decisions about the ethicality of the nine behaviors would predict their
perceptions of competitors, associates, and their own level of ethical
behavior.
The results of the three regression analyses clearly show
that the ethical judgments called for in the first nine questions are very good
predictors of the perceived level of ethics of competitors and associates
(74.155% and 71.421% of variation accounted for), but only average predictors
for ones own perceived level of ethics (51.927% of variation accounted for).
(see table 1 below).
Table 1
Percent of
Variation of the Prediction.
|
|
Multiple R |
R Square |
Adjusted R Square |
Standard Error |
AOV of Regression (F Ratio) |
Significance of F Ratio |
|
COMPET |
.86113 |
.74155 |
.72778 |
.38746 |
53.87634 |
p < .0000 |
|
ASSOC |
.84517 |
.71431 |
.69910 |
.36066 |
46.95118 |
p < .0000 |
|
SELF |
.72060 |
.51927 |
.49367 |
.53327 |
20.28303 |
p < .0000 |
All of the questions were included in the regression
equations for all three analyses. In the competitor analysis, all questions
were highly significant contributors to the prediction of ethical level with
the lowest significance level being questions 7(t = -2.925, p <
.0039)and 6 (t = -3.967, p < .0001) All other questions were
significant at the p <.00001 level or better. In the associate analysis,
question 7 was not a significant contributor (t = -1.143, p < .2547).
In the self analysis, question 7 was not a significant contributor (t =-.743,
p < .4584) and question 6 was a marginal contributor (t = -1.964, p
< .0512).
The Effectiveness
of Assuring Anonymity
Most survey research is done anonymously to protect the
rights of the respondent and to increase the accuracy and honesty of responses.
While anonymity may be assured with great care by the researcher, a more
pertinent concern for the validity of the research is the degree to which the
assurance of anonymity is truly believed by the subject. If it is not believed
by the respondent, it is impossible to determine the extent to which this
affects the truthfulness of responses. For instance, the gathering of
demographic information about respondents may be pertinent and in some cases
essential to the research being done, but such information may also identify
the subject. For example; age, gender, number of children, marital status,
company name, position in a company, etc. may clearly identify whose survey it
is. It is reasonable to assume that respondents may also recognize that their
anonymity is not assured and that they may be less than candid in their
responses. The researcher needs to determine what if any demographic
information is essential to his or her study and forego asking for other kinds
of information.
Researcher
driven vs. Respondent driven surveys
Those of us who administer surveys must constantly keep in
mind that respondents may not want to take the time required to answer all of
the questions to which we may want answers (Groves, 1989). Personal experience
must make it painfully obvious that the longer the survey, the less and less
willing and accurate we tend to be with our response. Not surprisingly, those
of us who design surveys are probably less likely to be people who fill out
surveys and we perhaps look at them differently than the general population.
Participants will almost certainly give more accurate responses to ten question
surveys than they are likely to give in one hundred question surveys.
It is also true that the order of the questions and the
process of answering them is likely to change the nature of the answers. The
survey format is in fact a sort of guided thought process in which the
researcher, leads the participant through a series of evaluative judgments. The
way in which we ask questions and the order in which we ask them necessarily
influences those responses. It was for that specific reason that the survey
given in this research asked the participant to evaluate a series of ethical
issues before asking him or her to evaluate his or her own level of ethics.
Presumably, the questions asked would be the most recent and therefore would have
a stronger likelihood of being the criteria upon which these evaluations would
be based.
The first nine questions called for relatively objective
answers. That is, how ethical do you consider these behaviors, rather than do
you engage in these behaviors. They were not intended to be an exhaustive
sample, but rather a representative sample of the kinds of questions which are
asked with regard to business ethics. The point of the research being to
ascertain how predictive these sample questions would be of ones own ethical
behavior vs how predictive they would be of ones perception of the general
ethical environment both with regard to associates (those who are perceived as
like us) and competitors (those who might be perceived as unlike us). It is
interesting to note that the percent of variation accounted for was essentially
the same in both of those categories, but considerably different from the
evaluation of ones own ethical level. This clearly demonstrates that
respondents tend to make such decisions in response to their perception of the
external ethical situation rather than by making personal revelations. While
there are other interpretations which might be given to such responses, these
seem most relevant to this researcher.
Virtual
Sampling Using a Proximity Sample
The concept of random sampling requires that the sample be
randomly dispersed throughout the population. In all honesty most of us must
admit that our research samples are quasi-random at best, convenience
samples in general, and local sub group samplings at worst (such as classes of
sophomores attending our institutions in our particular fields of study.
The logic of using the participant as observer is that it
increases the level of objectivity of the participant. It may also be argued
that it increases the `virtual size' of our sample. That is, we are in a sense
asking the subject to evaluate the sample of individuals within his or her own
proximity. Clearly the strongest influence will still be the individual's
perception of him or herself, then of close associates, then of more distant
associates. It will gradually disperse itself until it includes ones perception
of ones cultural grouping and it will finally generalize to the entire
population being evaluated.
The disadvantages are, that: a) the subject is not a skilled
observer; b) the subject's own belief system will color his or her own
perceptions of others; c) the sample will be a biased proximity sample centered
on the individual him or herself and on those individuals most closely
associated with him or her.
This `virtual sample' is, of course, not unlike the concept
of a normal distribution of the mean with the exception that the mean will not
be of the population but of the individual him or herself and, most probably,
of individuals most like him or herself, since those are the individuals with
which the individual is most closely associated and about whom the participant
is most knowledgeable. In fact, one might reasonably assume that 68% of the
opinions held by this group will fall within plus or minus one standard
deviation of those of the individual. The individual responses thus constitute
a personally observed sample distribution.
While there will be varying degrees of accuracy of such a
subjective `virtual sample', it may be reasonably assumed that in the
distribution of `virtual samples' there will be some observers who are
extremely bad at assessing the behavior of others and their virtual samples
will constitute bad samples. Some observers, however, will be extremely astute
and those will constitute excellent samples. Most `virtual samples' will be
just average (approximately 68% of them) and thus will fall into the region
within plus or minus one standard deviation of the mean `virtual sample'.
`virtual samples' would therefore appear to have the same distribution as
actual sample means taken from the population and the standard error of the
mean may be assumed to be essentially the same. If we are willing to trust one
`actual sample' taken from a population as representative of the population,
doesn't it make sense to trust 100 `virtual samples' taken from that same
population even though any one individual `virtual sample' may range from bad
to excellent? To some extent, whether or not it is acknowledged, all individual
survey research is subject to these same restrictions since we have no way of
determining whether the information participants give us is accurate except by
asking them.
Undoubtedly the belief system of the individual who
constitutes the `virtual sample' will be projected upon that sample and will
strongly color those perceptions, but it must be acknowledged that, those
perceptions were formed by immersion in that same group and by absorption of
that value system. The advantage is that the individual will be engaging in
self-disclosure of his or her perceptions without feeling the negative
aspect associated with giving personal revelations about oneself. One is thus
put in the empowered position of using his or her own personal telescope to
assess his or her own world, rather than being put under a microscope and being
observed as a specimen.
In addition, the scope of the research is expanded to a
population larger than that actually sampled. This generalization from the
specific sample to the general population is the basis of all statistical
research. Since the statistical assumptions remain the same as for any other
research technique using sample populations, no statistical laws are violated.
Nevertheless, this study suggests that accuracy of results may be enhanced by
making a conceptual shift in the way survey surveys are designed and the way
survey participants are viewed. `virtual sampling' merely acknowledges the
individual observer's assimilation of the perceived values of those in close
proximity to the observer and accommodates this perception to increase the
accuracy of the respondent, while removing the stigma associated with self
revelation.
To the extent that the real sample is truly random, the
`virtual sample' will conceptually provide access a larger portion of the
entire normal distribution. The accompanying research strongly supports thus
contention in that the responses given in the survey predict the ethical level
of the `virtual sample' (both associates and competitors) with much greater
accuracy than it does the ethical level of the individual.
Brookhouse, K., Guion, R.M. and
Doherty, M.;1986, `Social Desirability Response Bias as One Source of the
Discrepancy Between Subjective Weights and Regression Weights', Organizational Behavior and Human Decision
processes 37, pp. 316-328.
Groves, R. M.: 1989, Survey Errors and Survey Costs (Wiley,
New York).
Hair, J.F., Anderson, R.E., Tatham, R.L., and Black, W.C.:
1995, Multivariate Data Analysis with
Readings (Prentice Hall, Englewood Cliffs, N.J.).
Henthorne, T.L., Robin, D.P., and
Reidenbach, R.E.: 1992, Journal of
Business Ethics 11, pp.849-856.
Likert, R.: 1932, A Technique for the Measurement ofAttitudes (New
York).
Neter, J., Wasserman, W., and Kutner, M.H.: 1990, Applied Linear Statistical Models (Irwin,
Homewood, 1L).
Phillips, D.L. and Clancy, K.J.:
1972, "Some Effects of `Social desirability' in survey studies", American Journal of Sociology 77, No. 5,
pp. 921-940.
Randall, D.M. and Fernandes, M.F.: 1991, `The Social
Desirability Response bias in Ethics Research', Journal of Business Ethics 10: pp.805-817.
Reidenbach, R.E. Robin, D. P, and
Dawson.: 1991, `An Application and Extension of a Multidimensional Ethics Scale
to Selected Marketing Practices and Marketing Groups', Journal of the Academy ofMarketing Science 19 (Spring), pp.83-92.
Reidenbach, R.E. and Robin, D. P.:
1990, `Toward the development of a Multidimensional scale for Improving
evaluations of Business Ethics', Journal
of Business Ethics 9: pp. 639-653.
Reidenbach, R.E. and Robin, D. P.:
1988, `Some Initial steps Toward Improving the Measurement of ethical
Evaluations of Marketing Activities', Journal
of Business Ethics 7, pp. 871-879.