The authors selected the risk factor items of the WAVR-21 based
upon rational consideration of the reliability and validity of
accumulated research findings on violence risk and threat assessment.
Empirical research concerning the WAVR-21 is underway and is intended
to establish the interjudge reliability of the instrument, and
the validity of the instrument in relationship to current and
future violent behavior in the workplace. Such research will take
years to complete, be peer reviewed, published in scientific journals,
and referenced on this site.
We invite
those researchers who are independent of our self-interest in
seeing the WAVR-21 widely used, to conduct their own research
on the instrument. Although the marketing of the WAVR-21 is proprietary,
the structure and the content are not, and the instrument must
withstand the scrutiny of science to determine its future value.
Interrater
Reliability Study of the WAVR-21
Fifteen
raters completed a 1-day training course in the use of the WAVR-21
in March, 2008. They were then asked to each score 5 of 12 cases
developed for the purpose of this study from the files of Drs.
Stephen White, Reid Meloy, and Stephen Hart.
Raters
Raters
were recruited to ensure the diversity of profession and extent
of experience in workplace violence risk assessment. Raters included
6 police officers, 6 mental health/EAP professionals or doctoral
psychology students, and 3 human resource professionals. All were
currently involved in the assessment of workplace violence risk
in their respective employment or training sites.
Cases
The
cases were stripped of identifying information. Cases were deliberately
chosen to be diverse in terms of nature and severity of risk posed
to others. Case materials included personnel records, personal
interviews with perpetrators and victims, police reports, and
mental health reports. Cases did not include any professional
opinions regarding risk of violence or recommendations for risk
management. Amount of material sent to the raters varied from
2-20 pages, but raters of the same case received the exact same
material.
WAVR
Ratings
Each
rater was randomly assigned 5 cases for an anticipated N=75 scored
cases. Raters made their WAVR-21 ratings independently of each
other based solely on the case material, the manual provided to
each of them, and their training. The rater determined presence
of each of the 21 items on a 3-point ordinal scale (0=absent/mild;
1= present; 2=prominent); an index of overall risk based upon
the sum of WAVR 21 item ratings; and judgment ratings of risk
for workplace violence and risk for serious physical harm using
a 4-point ordinal scale (0=low risk; 1=moderate risk; 2=high risk;
3=imminent risk). One case was excluded due to the raters indicating
there was insufficient information to permit scoring. 1 rater
was excluded who did not complete the ratings in the manner requested.
Data
Analysis
Preliminary
analysis were based on ratings for 11 cases by 3 different raters
(N=33). Interrater reliability was determined using intraclass
correlation coefficients for single ratings, or ICC1, calculated
using a mixed effects (absolute agreement) model. ICC1 is mathematically
equivalent to weighted kappa.
ICC1
values were interpreted as follows: <.40 = poor; .41-.49 =
fair; .50-.74 = good; and >.74 = excellent.
Results
ICC1
values for each item ranged from 0.02 to 0.92, with Median .50.
This means that half the items had good to excellent reliability.
Six items had poor reliability. In 3 cases this was the result
of restricted range in the ratings, i.e., almost every rater gave
a rating of 0 in every case.
Overall
Present Rating
The
overall present rating, based on the sum of the individual item
ratings, was 0.84, indicating excellent reliability.
Summary
Rating for Workplace Violence
The
overall workplace violence summary rating was 0.67, indicating
good reliability.
Summary
Rating for Serious Physical Harm
The
overall serious physical harm summary rating was 0.79, indicating
excellent reliability.
Discussion
These
findings of interrater reliability are similar to those obtained
using other structured professional judgment guidelines for assessment
of general violence risk (HCR-20, SAVRY), intimate partner violence
(SARA, B-SAFER), and risk for sexual violence (SVR-20, RSVP).
This
study indicates good to excellent interrater agreement of the
WAVR-21 among professionals who are engaged in workplace violence
risk assessment, whether members of law enforcement, mental health,
or human resource professions. These findings will need to be
replicated with other cases and raters in other settings. The
strength of this study is the care that was taken to insure variability
of cases, selection of raters, and random assignment. The weakness
of the study is the use of retrospective case files and the absence
of naturalistic ratings from actual workplace environments during
the unfolding of a potential workplace violence case. This study
also does not demonstrate validity of the instrument, i.e., its
ability to predict workplace violence, or its concordance with
other violence risk instruments utilized to assess the same case
at the same time. Such findings await further research.
