PARAMETRIC AND NONPARAMETRIC MULTIPLE COMPARISONS USING SAS Chandu M. Patel, Ortho Pharmaceutical Corporation
Abstract
CASE I:
ni=nj, ai 2= aj 2
(i) Tukey (T) Method: The multiple range test of Tukey (1953) is an Mep that provides the following 100(1.)% simultaneous Confidence Intervals (CI) for the k* pairwise
Many statistical computing packages including
SAS provide limited methods of Multiple Comparison Procedures (MCP). There is a need for some convenient way to perform a variety of MCP. Many authors including Dunnett (1980a,
differences
~i~j
YiYj ~ SR.,k,v[s2/ n]1/2
1980b) described several useful parametric MCP for three cases:
Where Vi denotes the sample mean for the ith
group, SR. k v is the. point of the distributibn'of the Studentized Range (SR) of k and
(III)
normal variates, and s2 is the estimate of 0 2 based on v df. The sample sizes ni are assumed
nij,nj, ai2j,ai
to be equal to n. The SR tables have been tabulated by Harter (1969).
Rank analysis of variance by Kruskal and Wallis (1952) with correction for ties is described.
Also, MCP based on ranks proposed by Dunn (1964) is described to compare treatments with a control and all possible comparisons.
CASE II:
(i) Scheffe (S) Method: Scheffe (1953) method gives the following 100(1.)% joint CI for the
The purposes of this paper are to describe how
SAS may be used to perform the computations for the above parametric and nonparametric MCP
pairwise contrasts
including the KruskalWallis test with ties corrected, to provide macros using the MATRIX procedure for performing these calculations, and
~iUj
YiYj ~ [(kl)F(.,k1,v)s2(1/ni+1/ nj)]1/2 Where F(a,kl,v) is the upper a point of the centra1 F distribution with kl and v df.
to illustrate them with a numerical example.
(ii) Bonferroni (B) Method: Miller (1966) gives the Bonferroni 100(1.)% joint CI for "i"j
PARAMETRIC MULTIPLE COMPARISONS
YiYj ~ t./2k*,v[s2(1/ni+1/ nj)]1/2 Where ta v is the upper a point of the t distribution with v df. The Bonferroni ttab1es
Consider oneway fixed effect ANOVA model: Yij=~i +
n If' .1n·J' a 1L  a·J 2
have been tabulated by Bailey (1977).
e;j; i=l ••.. k. j=l •..• ni
Where eij ~ NID (O,a 2) random variables and the and 0 2 are unknown parameters. Let Vi be the
(iii) OunnSidak (OS) Method: Following Sidak (1967) and Dunn (1974), an improved Btype method for which ";"j yields the following 100(1.)% joint CI
Pi
estimate of the sample mean for the ith group
and s2 be the Mean Square Error (MSE) estimate with v=nk degrees of freedom (DF)~ where n=Eni and ni is the number of samples per group.
ViVj ~ t.*,v[s2(1/ni+1/ nj)]1/2 Where a*=1/21/2(10)1/k*. The ttables for OS method have been tabulated by Games (1977).
The problem of simu1taneous estimation of the entire class of k*=k(kl)/2 pairwise comparisons Ui~j of the k means are divided into three caseS:
(iv) Hochberg (GT2) Method: Hochberg (1974) gives 100(10)% joint CI for "i"j
ViVj ~ SMMo ,k*,v[s2(I/ni+1/ nj)]1/2 and
(III)
For each
nij,nj, ai 2 j,aj2
case~
Where SMM. k* v is the upper 0 point of the studentize~ Maximum Modulus (SMM) distribution with parameter k* and v df. The SMM tables have been tabulated by Staline and Ury (1979).
selected Multiple Comparison
Procedures (MCP) are described.
893
(v) Spjotvoll  Stoline (T') Method: Spjotvoll  Stoline (1973) 9ive 100(1«)% joint CI for "i"j
Consider oneway fixed effect ANOVA model:
YiYj ~ SAR.,k,v[s2MIN(ni,nj)]1/2 Where SAR is the upper
a
ni~nj, ai2~ajZ
CASE Ill:
Yij=lJi + eij; Where eij ~ NID (O,ai 2 ) and Ui and cri 2 are unknown for i=l, ... k, and ~=l, ... ni' Let Vi be
point of the studentized
Augmented Range (SAR) distribution with parameter k and y df. The SAR tables have been tabulated by Staline (1978).
the ith sample mean and si
be the estimate of
ai 2 based on Vi df independent of Vi. The jOint CI estimates for the k* differences uiuj are
described by the following methods.
(vi) Gabriel (G) Method: Gabriel (1978) gives 100(10)% joint CI for "iPj
(i) Games and Howell (GH) Method: Games and Howell (1976) proposed 100(1.)% joint CI
'ri'rj ~ SMM.,k*,vS[I/JTrii + 1/./2I1j]
for
This method does Produce narrower CI than the
lqJI j
YiYj ~ Aij,.,k[Si 2/ ni+ s}/nj]I/2
CAl
GT2 method, although G method may be somewhat liberal, especially for those cases with small • and large imbalances. If graphical
Where A;J' , • , k=SR. , k, v'lJ./J2
considerations are important, the G method might be preferred.
and Vij=(si2/ni+Sj2/nj)2/(s;4/n;2vi+Sj4/nj2vj) denotes the Welch (1938) approximate formula for df.
(vii) TukeyKramer (TK) Method: The TK method is known as Kramer's (1956) method and was first discussed by Tukey (1953). This is an
(ii) Cochran (C) Method: The 100(1.)% jOint CI for "i"j 9iven by equation (A),
approximate extension of the T method for unequal
sample size.
The 100(1«)% joint CI for "iOj is
Where A;j,a,k=SR*o.,k,Vi/J2 and
YiYj ~ SR.,k,v[s2(I/ni+1/ nj)/2]1/2
SR* k
For the balanced case, the TK method reduces to T method.
k v,·Si2/ni+SR« •k• v·sJ·2/nJ· .. _ SR. •• J 2 2 (si /ni+sj /nj)
a, 'VlJ 
TK method always produces narrower CI
than the S, B, OS, GT2, T' and G methods and therefore TK method is recommended for the simultaneous estimation of all pairwise mean
corresponds to the weighted average of student1s
t proposed by Cochran (1964) as an approximate solution to the BehrensFisher problem for k=2
differences.
groups.
(viii) Multiple Comparison with a Control: Considering first group as control, Dunnett
(iii) Tamhane (T2) Method: Tamhane (1977, 1979) proposed 100(1.)% joint CI for "iPj
(1955, 1964) gives 100(1.)% joint CI for "i"1 YiYl ~ MVT.,k_l,v[s2(1/ni+1/ nl)]1/2
Y"iY"j ~ t.*,Vij[SI2/ni+sl/nj]I/2
Where MVT. kI v is the upper. point of the
Where t a * v .. is the twosided (1* point of
multivariate t'distribution with parameter k and
v df. The 0 tables have been tabulated by Dunnett (1955, 1964).
student's't'Jdistribution with Vij df and .*=1(1.)1/ k*.
(ix) Comparison of several dose levels with a zero dose control: Following Bartholomew
(iv) Tamhane (T3) Method: Tamhane (1977,1979) proposed 100(1.)% joint CI for "iPj
(1961), compute the means as increasing or
YiYJ· + SMM. k*
decreasing levels of dose. Williams (1971, 1972) proposed the 100(1.)% jOint CI for the comparison of several dose levels with a zero

"
V'
,[s;2/ni+sJ·2/nJ·]1/2
1J
Where SMM. k* v·' is the. point of the SMM of k* uncorre'at~dlriormal variates with Vij df.
dose control
(v) BrownForsythe (BF) Method: Brown and Forsythe (1974) proposed 100(1.)% joint CI for "i"j
'ri*Yl ~ W.,k_I,v[s2(I/ni+1/nl)]1/2 Where W. kl v is the upper. point of the
Williams't a~stribution with with parameter kl and v af, and V;* is the ordered mean of dose i.
'riYj ~ [(kI)F(.,kI,Vij)(s;2/ni+Sj2/nj)]1/2
The Wtables have been tabulated by Williams (1971,1972).
Comparing C method with T2 and T3 method, it is observed that the C method is preferrable for large df and T3 method is preferrable for small df.
The GM method is also recommended at the
risk of being somewhat liberal.
894
Three macros (i) Multiple Comparison with homogeneous variances (ii) Multiple Comparisons with heterogeneous variances and (iii) Kruskal
KRUSKALWALLIS TEST WITH CORRECTION OF TIES
Wallis test with ties corrected and Dunn's Multiple Comparisons were programmed using
Consider oneway fixed effect model: Yij=~i +
PROC MATRIX of SAS (SAS Users Guide, 1979 Edition
eij' i_l, ... k, jl,2, ••. nio
SAS Institute Inc., Box SOOO, Cary, North
Carolina). One data set is utilized to illustrate the method in SAS. The program
Where eij are independent, identically distributed (iid) random variables and "i are ith treatment effect. To test whether or not all
listing is available by sending the request to the author.
treatment groups have been drawn from the same
population, Kruskal and Wallis (1952) proposed the following test based on ranks.
EXAMPLE
Rank all N=Eni observations from the smallest to the largest and the rank sums for each
The data for this example may be found in Winer (1971). There are four groups each having B, 6, 5 and 7 observations per group. The summary of
treatment are computed. Compute the test statistic H*=H/CT
results of the above methods are given below.
Where H=[12/N(N+l)] ~ (Ri2/ni)3(N+l) g CT=I[ E (t s3t s )/(N3_N)] s=1
SUMMARY OF RESULTS
Ri=Rank sum for treatment ;
GROUP
N
MEAN
VAR •.
S.E.
1
8 6 5 7
2.25 3.00 7.00 9.43
1.07 2.00 5.00 5.95
0.36 0.58 1.00 0.92
g=number of tied groups and
2 3 4
ts=size of tied group s. In particular, if there are no tied observations. then g=N, t s=1, so that correction for ties (CT)
ANOVA TABLE
reduces to 1. The above test statistic H* has an asymptotic chisquare distribution based on kl df.
SOURCE
OF
MS
FCAL
P
Group Error
3 22 25
79.97 3.33
24.03
0.01
Total
NONPARAMETRIC MULTIPLE COMPARISONS BASED ON KRUSKALWALLIS RANK SUMS
k=4, v=22, s2=3.33 and k*=k(kl)/2_6
(i) Multiple comparison with a control. Considering first group as control, Dunn (1964) gives 100(Ia)% joint CI for "i"1
SUMMARY OF RESULTS OF MULTIPLE COMPARISON CASE II:
~j~~1 ~ Za/2P[S2(llni+l/nl)]1/2
Where ~i= ~ Ri/ni
COMPARISON 1 1 1 2 2 3
g
S2=[N(N+l)/12 E (t s3t s )/12(Nl)] s=1 Za=upper a point of the standard normal distribution
and
vs vs vs vs vs vs
2 3 4 3 4 4
nilnj, oi 2=Oj2
MOIFF
501
TCAL
0.75 4.75 7.18 4.00 6.43 2.43
0.985 1.040 0.944 1.105 1.015 1.068
0.76 4.57 7.60 3.62 6.33 2.27
p=number of comparisons performed.
(ii) All Multiple Comparisons: Dunn (1964) proposed 100(10)% joint CI for "i"j
CRITICAL VALUES USING Bonferroni (B) Method OunnSidak (OS) Method Hochberg (GT2) Method
RiRj ~ Za/2k*[S2(1/ni+1/ nj)]1/2 Where k*=k(kl)/2
895
p=D.05 2.90 2.89 2.875
p=O.01 3.58 3.58 3.575
CASE II I:
TUKEYKRAMER (TK) TEST COMPARISON I I I 2 2 3
vs vs vs vs vs vs
2 3 4 3 4 4
MDIFF
SOl
TCAL
TTABLE
0.75 4.75 7.1.8 4.00 6.43 2.43
0.697 0.735 0.668 0.781 0.718 0.755
1.08 6.46 10.75 5.12 8.96 3.21
SR4 22 p=.D5=3.93 p=.01=4.96
COCHRAN (C) METHOO
I I I 2 2 3
vs vs vs vs vs vs
2 3 4 3 4 4
MDIFF
SOl
TCAL
0.75 4.75 7.18 4.00 6.43 2.43
0.745 0.816 0.890 0.816 0.745 0.816
1.01 5.82 10.41 4.90 8.63 2.98
1 1 1 2 2 3
vs vs vs vs vs vs
2 3 4 3 4 4
MDIFF
SOl
TCAL
0.75 4.75 7.18 4.00 6.43 2.43
0.983 1.033 0.944 1.103 1.014 1.064
0.76 4.60 7.61 3.63 6.34 2.28
1 1 1 2 2 3
vs vs vs vs vs vs
2 3 4 3 4 4
EST. OF
TTABLE AT P=.05
vs vs vs vs vs vs
0.75 4.75 7.18 4.00 6.43 2.43
1.10 4.46 7.24 3.46 5.91
8 5 7 6 9 9
3.582 3.983 3.444 3.977 3.529 3.793
2 3 4 3 4 4
1.7g
GAMESHOWELL (GH) ME THor
p=.0~=3.93
p=.01=4.96
COMPARISON
MDIFF
TCAL
TTA8LE AT P=.05
vs vs vs vs vs vs
0.75 4.75 7.18 4.00 6.43 2.43
1.10 4.46 7.24 3.46 5.91 1.79
3.203 3.691 3.309 3.465 3.125 3.125
1 1 1 2 2 3
TTABLE
2 3 4 3 4 4
SMM6 22
TAMHANE (T2, T3) METHOD
p=.0~=2 .875
p=.D1=3.575
MDIFF
FCAL
PROB.
0.75 4.75 7.18 4.00 6.43 2.43
0.19 6.95 19.27 4.37 13.37 1.72
0.90 0.002 <0.01 0.015 <0.01 0.19
COMPARISON
MDIFF
SOl
TCAL
1 vs 2 1 vs 3 1 vs 4
0.75 4.75 7.18
0.985 1.040 0.944
0.76 4.57 7.60
p=~:~~
p=O.OI
1.83
2.58
COMPARISON MDIFF
TCAL
0.75 4.75 7.18 4.00 6.43 2.43
1.10 4.46 7.24 3.46 5.91 1.79
1 1 1 2 2 3
vs vs vs vs vs vs
2 3 4 3 4 4
PROB FOR T3METHOO AT P.05 AT P=.OI 3.39 3.93 3.49 3.71 3.30 3.30
4.61 5.81 4.81 5.31 4.40 4.40
Prob for T2Method could be obtained by using ttables at Vij df and at a* level. BROWNFORSYTHE (BT) METHOD COMPARISON
MULTIPLE COMPARISON WITH CONTROL OR ZERO DOSE
Dunnett (D) Test = MVT3 22 Williams (W) Test = W3 , 22
TCAL
SAR4 22
SCHEFFE (S) TEST COMPARI SON
MDIFF
TTABLE
GABRIAL (G) METHOD COMPARISON
COMPARISON I I I 2 2 3
SPJOTVOLL  STOLINE (T') METHOD COMPARISON
ni_nj, ai 2_ a}
1 1 1 2 2
vs vs vs vs vs
2 3 4 3 4
3 vs 4
"T.25"
896
MDIFF
FCAL
P
0.75 4.75 7.18 4.00 6.43 2.43
0.40 6.63 17 .45 4.00 11.64 1.06
0.76 0.03 0.001 0.07 0.002 0.41
Dunnett, C. W. (1964), New TaDles for Multiple
NONPARAMETRIC TEST SUMMARY OF RESULTS
Comparisons with a Control, Biometrics, 20,
482491 GROUP
!!
RANKSUM
RANKMEAN
1
8 6 5 7
52.5 55.0 90.5 153.0
6.56 9.17 18.10 21.86
2
3 4
Dunnett, C. W. (1980a), Pairwise Multiple Comparisons in the Homogeneous Variance, Unequal Sample Size Case, J.A.S.A., 75, 789795
Dunnett, C. W. (1980b), Pairwise Multiple Comparisons in the Unequal Variance Case,
Correction for Ties (CT) Variance (52)
= =
0.988 57.8
J.A.S.A., 75, 796800 Dunn, O. J. (1964), Multiple Comparisons Using
KruskalWallis Test
with Ties Corrected _ x 2=18.90 p = 0.01
Rank Sums, Technometrics, 6, 241252
Dunn, O. J. (1974), On Multiple Tests and Confidence Intervals, Communications in
MULTIPLE COMPARISON WITH CONTROL (DUNN METHOD) COMPARISON
RANK MOIFF
SOl
ZCAL
P*
I vs 2 1 vs 3 1 vs 4
2.60 11.54 15.30
4.11 4.33 3.94
0.63 2.66 3.89
0.53 0.008 0.01
Statistics, 3, 101103
Games, P. A. (1977), An Improved t Table for Simultaneous Control on g Contrasts, J.A.S.A., 72, 531534 Games, P. A. and Howell, J. F. (1976), Pairwise Multiple Comparison Procedures With Unequal n's and/or Variances: A Monte Carlo Study, Journal
of Educational Statistics, I, 113125
FOR SHIRLEY'S TEST USE WILLIAM'S TABLE
Gabriel,
ALL MULTIPLE COMPARISONS (DUNN METHOD)
K.
R. (1978), A simple Method of
Multiple Comparisons of Means, J.A.S.A., 73,
724729
COMPARISON
RANK MOIFF
SOl
ZCAL
P*
1 vs 2 1 vs 3 1 vs 4 2 vs 3 2 vs 4 3 vs 4
2.60 11.54 15.30 8.93 12.69 3.76
4.11 4.33 3.94 4.60 4.23 4.45
0.63 2.66 3.B9 1.94 3.00 0.84
0.53 0.008 0.01 0.052 0.003 0.399
Hochberg, Y. (1974), Some Generalizations of the TMethod in Simultaneous Inference, Journal of Multiple Analysis, 4, 224234 Kramer, C. Y. (1956), Extension of Multiple Range Tests to Group Means With Unequal Number of Replications, Biometrics, 12, 307310
Kruskal, W. H. and Wallis, W. A. (1952), Use of Ranks in OneCriterion Variance Analysis,
• These pvalues are at .level and they should De modified Dy .*=./2p level to oDtain overall ~level
J.A.S.A., 47, 583621
of significance.
References
Miller, R. G., Jr. (1966), Simultaneous Statistical Inference, McGrawHill Book Co., New York
Bartholomew, D. J. (1961), A Test of Hom0geneity
Shirley, E. (1977), A NonParametric Equivalent of Williams Test for Contrasting Increasing Dose Levels of Treatment, Biometrics, 33,
of Means Under Restricted Alternatives,
J. R. Stat. Soc., B23, 239281
386389
Bailey, B. J. R. (1977), Tables of the Bonferroni t Statistics, J.A.S.A., 72, 469477
Stoline, M. R. (1978), Tables of the Studentized Augmented Range and Application to Problems of
Multiple Comparison, J.A.S.A., 73, 656660
Brown, M. B. and Forsythe, A. B. (1974), Robust Tests for the Equality of Variances, J.A.S.A., 69, 364367
Stoline, M. R. and Ury, H. K. (1979), Tables of the Studentized Maximum Modulus Distribution and an Application to Multiple Comparisons Among Means, Technometrics. 21, 8793
Cochran, W. G. (1964), Approximate Significance levels of the BehrensFisher Test, Biometrics,
20, 191195
Scheffe, H. (1953), A Method for Judging all Contrasts in the Analysis of Variance, Biometrika, 40, 87104
Dunnett, C. W. (1955), A Multiple Comparison Procedure for Comparing Several Jreatments With a Control, J.A.S.A., 50, 10961121
897
Sidak, Z. (1967), Rectangular Confidence Regions for the Means of Multivariate Normal
Distributions, J.A.S.A., 62, 626633 Spjotvoll, E. and Stoline, M. R. (1973), An Extension of the TMethod of Multiple Comparison to Include the Cases With Unequal Sample Sizes, J.A.S.A., 68, 975978 Tamhane, A. C. (1977), Multiple Comparisons in Model lOneWay ANOVA With Unequal Variances,
Communications in Statistics, A6, 1532 Tamhane, A. C. (1979), A Comparison of Procedures for Multiple Comparisons of Means With Unequal
Variances, J.A.S.A., 74, 471480 Tukey, J. W. (1953), The Problem of Multiple Comparisons, unpublished report, Princeton University, Department of Statistics
Welch, B. L. (1938), The Significance of the Difference Between Two Means When the Population Variances are Unequal, Biometrika, 29, 350362 Winer, B. J. (1971), Statistical Principles in Experimental Design, Second Edition, McGrawHill Book Co., New York Williams, D. A. (1971), A Test for Differences Between Treatment Means When Several Dose
Levels are Compared With a Zero Dose Control, Biometrics, 27, 101117 Williams, D. A. (1972), A Comparison of Several Dose Levels with a Zero Dose Control, Biometrics, 28, 519531 ACKNOWLEDGMENT: The author is grateful to Dr. Ran S. Sharma for his helpful comments.
898