This task view collects information on R packages for experimental design
and analysis of data from experiments. With a strong increase in the number of
relevant packages, packages that focus on analysis only and do not make relevant
contributions for design creation are no longer added to this task view.
Please feel free to
suggest enhancements, and please send information on new packages or major
package updates if you think they belong here. Contact details are given on my
Web page
.
Experimental design is applied in many areas, and methods have been tailored
to the needs of various fields. This task view starts out with a section on
the historically earliest application area, agricultural experimentation.
Subsequently, it covers the most general packages, continues with
specific sections on industrial experimentation, computer experiments, and
experimentation in the clinical trials contexts
(this section is going to be removed eventually; experimental design packages for
clinical trials will be integrated into the clinical trials task view),
and closes with a section on various special
experimental design packages that have been developed for other specific purposes.
Of course, the division into fields is not always clearcut, and some packages from
the more specialized sections can also be applied in general contexts.
You may also notice that my own experience is mainly from industrial experimentation
(in a broad sense), which may explain a somewhat biased view on things.
Experimental designs for agricultural and plant breeding experiments
Package
agricolae
is by far the mostused package from this task view (status: October 2017).
It offers extensive functionality on experimental design
especially for agricultural and plant breeding experiments, which can also be useful
for other purposes. It supports
planning
of lattice designs, factorial designs,
randomized complete block designs, completely randomized designs,
(Graeco)Latin square designs, balanced incomplete block designs and alpha designs.
There are also various
analysis
facilities for experimental data, e.g. treatment
comparison procedures and several nonparametric tests, but also some quite specialized
possibilities for specific types of experiments. Package
desplot
is made
for plotting the layout of agricultural experiments. Package
agridat
offers a large repository of useful agricultural data sets.
Experimental designs for general purposes
There are a few packages for creating and analyzing experimental designs
for general purposes: First of all, the standard (generalized) linear model
functions in the base package stats are of course very important for analyzing
data from designed experiments (especially functions
lm(),
aov()
and the methods and functions for the resulting linear model objects). These are
concisely explained in Kuhnert and Venables (2005, p. 109 ff.); Vikneswaran (2005)
points out specific usages for experimental design (using function
contrasts(),
multiple comparison functions and some convenience functions like
model.tables(),
replications()
and
plot.design()).
Lawson (2014) is a good introductory textbook on experimental design in R, which
gives many example applications.
Lalanne (2012) provides an R companion to the wellknown book by Montgomery (2005);
he so far covers approximately the first
ten chapters; he does not include R's design generation facilities, but mainly
discusses the analysis of existing designs.
Package
GAD
handles
general balanced analysis of variance models with fixed and/or random effects
and also nested effects (the latter can only be random); they quote Underwood (1997) for this work.
The package is quite valuable, as many
users have difficulties with using the R packages for handling random or mixed effects.
Package
granova
offers some interesting nonstandard graphical representations for results of simplystructured
experiments (oneway and twoway layouts, paired data), package
ez
aims at supporting
intuitive analysis and visualization of factorial experiments based on package "ggplot2".

Package
AlgDesign
creates full
factorial designs with or without additional quantitative variables, creates mixture
designs (i.e., designs where the levels of factors sum to 1=100%; lattice designs are created only) and creates
D, A, or Ioptimal designs exactly or approximately, possibly with blocking, using the Federov (1972) algorithm.

Package
skpr
also provides optimal designs (D, I, A, Alias, G, T, or E optimal); a
selection of the optimality criteria can also be used for the stepwise creation of splitplot designs. The package
can also assess the power of designs and display diagnostic plots. At the moment (October 2017), the
algorithms used are not yet documented.

Package
OptimalDesign
likewise calculates unblocked D, A, or Ioptimal designs
(they use "IVoptimal" instead of "Ioptimal") exactly or approximately,
treating quantitative variables only, including mixture designs;
this package uses different algorithms (e.g. Atkinson, Donev and Tobias 2007, Harman and Filova 2014),
some of which rely on the availability of the gurobi software
(
http://www.gurobi.com/
, free for academics and academic institutions)
and its accompanying R package "gurobi" (not on CRAN).

Package
ICAOD
implements the
"Imperialist Competitive Algorithm for Optimal Designs" for nonlinear models
according to Masoudi, Holling and Wong (2016).
Package
LDOD
implements locally Doptimal designs
for some nonlinear and generalized linear models and
package
PopED
provides optimal designs for nonlinear mixed effect models.

There are various further packages that deal with optimal designs of different types:
Package
rodd
provides Toptimal designs,
also called optimal discriminating designs (Dette, Melas and Shpilev 2013,
Dette, Melas and Guchenko 2014), Package
acebayes
calculates optimal Bayesian designs using an approximate coordinate exchange algorithm.
Further optimal design packages for
very specific purposes are listed at the end of this view.

Package
conf.design
allows
to create a design with certain interaction effects confounded with blocks (function
conf.design()) and allows to combine existing designs in several ways
(e.g., useful for Taguchi's inner and outer array designs in industrial experimentation).

Package
ibd
creates and analyses incomplete block designs. Packages
PGM2,
RPPairwiseDesign
and
CombinS
all produce designs
related to (resolvable) (partially) balanced incomplete block designs. Package
PBIBD
also provides experts with some series of partially balanced incomplete block designs.

Package
crossdes
creates and analyses crossover designs of various types (including
latin squares, mutually orthogonal latin squares and Youden squares) that can for example
be used in sensometrics.

Package
DoE.base
provides full factorial designs with or without blocking
(function
fac.design) and orthogonal arrays (function
oa.design)
for main effects experiments
(those listed by Kuhfeld 2009 up to 144 runs, plus a few additional ones).
There is also some functionality for assessing the quality of orthogonal arrays,
related to Groemping and Xu (2014) and Groemping (2017),
and some analysis functionality with halfnormal effects plots in
quite general form (Groemping 2015).
Package
DoE.base
also forms the basis of a suite of related packages:
together with
FrF2
(cf. below) and
DoE.wrapper, it provides the work horse
of the GUI package
RcmdrPlugin.DoE
(beta version; tutorial available in Groemping 2011),
which integrates
design of experiments functionality into the RCommander (package "Rcmdr", Fox 2005)
for the benefit of those R users who cannot or do not want to do command line programming.
The role of package
DoE.wrapper
in that suite is to wrap
functionality from other packages into the input and output structure of the package suite
(so far for response surface designs with package
rsm
(cf. also below),
design of computer experiments with packages
lhs
and
DiceDesign
(cf. also below),
and , and Doptimal designs with package
AlgDesign
(cf. also above).

Package
DoE.MIParray
creates optimized orthogonal arrays (or even supersaturated arrays)
for factorial experiments. Arrays created with this package can be used as input
to function oa.design of package
DoE.base.
Note, however, that the package is only useful in combination with at least one of the commercial
optimizers
Gurobi
(Rpackage gurobi delivered with the software) or
Mosek
(Rpackage Rmosek downloadable from the vendor (an outdated version is on CRAN)).

Package
dae
provides various utility functions around experimental design
and R factors, e.g. a randomization routine that can handle various nested structures
(according to Bailey 1981) and functions for combining several factors into one
or dividing one factor into several factors.
Furthermore, the package provides features for postprocessing
objects returned by the
aov()
function, e.g. extraction of Yates effects
for 2level experiments.

Package
daewr
accompanies the book
Design and Analysis of Experiments with R
by Lawson (2014) and does not only provide data sets from the book but also some standalone functionality
that is not available elsewhere in R, e.g. definitive screening designs.

Package
OPDOE
accompanies the book
Optimal Experimental Design with
R
by Rasch et al. (2011). It has some interesting sample size estimation functionality,
but is almost unusable without the book (the first edition of which I would not recommend buying).

Package
blockTools
assigns units to blocks in order to end up with homogeneous sets
of blocks in case of too small block sizes; package
blocksdesign
permits
the creation of nested block structures.

There are several packages for determining sample sizes in experimental contexts,
some of them quite general, others very specialized. All of these are mentioned here:
packages
powerAnalysis,
powerbydesign
and
easypower
deal with estimating
the power, sample size and/or effect size for factorial experiments. Package
JMdesign
deals with the power for the special situation of jointly modeling longitudinal and survival data,
package
powerGWASinteraction
with the power for interactions in genome wide association studies,
package
ssizeRNA
with sample size for RNA sequencing experiments, and package
ssize.fdr
for sample sizes in microarray experiments (requesting a certain power while limiting the false discovery rate).
Experimental designs for industrial experiments
Some further packages especially handle designs for industrial experiments
that are often highly fractionated, intentionally confounded and have few extra degrees
of freedom for error.
Fractional factorial 2level designs are particularly important in industrial
experimentation.

Package
FrF2
(Groemping 2014) is the most comprehensive R package for
their creation. It generates regular Fractional Factorial
designs for factors with 2 levels as well as PlackettBurman type screening designs.
Regular fractional factorials default to maximum resolution minimum aberration designs
and can be customized in various ways, supported by an
incorporated catalogue of designs (including the designs catalogued by Chen, Sun and Wu 1993,
and further larger designs catalogued in Block and Mee 2005 and Xu 2009;
the additional package
FrF2.catlg128
provides a very large complete catalogue
for resolution IV 128 run designs with up to 23 factors for special purposes).
Analysiswise,
FrF2
provides simple graphical analysis tools (normal and halfnormal effects plots
(modified from
BsMD, cf. below), main effects
plots and interaction plot matrices similar to those in Minitab software, and a cube
plot for the combinations of three factors). It can also show the alias structure
for regular fractional factorials of 2level factors, regardless whether they have been
created with the package or not.
Fractional factorial 2level plans can also be created by other R packages,
namely
BHH2
and
qualityTools
(but do not use function pbDesign from
version 1.54 of that package!), or with a little bit more complication
by packages
conf.design
or
AlgDesign.
Package
ALTopt
provides optimal designs
for accelerated life testing.

Package
BHH2
accompanies the 2nd edition of the book by Box, Hunter and Hunter
and provides various of its data sets. It can generate full and fractional factorial
twoleveldesigns from a number of factors and a list of defining relations
(function
ffDesMatrix(), less comfortable than package FrF2).
It also provides several functions for analyzing data from 2level factorial
experiments: The function anovaPlot assesses effect sizes relative to residuals, and
the function
lambdaPlot()
assesses the effect of BoxCox transformations on
statistical significance of effects.

BsMD
provides Bayesian charts as
proposed by Box and Meyer (1986) as well as effects plots (normal, halfnormal and
Lenth) for assessing which effects are active in a fractional factorial experiment
with 2level factors.

Package
unrepx
provides a battery of methods for the assessment
of effect estimates from unreplicated factorial experiments, including many of
the effects plots also present in other packages, but also further possibilities.

The small package
FMC
provides factorial designs with minimal
number of level changes; the package does not take any measures to account for the
statistical implications this may imply. Thus, using this package must be considered very risky for many
experimental situations, because in many experiments some variability is caused
by level changes. For such situations (and they are
the rule rather than the exception), minimizing the level changes without taking
precautions in the analysis will yield misleading results.

Package
pid
accompanies an online book by Dunn (20102016) and also makes
heavy use of the Box, Hunter and Hunter book; it provides various data sets, which are
mostly from fractional factorial 2level designs.
Apart from tools for planning and analysing factorial designs, R also offers support for
response surface optimization for quantitative factors (cf. e.g. Myers and Montgomery 1995):

Package
rsm
supports sequential
optimization with first order and second order response surface models (central composite
or BoxBehnken designs), offering
optimization approaches like steepest ascent and visualization of the response
function for linear model objects. Also, coding for response surface investigations is
facilitated.

Package
DoE.wrapper
enhances design creation from package
rsm
with the possibilities of automatically choosing the cube portion of central
composite designs and of augmenting
an existing (fractional) factorial 2level design with a star portion.

The small package
rsurface
provides rotatable central composite designs
for which the user specifies the minimum and maximum of the experimental variables instead
of the corner points of the cube.

The small package
minimalRSD
provides central composite and BoxBehnken designs
with minimal number of level changes; the package does not take any measures to account for the
statistical implications this may imply. Thus, using this package must be considered very risky for many
experimental situations, because in many experiments some variability is caused
by level changes. For such situations (and they are
the rule rather than the exception), minimizing the level changes without taking
precautions in the analysis will yield misleading results.

Package
OptimaRegion
provides functionality for inspecting the optimal
region of a response surface for quadratic polynomials and thinplate spline models
and can compute a confidence interval for the distance between two optima.

Package
Vdgraph
implements a variance dispersion graph (Vining 1993) for response
surface designs created by package
rsm. Packages
VdgRsm
and
vdg
provide similar functionality with more variety.

Package
qualityTools
can also create central composite designs
and can visualize response surfaces.

Package
EngrExpt
provides a collection of data sets from the book
Introductory Statistics for Engineering Experimentation
by Nelson, Coffin and Copeland (2003).
In some industries, mixtures of ingredients are important; these require special designs,
because the quantitative factors have a fixed total.
Mixture designs are handled by packages
AlgDesign
(function
gen.mixture,
lattice designs),
qualityTools
(function
mixDesign,
lattice designs and simplex centroid designs), and
mixexp
(several small functions for simplex centroid,
simplex lattice and extreme vertices designs as well as for plotting).
Occasionally, supersaturated designs can be useful.
The two small packages
mkssd
and
mxkssd
provide fixed level and mixed level
kcirculant supersaturated designs. The aforementioned package
DoE.MIParray
can also
provide (small!) supersaturated arrays (by choosing resolution II), but requires the presence of at least one
of the commercial optimizers
Gurobi
or
Mosek
.
Experimental designs for computer experiments
Computer experiments with quantitative factors require special types of
experimental designs: it is often possible to include many different
levels of the factors, and replication will usually not be beneficial. Also, the
experimental region is often too large to assume that a linear or quadratic model adequately
represents the phenomenon under investigation. Consequently, it is desirable to fill
the experimental space with points as well as possible (spacefilling designs) in such
a way that each run provides additional information even if some factors turn out to be
irrelevant.
The
lhs
package provides latin hypercube designs for this purpose.
Furthermore, the package provides ways to analyse such computer experiments with
emphasis on what followup experiments to conduct. Another package with similar orientation
is the
DiceDesign
package, which adds further ways to construct spacefilling
designs and some measures to assess the quality of designs for computer experiments. The
package
DiceKriging
provides the kriging methodology which is often used for
creating meta models from computer experiments, the package
DiceEval
creates
and evaluates meta models (among others Kriging ones), and the package
DiceView
provides facilities for viewing sections of multidimensional meta models.
Package
MaxPro
provides maximum projection designs as introduced by
Joseph, Gul and Ba(2015). Package
SLHD
provides optimal sliced latin
hypercube designs according to Ba et al. (2015), package
sFFLHD
provides
sliced full factorialbased latin hypercube designs according to
Duan et al. (2017).
Package
minimaxdesign
provides
minimax designs and minimax projection designs according to Mak and Joseph (2016).
Package
tgp
is another package dedicated to planning and analysing
computer experiments. Here, emphasis is on Bayesian methods.
The package can for example be used with various kinds of (surrogate) models for
sequential optimization, e.g. with an expected improvement criterion for optimizing a noisy
blackbox target function. Packages
plgp
and
dynaTree
enhance the
functionality offered by
tgp
with particle learning facilities and learning for
dynamic regression trees.
Package
BatchExperiments
is also designed for computer
experiments, in this case specifically for experiments with algorithms to be run
under different scenarios. The package is described in a technical report by
Bischl et al. (2012).
Experimental designs for clinical trials
This task view only covers specific design of experiments packages
(which will eventually also be removed here); there may be some
grey areas. Please, also consult the
ClinicalTrials
task view.

Package
experiment
contains tools for clinical experiments,
e.g., a randomization tool, and it provides a few special analysis options for clinical
trials.

Package
ThreeArmedTrials
provides design and analysis tools for threearmed
superiority or noninferiority trials. Beside the standard functionality, the package
includes the negative Binomial response situation discussed in Muetze et al. (2016).

Package
gsDesign
implements group sequential designs, package
GroupSeq
gives a GUI for probability spending in such designs,
package
OptGS
nearoptimal balanced group sequential designs.
Package
gsbDesign
evaluates operating characteristics for group sequential Bayesian designs.
Package
gset
handles group sequential equivalence testing. Package
seqDesign
handles group sequential twostage treatment efficacy trials with timetoevent endpoints.

Package
binseqtest
handles sequential single arm binary response trials.

Package
asd
implements adaptive seamless designs (see e.g. Parsons et al. 2012).

Package
OptInterim
is for two and threestage designs for longterm binary endpoints.

Packages
bcrm
and
crmPack
offer Bayesian CRM designs.

Package
MAMS
offers designs for multiarm multi stage studies,
BayesMAMS
provides a
Bayesian sample size calculations for these.

Package
BOIN
provides Bayesian optimal interval designs, which are used in phase I clinical
trials for finding the maximum tolerated dose.

The
DoseFinding
package provides functions for the design and analysis
of dosefinding experiments (for example pharmaceutical Phase II clinical trials);
it combines the facilities of the "MCPMod" package (maintenance discontinued;
described in Bornkamp, Pinheiro and Bretz 2009) with a special type of optimal designs for
dose finding situations (MEDoptimal designs, or Doptimal designs, or a mixture of both;
cf., Dette et al. 2008).

Package
VNM
provides multiobjective optimal designs for simultaneously optimizing
inference about the shape of the doseresponse curve, ED50 and minimum effective dose (MED) for certain classes of
logistic models.

Package
TEQR
provides toxicity equivalence range designs (Blanchard and Longmate 2010) for phase I clinical trials,
package
pipe.design
socalled
product of independent beta probabilities dose escalation
(PIPE)
designs for phase I. Package
dfpk
implements a Bayesian dosefinding design
using pharmacokinetics for phase I trials. Package
dfcrm
provides designs
for classical or TITE continual reassessment trials in phase I.

Packages
dfcomb
and
dfmta
provide
phase I/II adaptive dosefinding designs for
combination studies or singleagent molecularly targeted agent, respectively.

Packages
ph2bayes
and
ph2bye
are concerned with
Bayesian single arm phase II trials.

Package
sp23design
claims to offer seamless integration of phase II to III.
Experimental designs for special purposes
Various further packages handle special situations in experimental design:

Package
desirability
provides ways to combine several target criteria into a desirability function in order to simplify
multicriteria analysis; desirabilities are also offered as part of package
qualityTools.

osDesign
designs studies nested in observational studies,
designmatch
can also be useful for this purpose.

qtlDesign
is for quantitative trait locus designs,

toxtestD
creates optimal designs for binary toxicity tests,

hiPOD
provides optimal designs for pooled next generation sequencing experiments,

designGG
creates optimal designs for genetical genomics experiments (see Li et al. 2009),

packages
optbdmaeAT,
optrcdmaeAT
and
soptdmaeA
provide optimal block designs,
optimal rowcolumn designs, and sequential optimal or nearoptimal block or rowcolumn designs for
twocolour cDNA microarray experiments, with optimality according to an A, MV, D or Ecriterion.

Package
docopulae
implements optimal designs for copula models according to
Perrone and Mueller (2016),

optDesignSlopeInt
provides an optimal design for the estimation of the ratio of slope to intercept, and

Package
MBHdesign
provides spatially balanced designs,
allowing the inclusion of prespecified (legacy) sites. The more elaborate package
geospt
allows to optimize spatial networks of sampling points (see e.g. Santacruz, Rubiano and Melo 2014).

Package
SensoMineR
contains special designs for
sensometric studies, e.g., for the triangle test.

Package
choiceDes
creates
choice designs with emphasis on discrete choice models and MaxDiff functionality; it is based on optimal designs.
Package
support.CEs
provides tools for creating stated choice designs
for market research investigations, based on orthogonal arrays.

Package
odr
creates optimal designs for cluster randomized trials
under condition and unitspecific cost structures.

Package
bioOED
offers sensitivity analysis and optimal design for
microbial inactivation.
Key references for packages in this task view

Atkinson, A.C. and Donev, A.N. (1992).
Optimum Experimental Designs.
Oxford: Clarendon Press.

Atkinson, A.C., Donev, A.N. and Tobias, R.D. (2007).
Optimum Experimental Designs, with SAS.
Oxford University Press, Oxford.

Ba,S., Brenneman, W.A. and Myers, W.R. (2015). Optimal Sliced Latin Hypercube Designs.
Technometrics
57
479487.

Bailey, R.A. (1981). A unified approach to design of experiments.
Journal of the Royal Statistical Society, Series A
144
214223.

Ball, R.D. (2005). Experimental Designs for Reliable Detection of Linkage
Disequilibrium in Unstructured Random Population Association Studies.
Genetics
170
859873.

Bischl, B., Lang, M., Mersmann, O., Rahnenfuehrer, J. and Weihs, C. (2012).
Computing on high performance clusters with R: Packages BatchJobs and
BatchExperiments
.
Technical Report 1/2012
, TU Dortmund, Germany.

Blanchard, M.S. and Longmate, J.A. (2010).
Toxicity equivalence range design (TEQR): A practical Phase I design.
Contemporary Clinical Trials
doi:10.1016/j.cct.2010.09.011.

Block, R. and Mee, R. (2005). Resolution IV Designs with 128 Runs.
Journal of Quality Technology
37
282293.

Bornkamp B., Pinheiro J. C., and Bretz, F. (2009).
MCPMod: An R Package for the Design and Analysis of DoseFinding Studies
.
Journal of Statistical Software
29
(7) 123.

Box G. E. P, Hunter, W. C. and Hunter, J. S. (2005).
Statistics for Experimenters
(2nd edition). New York: Wiley.

Box, G. E. P and R. D. Meyer (1986). An Analysis for Unreplicated Fractional
Factorials.
Technometrics
28
1118.

Box, G. E. P and R. D. Meyer (1993). Finding the Active Factors in Fractionated Screening
Experiments.
Journal of Quality Technology
25
94105.

Chasalow, S., Brand, R. (1995). Generation of Simplex Lattice Points.
Journal of the Royal Statistical Society, Series C
44
534545.

Chen, J., Sun, D.X. and Wu, C.F.J. (1993). A catalogue of 2level and 3level orthogonal arrays.
International Statistical Review
61
131145.

Consonni, G. and Deldossi, L. (2015), Objective Bayesian model discrimination in followup
experimental designs DOI 10.1007/s1174901504613. TEST.

Collings, B. J. (1989). Quick Confounding.
Technometrics
31
107110.

Cornell, J. (2002).
Experiments with Mixtures
. Third Edition. Wiley.

Crabbe, M., Akinc, D. and Vandebroek, M. (2014). Fast algorithms to
generate individualized designs for the mixed logit choice model.
Transportation Research Part B: Methodological
60
, 115.

Daniel, C. (1959). Use of Half Normal Plots in Interpreting Two Level Experiments.
Technometrics
1
311340.

Derringer, G. and Suich, R. (1980). Simultaneous Optimization of Several Response Variables.
Journal of Quality Technology
12
214219.

Dette, H., Bretz, F., Pepelyshev, A. and Pinheiro, J. C. (2008).
Optimal Designs for Dose Finding Studies.
Journal of the American Statisical Association
103
12251237.

Dette, H., Melas, V.B. and Shpilev, P. (2013). Robust Toptimal discriminating designs.
The Annals of Statistics
41
16931715.

Dette H., Melas V.B. and Guchenko R. (2014). Bayesian Toptimal discriminating designs.
ArXiv link
.

Duan, W., Ankenman, B.E. Sanchez, S.M. and Sanchez, P.J. (2017).
Sliced Full FactorialBased Latin Hypercube Designs as a Framework for a
Batch Sequential Design Algorithm.
Technometrics
59
, 1122.

Dunn, K. (20102016).
Process Improvement Using Data
.
Online book.

Federov, V.V. (1972).
Theory of Optimal Experiments.
Academic Press, New York.

Fox, J. (2005).
The R Commander:
A BasicStatistics Graphical User Interface to R
.
Journal of Statistical Software
14
(9) 142.

Gramacy, R.B. (2007).
tgp: An R Package
for Bayesian Nonstationary, Semiparametric Nonlinear Regression and Design by Treed Gaussian Process Models
.
Journal of Statistical Software
19
(9) 146.

Groemping, U. (2011).
Tutorial for designing experiments using the R package RcmdrPlugin.DoE
.
Reports in Mathematics, Physics and Chemistry
,
Department II, Beuth University of Applied Sciences Berlin.

Groemping, U. (2014). R Package FrF2 for Creating and Analysing Fractional Factorial 2Level Designs.
Journal of Statistical Software
56
(1) 156.

Groemping, U. (2015). Augmented Half Normal Effects Plots in the Presence
of a Few Error Degrees of Freedom.
Quality and Reliability Engineering International
31
, 11851196. DOI: 10.1002/qre.1842.

Groemping, U. (2017). Frequency Tables for the Coding Invariant Quality Assessment of Factorial Designs.
IISE Transactions
49
, 505517.

Groemping, U. and Xu, H. (2014). Generalized resolution for orthogonal arrays.
The Annals of Statistics
42
918939.

Harman R., Filova L. (2014): Computing efficient exact designs of experiments using integer
quadratic programming,
Computational Statistics and Data Analysis
71
11591167

Hoaglin D., Mosteller F. and Tukey J. (eds., 1991).
Fundamentals of Exploratory Analysis of Variance
.
Wiley, New York.

Jones, B. and Kenward, M.G. (1989).
Design and Analysis of CrossOver Trials
. Chapman and
Hall, London.

Johnson, M.E., Moore L.M. and Ylvisaker D. (1990). Minimax and maximin distance designs.
Journal of Statistical Planning and Inference
26
131148.

Joseph, V. R., Gul, E., and Ba, S. (2015). Maximum Projection Designs for
Computer Experiments.
Biometrika
102
371380.

Kuhfeld, W. (2009). Orthogonal arrays. Website courtesy of SAS Institute Inc., accessed August 4th 2010.
URL
http://support.sas.com/techsup/technote/ts723.html
.

Kuhnert, P. and Venables, B. (2005)
An Introduction to R: Software for Statistical
Modelling & Computing
. URL
http://CRAN.Rproject.org/doc/contrib/Kuhnert+VenablesR_Course_Notes.zip
.
(PDF document (about 360 pages) of lecture notes in combination with the data sets and R scripts)

Kunert, J. (1998). Sensory Experiments as Crossover Studies.
Food Quality and Preference
9
243253.

Lalanne, C. (2012). R Companion to Montgomerys Design and Analysis of Experiments.
Manuscript, downloadable at URL
http://www.aliquote.org/articles/tech/dae/dae.pdf
.
(The file accompanies the book by Montgomery 2005 (cf. below).)

Lawson, J. (2014).
Design and Analysis of Experiments with R.
Chapman and Hall/CRC, Boca Raton.

Lenth, R.V. (1989). Quick and Easy Analysis of Unreplicated Factorials.
Technometrics
31
469473.

Lenth, R.V. (2009).
ResponseSurface Methods in R, Using rsm
.
Journal of Statistical Software
32
(7) 117.

Y. Li, M. Swertz, G. Vera, J. Fu, R. Breitling, and R.C. Jansen.
designGG:
An Rpackage and Web tool for the optimal design of genetical genomics experiments.
BMC Bioinformatics
10
:188

Mak, S., and Joseph, V.R. (2016). Minimax designs using clustering.
Journal of Computational and
Graphical Statistics
. In revision.

Martens, H., Mage, I., Tondel, K., Isaeva, J., Hoy, M. and Saebo, S. (2010).
Multilevel binary replacement (MBR) design for computer experiments in highdimensional
nonlinear systems,
J. Chemom.
24
748756.

Masoudi, E., Holling, H. and Wong, W.K. (2016). Application of imperialist competitive algorithm
to find minimax and standardized maximin optimal designs.
Computational Statistics and Data Analysis
,
in press. DOI: 10.1016/j.csda.2016.06.014

Mee, R. (2009).
A Comprehensive Guide to Factorial TwoLevel Experimentation.
Springer, New York.

Montgomery, D. C. (2005, 6th ed.).
Design and Analysis of Experiments.
Wiley, New York.

Muetze,T., Munk, A. and Friede, T. (2016). Design and analysis of threearm trials with negative
binomially distributed endpoints.
Statistics in Medicine
35
(4) 505521.

Myers, R. H. and Montgomery, D. C. (1995).
Response Surface Methodology: Process and Product
Optimization Using Designed Experiments.
Wiley, New York.

Nelson, P.R., Coffin, M. and Copeland, K.A.F. (2003).
Introductory Statistics for Engineering
Experimentation.
Academic Press, San Diego.

Parsons N, Friede T, Todd S, Valdes Marquez E, Chataway J, Nicholas R, Stallard N. (2012).
An R package for implementing simulations for seamless phase II/III clinicals trials using
early outcomes for treatment selection.
Computational Statistics and Data Analysis
56
,
11501160.

Perrone, E. and Mueller, W.G. (2016) Optimal designs for copula models,
Statistics
50
(4), 917929. DOI: 10.1080/02331888.2015.1111892

Plackett, R.L. and Burman, J.P. (1946). The design of optimum multifactorial experiments.
Biometrika
33
305325.

Rasch, D., Pilz, J., Verdooren, L.R. and Gebhardt, A. (2011).
Optimal Experimental
Design with R.
Chapman and Hall/CRC. (caution, does not live up to its title!)

Rosenbaum, P. (1989). Exploratory Plots for Paired Data.
The American Statistician
43
108109.

Sacks, J., Welch, W.J., Mitchell, T.J. and Wynn, H.P. (1989). Design and analysis of computer experiments.
Statistical Science
4
409435.

Santacruz, A., Rubiano, Y., Melo, C., 2014. Evolutionary optimization of spatial sampling networks
designed for the monitoring of soil carbon. In: Hartemink, A., McSweeney, K. (Eds.).
Soil Carbon.
Series: Progress in Soil Science. (pp. 7784). Springer, New York.

Santner T.J., Williams B.J. and Notz W.I. (2003).
The Design and Analysis of Computer Experiments.
Springer, New York.

Sen S, Satagopan JM and Churchill GA (2005). Quantitative Trait Locus Study Design from an Information
Perspective.
Genetics
170
447464.

Stein, M. (1987). Large Sample Properties of Simulations Using Latin Hypercube Sampling.
Technometrics
29
143151.

Stocki, R. (2005). A Method to Improve Design Reliability Using Optimal Latin Hypercube Sampling.
Computer Assisted Mechanics and Engineering Sciences
12
87105.

Underwood, A.J. (1997).
Experiments in Ecology: Their Logical Design and Interpretation Using Analysis of Variance.
Cambridge University Press, Cambridge.

Vikneswaran (2005).
An R companion to "Experimental Design".
URL
http://CRAN.Rproject.org/doc/contrib/VikneswaranED_companion.pdf
.
(The file accompanies the book "Experimental Design with Applications in Management, Engineering
and the Sciences" by Berger and Maurer, 2002.)

Vining, G. (1993). A Computer Program for Generating Variance Dispersion Graphs.
Journal of Quality Technology
25
4558. Corrigendum in the same volume, pp. 333335.

Xu, H. (2009). Algorithmic Construction of Efficient Fractional Factorial Designs With Large Run Sizes.
Technometrics
51
262277.

Yin, J., Qin, R., Ezzalfani, M., Sargent, D. J., and Mandrekar, S. J. (2017).
A Bayesian dosefinding design incorporating toxicity data from multiple treatment cycles.
Statistics in Medicine
36
, 6780. doi: 10.1002/sim.7134.