crew: A Distributed Worker Launcher Framework

In computationally demanding analysis projects, statisticians and data scientists asynchronously deploy long-running tasks to distributed systems, ranging from traditional clusters to cloud services. The 'NNG'-powered 'mirai' R package by Gao (2023) <doi:10.5281/zenodo.7912722> is a sleek and sophisticated scheduler that efficiently processes these intense workloads. The 'crew' package extends 'mirai' with a unifying interface for third-party worker launchers. Inspiration also comes from packages. 'future' by Bengtsson (2021) <doi:10.32614/RJ-2021-048>, 'rrq' by FitzJohn and Ashton (2023) <https://github.com/mrc-ide/rrq>, 'clustermq' by Schubert (2019) <doi:10.1093/bioinformatics/btz284>), and 'batchtools' by Lang, Bischel, and Surmann (2017) <doi:10.21105/joss.00135>.

Version: 0.9.1
Depends: R (≥ 4.0.0)
Imports: cli (≥ 3.1.0), data.table, getip, later, mirai (≥ 0.12.0), nanonext (≥ 0.12.0), processx, promises, ps, R6, rlang, stats, tibble, tidyselect, tools, utils
Suggests: knitr (≥ 1.30), markdown (≥ 1.1), rmarkdown (≥ 2.4), testthat (≥ 3.0.0)
Published: 2024-03-25
Author: William Michael Landau ORCID iD [aut, cre], Daniel Woodie [ctb], Eli Lilly and Company [cph]
Maintainer: William Michael Landau <will.landau.oss at gmail.com>
BugReports: https://github.com/wlandau/crew/issues
License: MIT + file LICENSE
URL: https://wlandau.github.io/crew/, https://github.com/wlandau/crew
NeedsCompilation: no
Language: en-US
Materials: NEWS
In views: HighPerformanceComputing
CRAN checks: crew results

Documentation:

Reference manual: crew.pdf
Vignettes: Controller groups
Introduction to crew
Launcher plugins
Integration with promises
Known risks of crew
Asynchronous Shiny apps

Downloads:

Package source: crew_0.9.1.tar.gz
Windows binaries: r-devel: crew_0.9.1.zip, r-release: crew_0.9.1.zip, r-oldrel: crew_0.9.1.zip
macOS binaries: r-release (arm64): crew_0.9.1.tgz, r-oldrel (arm64): crew_0.9.1.tgz, r-release (x86_64): crew_0.9.1.tgz
Old sources: crew archive

Reverse dependencies:

Reverse imports: crew.aws.batch, crew.cluster
Reverse suggests: targets

Linking:

Please use the canonical form https://CRAN.R-project.org/package=crew to link to this page.