Launcher plugins

About

crew lets you write custom launchers for different types of workers that connect over the local network. This flexibility can extend crew to platforms like SLURM, AWS Batch, and Kubernetes.

Vision

These launcher plugins need not become part of the crew package itself. You can write your plugin in a simple R script, or you write it in a custom R package that depends on crew. Published packages with launcher plugins are powerful extensions that enhance crew for the entire open-source community. See R Packages by Hadley Wickham and Jenny Bryan for how to write an R package.

Scope

This vignette demonstrates how to write a crew launcher plugin. It assumes prior familiarity with R6 classes and the computing platform of your plugin.

How it works

To create your own launcher plugin, write an R6 subclass of crew_class_launcher with a launch_worker() method analogous the one in the local process launcher. launch_worker() must accept the same arguments as the local process launch_worker() method, generate a call to crew_worker(), and then submit a new job or process to run that call.

Termination

We recommend you implement an optional terminate_worker() method. Although mirai has its own way of terminating workers, it only works if the worker already connected, and it cannot reach workers that fail to connect and hang in a crashed state. An optional terminate_worker() method in your crew launcher plugin is extra assurance that these workers will exit.

If you implement a custom terminate_worker() method, it must not throw an error (and should not throw a warning or message) if the worker is already terminated. In addition, it must accept a handle that identifies the worker, and this handle must be the return value of the previous call to launch_worker(). A handle can be any kind of R object: a process ID, a job name, an R6 object returned by callr::r_bg(), etc.

Example

The following is a custom custom launcher class whose workers are local R processes on Unix-like systems.¹

custom_launcher_class <- R6::R6Class(
  classname = "custom_launcher_class",
  inherit = crew::crew_class_launcher,
  public = list(
    launch_worker = function(call, launcher, worker, instance) {
      bin <- file.path(R.home("bin"), "R")
      processx::process$new(command = bin, args = c("-e", call))
    },
    terminate_worker = function(handle) {
      handle$kill()
    }
  )
)

Inside launch_worker(), the processx::process$new(command = bin, args = c("-e", call)) line runs the crew_worker() call in an external R process. This process runs in the background, connects back to crew and mirai over the local network, and accepts the tasks you push to the controller. processx::process$new() also returns a handle which the terminate_worker() method can use to force-terminate the process if appropriate. mirai has its own way to terminate workers, so a terminate_worker() method is not strictly required, but it is a useful safeguard in case a worker hangs in a crashed state before it establishes a connection.

Every launch_worker() method must accept arguments call, launcher, worker, and instance. The method does not actually need to use all these arguments, but they must be present in the function signature.

• call: a text string with a call to crew_worker().
• launcher: the name of the launcher. Each worker instance launched with the launcher will receive the same launcher argument in launch_worker().
• worker: integer index of the worker in the current list of workers, always between 1 and the maximum number of concurrent workers.
• instance: a text string that uniquely identifies each worker instance. The value of instance can be used to set informative job names, and the values of launcher and worker can provide additional context.

To see what the call object looks like, create a new launcher and run the call() method.

library(crew)
launcher <- crew_launcher_local()
launcher$call(
  socket = "ws://127.0.0.1:5000/3/aa9c59ea",
  launcher = "my_launcher",
  worker = 3L,
  instance = "aa9c59ea"
)
#> [1] "crew::crew_worker(settings = list(url = \"ws://127.0.0.1:5000/3/aa9c59ea\", maxtasks = Inf, idletime = Inf, walltime = Inf, timerstart = 0L, exitlinger = 1000, cleanup = 1L, asyncdial = FALSE), launcher = \"my_launcher\", worker = 3L, instance = \"aa9c59ea\")"

Helper

It is useful to have a helper function that creates controllers with your custom launcher. It should:

1. Accept all the same arguments as crew_controller_local().
2. Create a router object using crew_router().
3. Create a launcher object with the new() method of your custom launcher class.
4. Create a new controller using crew_controller().
5. Scan the controller for obvious errors using the validate() method of the controller.

Feel free to borrow from the crew_controller_local() source code. For packages, you can use the @inheritParams roxygen2 tag to inherit the documentation of all the arguments instead of writing it by hand. You may want to adjust the default arguments based on the specifics of your platform, especially seconds_launch if workers take a long time to launch.

#' @title Create a controller with the custom launcher.
#' @export
#' @description Create an `R6` object to submit tasks and
#'   launch workers.
#' @inheritParams crew::crew_controller_local
crew_controller_custom <- function(
  name = "custom controller name",
  workers = 1L,
  host = NULL,
  port = NULL,
  seconds_launch = 30,
  seconds_interval = 0.01,
  seconds_timeout = 5,
  seconds_idle = Inf,
  seconds_wall = Inf,
  seconds_exit = 1,
  tasks_max = Inf,
  tasks_timers = 0L,
  reset_globals = TRUE,
  reset_packages = FALSE,
  reset_options = FALSE,
  garbage_collection = FALSE,
  auto_scale = "demand"
) {
  router <- crew::crew_router(
    name = name,
    workers = workers,
    host = host,
    port = port,
    seconds_interval = seconds_interval,
    seconds_timeout = seconds_timeout
  )
  launcher <- custom_launcher_class$new(
    name = name,
    seconds_launch = seconds_launch,
    seconds_interval = seconds_interval,
    seconds_timeout = seconds_timeout,
    seconds_idle = seconds_idle,
    seconds_wall = seconds_wall,
    seconds_exit = seconds_exit,
    tasks_max = tasks_max,
    tasks_timers = tasks_timers,
    reset_globals = reset_globals,
    reset_packages = reset_packages,
    reset_options = reset_options,
    garbage_collection = garbage_collection
  )
  controller <- crew::crew_controller(
    router = router,
    launcher = launcher,
    auto_scale = auto_scale
  )
  controller$validate()
  controller
}

Informal testing

Before you begin testing, please begin monitoring local processes and remote jobs on your platform. In the case of the above crew launcher which only creates local processes, it is sufficient to start htop and filter for R processes, or launch a new R session to monitor the process table from ps::ps(). However, for more ambitious launchers that submit workers to e.g. AWS Batch, you may need to open the CloudWatch dashboard, then view the AWS billing dashboard after testing.

When you are ready to begin testing, try out the example in the README, but use your your custom controller helper instead of crew_controller_local().

First, create and start a controller. You may wish to monitor local processes on your computer to make sure the mirai dispatcher starts.

library(crew)
controller <- crew_controller_custom(workers = 2)
controller$start()

Try pushing a task that gets the local IP address and process ID of the worker instance.

controller$push(
  name = "get worker IP address and process ID",
  command = paste(getip::getip(type = "local"), ps::ps_pid())
)

Wait for the task to complete and look at the result.

controller$wait()
result <- controller$pop()
result$result[[1]]
#> [1] "192.168.0.2 27336"

Please use the result to verify that the task really ran on a worker as intended. The process ID above should agree with the one from the handle (except on Windows because the actual R process may be different from the R.exe process created first). In addition, if the worker is running on a different computer, the worker IP address should be different than the local IP address. Since our custom launcher creates local processes, the IP addresses are the same in this case, but they should be different for a SLURM or AWS Batch launcher.

getip::getip(type = "local")
#> "192.168.0.2"
controller$launcher$workers$handle[[1]]$get_pid()
#> [1] 27336

If you did not set any timeouts or task limits, the worker that ran the task should still be running and connected. The other worker had no tasks, so it did not need to start an instance.

controller$summary(columns = starts_with("worker"))
#> # A tibble: 2 × 5
#>   worker_connected worker_launches worker_instances worker_socket                      
#>   <lgl>                      <int>            <int> <chr>                              
#> 1 TRUE                           1                1 ws://10.0.0.9:49612/1/bbc1a2241dd8…
#> 2 FALSE                          0                0 ws://10.0.0.9:49612/2/c488c874a07e…

When you are done, terminate the controller. This terminates the mirai dispatcher process and the crew workers.

controller$terminate()

Finally, use the process monitoring interface of your computing platform or operating system to verify that all mirai dispatchers and crew workers are terminated.

Load testing

If the informal testing succeeded, we recommend you scale up testing to more ambitious scenarios. As one example, you can test that your workers can auto-scale and quickly churn through a large number of tasks.

library(crew)
controller <- crew_controller_custom(
  seconds_idle = 2L,
  workers = 2L
)
controller$start()
# Push 100 tasks
for (index in seq_len(100L)) {
  name <- paste0("task_", index)
  controller$push(name = name, command = index, data = list(index = index))
  message(paste("push", name))
}
# Wait for the tasks to complete.
controller$wait()
# Wait for the workers to idle out and exit on their own.
crew_retry(
  ~all(controller$summary()$worker_connected == FALSE),
  seconds_interval = 1,
  seconds_timeout = 60
)
# Do the same for 100 more tasks.
for (index in (seq_len(100L) + 100L)) {
  name <- paste0("task_", index)
  controller$push(name = name, command = index, data = list(index = index))
  message(paste("push", name))
}
controller$wait()
crew_retry(
  ~all(controller$summary()$worker_connected == FALSE),
  seconds_interval = 1,
  seconds_timeout = 60
)
# Collect the results.
results <- NULL
while (!is.null(result <- controller$pop(scale = FALSE))) {
  if (!is.null(result)) {
    results <- dplyr::bind_rows(results, result)
  }
}
# Check the results
all(sort(unlist(results$result)) == seq_len(200L))
#> [1] TRUE
length(unique(results$instance))
#> [1] 4
# View worker and task summaries.
View(controller$summary())
# Terminate the controller.
controller$terminate()
# Now outside crew, verify that the mirai dispatcher
# and crew workers successfully terminated.

Tips

• The name and token arguments of the launch_worker() method can help construct informative job names.
• For efficiency, every launch can happen asynchronously. There is no need to wait for the worker to start. When it starts, it will connect back to the mirai dispatcher over the local network and start accepting tasks.
• If you implement a terminate_worker() method, each worker termination may also happen asynchronously. In rare cases when you do not trust the platform to terminate the worker on the first request, you can use crew_retry() to wait for the job to exit, but this may reduce efficiency.
• The source code of the built-in local process launcher is a helpful reference.