Deploying a prediction service with Plumber

This document shows how you can deploy a fitted model as a web service using ACR, ACI and AKS. The framework used is Plumber, a package to expose your R code as a service via a REST API.

Fit the model

We’ll fit a simple model for illustrative purposes, using the Boston housing dataset which ships with R (in the MASS package). To make the deployment process more interesting, the model we fit will be a random forest, using the randomForest package. This is not part of R, so we’ll have to install it from CRAN.

data(Boston, package="MASS")
install.packages("randomForest")
library(randomForest)

# train a model for median house price as a function of the other variables
bos_rf <- randomForest(medv ~ ., data=Boston, ntree=100)

# save the model
saveRDS(bos.rf, "bos_rf.rds")

Scoring script for plumber

Now that we have the model, we also need a script to obtain predicted values from it given a set of inputs:

# save as bos_rf_score.R

bos_rf <- readRDS("bos_rf.rds")
library(randomForest)

#* @param df data frame of variables
#* @post /score
function(req, df)
{
    df <- as.data.frame(df)
    predict(bos_rf, df)
}

This is fairly straightforward, but the comments may require some explanation. They are plumber annotations that tell it to call the function if the server receives a HTTP POST request with the path /score, and query parameter df. The value of the df parameter is then converted to a data frame, and passed to the randomForest predict method.

Create a Dockerfile

Let’s package up the model and the scoring script into a Docker image. A Dockerfile to do this would look like the following. This uses the base image supplied by Plumber (trestletech/plumber), installs randomForest, and then adds the model and the above scoring script. Finally, it runs the code that will start the server and listen on port 8000.

# example Dockerfile to expose a plumber service

FROM trestletech/plumber
MAINTAINER Hong Ooi <hongooi@microsoft.com>

# install the randomForest package
RUN R -e 'install.packages(c("randomForest"))'

# copy model and scoring script
RUN mkdir /data
COPY bos_rf.rds /data
COPY bos_rf_score.R /data
WORKDIR /data

# plumb and run server
EXPOSE 8000
ENTRYPOINT ["R", "-e", \
    "pr <- plumber::plumb('/data/bos_rf_score.R'); pr$run(host='0.0.0.0', port=8000)"]

Build and upload the image

The code to store our image on Azure Container Registry is as follows. If you are running this code, you should substitute the values of tenant, app and/or secret from your Azure service principal. Similarly, if you are using the public Azure cloud, note that all ACR instances share a common DNS namespace, as do all ACI and AKS instances.

For more information on how to create a service principal, see the AzureRMR readme.

library(AzureContainers)

az <- AzureRMR::az_rm$new(
    tenant="72f988bf-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    app="f72c1733-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    password="xxxxxxxxx")

# create a resource group for our deployments
deployresgrp <- az$
    get_subscription("35975484-xxxx-xxxx-xxxx-xxxxxxxxxxxx")$
    create_resource_group("deployresgrp", location="australiaeast")

# create container registry
deployreg_svc <- deployresgrp$create_acr("deployreg")

# build image 'bos_rf'
call_docker("build -t bos_rf .")

# upload the image to Azure
deployreg <- deployreg_svc$get_docker_registry()
deployreg$push("bos_rf")

If you run this code, you should see a lot of output indicating that R is downloading, compiling and installing randomForest, and finally that the image is being pushed to Azure. (You will see this output even if your machine already has the randomForest package installed. This is because the package is being installed to the R session inside the container, which is distinct from the one running the code shown here.)

All Docker calls in AzureContainers, like the one to build the image, return the actual docker commandline as the cmdline attribute of the (invisible) returned value. In this case, the commandline is docker build -t bos_rf . Similarly, the push() method actually involves two Docker calls, one to retag the image, and the second to do the actual pushing; the returned value in this case will be a 2-component list with the command lines being docker tag bos_rf deployreg.azurecr.io/bos_rf and docker push deployreg.azurecr.io/bos_rf.

Deploy to an Azure Container Instance

The simplest way to deploy a service is via a Container Instance. The following code creates a single running container which contains our model, listening on port 8000.

# create an instance with 2 cores and 8GB memory, and deploy our image
deployaci <- deployresgrp$create_aci("deployaci",
    image="deployreg.azurecr.io/bos_rf",
    registry_creds=deployreg,
    cores=2, memory=8,
    ports=aci_ports(8000))

To check on the progress of the deployment, call the object’s sync_fields() method. This will update and return its provisioning state, which will be "Creating" while the instance is being created, and "Succeeded" once it is running. Deploying a simple container like this usually takes less than a minute.

deployaci$sync_fields()
#> [1] "Succeeded"

Once the instance is running, let’s call the prediction API with some sample data. By default, AzureContainers will assign the container a domain name with prefix taken from the instance name. The port is 8000 as specified in the Dockerfile, and the URI path is /score indicating we want to call the scoring function defined earlier.

The data to be scored—the first 10 rows of the Boston dataset—is passed in the body of the request as a named list, encoded as JSON. A feature of Plumber is that, when the body of the request is in this format, it will extract the elements of the list and pass them to the scoring function as named arguments. This makes it easy to pass around relatively large amounts of data, eg if the data is wide, or for scoring multiple rows at a time. For more information on how to create and interact with Plumber APIs, consult the Plumber documentation.

response <- httr::POST("http://deployaci.australiaeast.azurecontainer.io:8000/score",
    body=list(df=MASS::Boston[1:10,]), encode="json")
httr::content(response, flatten=TRUE)
#> [1] 25.9269 22.0636 34.1876 33.7737 34.8081 27.6394 21.8007 22.3577 16.7812 18.9785

Deploy to a Kubernetes cluster

Deploying a service to a container instance is simple, but lacks many features that are important in a production setting. A better alternative for production purposes is to deploy to a Kubernetes cluster. Such a cluster can be created using Azure Kubernetes Service (AKS).

# create a Kubernetes cluster with 2 nodes, running Linux (the default)
deployclus_svc <- deployresgrp$create_aks("deployclus", agent_pools=aks_pools("pool1", 2))

Unlike an ACI resource, creating a Kubernetes cluster can take several minutes. By default, the create_aks() method will wait until the cluster provisioning is complete before it returns.

Having created the cluster, we can deploy our model and create a service. We’ll use a YAML configuration file to specify the details for the deployment and service API. The image to be deployed is the same as before.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bos-rf
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: bos-rf
    spec:
      containers:
      - name: bos-rf
        image: deployreg.azurecr.io/bos_rf
        ports:
        - containerPort: 8000
        resources:
          requests:
            cpu: 250m
          limits:
            cpu: 500m
      imagePullSecrets:
      - name: deployreg.azurecr.io
---
apiVersion: v1
kind: Service
metadata:
  name: bos-rf-svc
spec:
  selector:
    app: bos-rf
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 8000

The following code will obtain the cluster endpoint from the AKS resource and then deploy the image and service to the cluster. The configuration details for the deployclus cluster are stored in a file located in the R temporary directory; all of the cluster’s methods will use this file. Unless told otherwise, AzureContainers does not touch your default Kubernetes configuration (~/kube/config).

# get the cluster endpoint
deployclus <- deployclus_svc$get_cluster()

# pass registry authentication details to the cluster
deployclus$create_registry_secret(deployreg, email="hongooi@microsoft.com")

# create and start the service
deployclus$create("bos_rf.yaml")

To check on the progress of the deployment, run the get() methods specifying the type and name of the resource to get information on. As with Docker, these correspond to calls to the kubectl commandline tool, and again, the actual commandline is stored as the cmdline attribute of the returned value.

deployclus$get("deployment bos-rf")
#> Kubernetes operation: get deployment bos-rf  --kubeconfig=".../kubeconfigxxxx"
#> NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
#> bos-rf    1         1         1            1           5m

deployclus$get("service bos-rf-svc")
#> Kubernetes operation: get service bos-rf-svc  --kubeconfig=".../kubeconfigxxxx"
#> NAME         TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)          AGE
#> bos-rf-svc   LoadBalancer   10.0.8.189   52.187.249.58   8000:32276/TCP   5m

Once the service is up and running, as indicated by the presence of an external IP in the service details, let’s test it with a HTTP request. The response should be the same as it was with the container instance.

response <- httr::POST("http://52.187.249.58:8000/score",
    body=list(df=MASS::Boston[1:10,]), encode="json")
httr::content(response, simplifyVector=TRUE)
#> [1] 25.9269 22.0636 34.1876 33.7737 34.8081 27.6394 21.8007 22.3577 16.7812 18.9785

Finally, once we are done, we can tear down the service and deployment:

deployclus$delete("service", "bos-rf-svc")
deployclus$delete("deployment", "bos-rf")

And if required, we can also delete all the resources created here, by simply deleting the resource group (AzureContainers will prompt you for confirmation):

deployresgrp$delete()