CoreWeave
Search…
Custom - BASNET

Introduction

This example demonstrates deploying an auto-scaling Inference service from a pre-existing docker image. This can be useful when deploying off-the-shelf models that aren't available as ie. Tensorflow SavedModels. One example of this is the IBM COCO Based Object Detector. An example InferenceService for that also exists in this repository. The rest of this example will focus on a public wrapped version of the BASNet object detection model. This example and the test client is based on work by Cyril Diagne.
To follow along, clone the manifests from GitHub.
Input
Output

Getting Started

After installing kubectl and adding your CoreWeave Cloud access credentials, the following steps will deploy the Inference Service. Clone all the files in this repository to follow along.
  1. 1.
    Apply the resources. This can be used to both create and update existing manifests
    1
    $ kubectl apply -f basnet-inferenceservice.yaml
    2
    inferenceservice.serving.kubeflow.org/basnet configured
    Copied!
  2. 2.
    List pods to see that the Transformer and Predictor have launched successfully
    1
    $ kubectl get pods
    2
    NAME READY STATUS RESTARTS AGE
    3
    basnet-predictor-default-sj9kr-deployment-76b67d669-4gjrp 2/2 Running 0 34s
    Copied!
    If the predictor fails to init, look in the logs for clues kubectl logs basnet-predictor-default-sj9kr-deployment-76b67d669-4gjrp kfserving-container.
  3. 3.
    Once all the Pods are running, we can get the API endpoint for our model. Since this model doesn't adhere to the Tensorflow V1 HTTP API, we can't use the API endpoint provided by kubectl get inferenceservices. We have to hit up the predictor directly.
    1
    $ kubectl get ksvc
    2
    NAME URL LATESTCREATED LATESTREADY READY REASON
    3
    basnet-predictor-default https://basnet-predictor-default.tenant-test.knative.chi.coreweave.com basnet-predictor-default-sj9kr basnet-predictor-default-sj9kr True
    Copied!
    The URL in the output is the public API URL for your newly deployed model.
  4. 4.
    Enter the client directory. You can either run the test client locally or in docker. The output will be in images/output.png.
    1
    $ cd client/
    2
    $ export SERVICE_URL=https://basnet-predictor-default.tenant-test.knative.chi.coreweave.com
    3
    $ docker build -t test .; docker run --rm -it -v $(pwd)/images:/app/images test --basnet_service_host $SERVICE_URL
    4
    INFO:root: > sending to BASNet...
    5
    INFO:root:200
    6
    INFO:root: > saving results...
    7
    INFO:root: > opening mask...
    8
    INFO:root: > compositing final image...
    9
    INFO:root: > saving final image...
    10
    $ open images/output.png
    Copied!
  5. 5.
    Remove the inference service
    1
    $ kubectl delete inferenceservices basnet
    2
    inferenceservice.serving.kubeflow.org "basnet" deleted
    Copied!
Last modified 1yr ago