Skip to main content

PVC Model Serving


This example is virtually the same as the S3 based one, instead of repeating all steps please read that example first as this is an abbreviated version.

Getting Started

  1. Follow the guide to export the model through Jupyter first. The model can be placed into a CoreWeave filesystem through many other methods, such as via an SSH or rsync container. Please note that this repository does not include the PVC definition, as it is defined in the Jupyter repository.

  2. Apply the resources. This can be used to both create and update existing manifests

     $ kubectl apply -f gpt-pvc-inferenceservice.yaml created
  3. List pods to see that the Transformer and Predictor have launched successfully

    $ kubectl get pods
    gpt-pvc-predictor-default-ljxhm-deployment-86596d6846-kwnkd 2/2 Running 0 34s
    gpt-pvc-transformer-default-f5lf9-deployment-86bc7b4fd8-g8m69 2/2 Running 0 34s

    If the predictor fails to start, it is probably due to the model being saved at the wrong path. kubectl logs gpt-pvc-predictor-default-ljxhm-deployment-86596d6846-kwnkd kfserving-container will contain an error if so.

  4. Once all the Pods are running, we can get the API endpoint for our model

    $ kubectl get ksvc
    gpt-pvc-predictor-default gpt-pvc-predictor-default-ljxhm gpt-pvc-predictor-default-ljxhm True
    gpt-pvc-transformer-default gpt-pvc-transformer-default-f5lf9 gpt-pvc-transformer-default-f5lf9 True

    We want to use the transformer endpoint to be able to make requests in cleartext. Ensure that both services are listed as Ready.

  5. The model is now available to access over the Internet. Use curl to test it out.

     $ curl -d '{"instances": ["That was easy"]}'
    {"predictions": ["That was easy to say, what else would you do, what would you do, would you say to your daughter and say to her, 'Where is the work you're doing, where is the work you're working on, and how are you doing it?' and she was like, 'I'm not going to be here, I can't do it!' and she became, you know, frustrated. And I think there's a different type of anxiety. There's this self-pity that comes in, and that's also why they call their child a 'brilliant' child.\n\nShe was always saying that when she was a little girl, there was something really important to do. But she really doesn't go. She knows that whatever she does, when she's ready, she's going to go into any school or program and that she's going to do. And she really needs to do that, because it's just so much more exciting to her now.\n\nIt made her less able to put her mind at the 'solution' to her child's difficulties \u2013 even as she had more opportunities than I or anyone could ever do, and at a time when we were trying a lot of things to find the balance in the world. And I"]}