Ray AI

Appliance Description

Ray is an open-source framework for distributed computing and machine learning workloads. The Ray appliance leveragles Ray’s Serve library to enable efficient deployment of inference APIs, and includes the vLLM for fast inference and serving.

Main Features

The Ray appliance includes two frameworks for running LLMs:

  • Hugging Face - Transformers, one of the most widely adopted frameworks for deploying large language models
  • vLLM, the open source, high-perfomance engine for serving large language models with low latency and high throughput
  • Configurable deployment options and behavior, controlled by contextualization parameters

Main References