Hugging Face has introduced a simplified approach to launching a private, OpenAI-compatible LLM endpoint on its infrastructure, achievable with just one command. This method eliminates the need for server provisioning or Kubernetes management, allowing users to pay per second for usage.
Once the server is operational, it can be accessed from various devices, making it an efficient solution for testing, evaluations, or batch generation tasks. For those seeking a managed, production-ready service, Hugging Face offers Inference Endpoints, which cater to different needs.
Setting Up the Server
To initiate the server, users must ensure they have a payment method or a positive prepaid credit balance, as the service is billed by hardware usage per minute. The prerequisites include having huggingface_hub version 1.20.0 or higher installed and being logged in locally. The command to launch the server utilizes the official vllm/vllm-openai image, requesting a GPU and exposing the necessary port.
The command structure is as follows:
hf jobs run --flavor a10g-large --expose 8000 --timeout 2h
vllm/vllm-openai:latest
vllm serve Qwen/Qwen3-4B --host 0.0.0.0 --port 8000Upon execution, the command will provide a URL for accessing the server. Users can query the model through the OpenAI API, requiring only their Hugging Face token for authentication.
Interacting with the Model
To interact with the model, users can utilize curl commands or Python scripts. For example, a simple curl command can send a message to the model and receive a response:
curl https://--8000.hf.jobs/v1/chat/completions
-H "Authorization: Bearer $(hf auth token)"
-H "Content-Type: application/json"
-d '{ "model": "Qwen/Qwen3-4B", "messages": [{"role": "user", "content": "Hello!"}] }'This interaction will yield a response formatted in the standard OpenAI JSON style.
Managing and Scaling the Server
Users are advised to stop the server when not in use, as billing occurs per second. The command to cancel the job is straightforward, and the cost for running an a10g-large instance is $1.50 per hour. For larger models, users can specify a different GPU flavor and adjust parameters such as tensor-parallel-size to optimize performance.
Hugging Face’s infrastructure allows for scaling up to larger models, such as the Qwen3.5 mixture-of-experts model, by adjusting the command parameters accordingly. Additionally, users can implement a chat interface using Gradio or SSH into the server for debugging and monitoring purposes.
In summary, Hugging Face’s vLLM on Jobs provides a flexible and efficient way to deploy and interact with large language models, catering to both experimental and production needs.
This article was produced by NeonPulse.today using human and AI-assisted editorial processes, based on publicly available information. Content may be edited for clarity and style.
