Build AI Agents with vLLM
Self-hosted high-throughput inference server
Available vLLM Models
| Model | ID |
|---|---|
| Llama 3.1 8B Instruct | meta-llama/Llama-3.1-8B-Instruct |
| Qwen 2.5 7B Instruct | Qwen/Qwen2.5-7B-Instruct |
Quick start code
agent.py
from agno.agent import Agent
from agno.models.vllm import VLLM
agent = Agent(
name="My vLLM Agent",
model=VLLM(id="meta-llama/Llama-3.1-8B-Instruct"),
instructions=["You are a helpful assistant."],
markdown=True,
)
agent.print_response("Hello! What can you help me with?")Install: pip install agno Then set API_KEY