feat(skills): vLLM and Ollama deployment plus LlamaFactory fine-tuning skills (#19)

Co-authored-by: Alice <alice@prismshadow.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 22:04:19 +08:00
1 parent cb283b6d9d
commit 636b25b63f
@@ -65,6 +65,9 @@ The built-in Skills, by group (the group manifest is `SKILL_GROUPS` in `packages
 | AI App Development | `penguin-sdk` | Build AI and RAG apps on the SDK: the createSession/run streaming loop plus a complete retrieval recipe with chunk-revealing citations |
 | | `penguin-cli` | Manage model API keys, default models and per-agent Vault secrets with the penguin CLI |
 | | `agenthub-models` | Call model APIs through `@prismshadow/agenthub`: streaming text, image generation, speech synthesis and embeddings |
+| | `vllm` | Deploy and serve LLMs with vLLM behind an OpenAI-compatible endpoint, with tool calling enabled for agent workloads |
+| | `ollama` | Deploy and serve local models with Ollama: pull and run them, then expose the OpenAI-compatible endpoint to apps and agents |
+| | `llamafactory` | Fine-tune LLMs with LlamaFactory: register datasets, train via YAML configs, merge LoRA adapters and serve the result |
 | Agent Tuning | `agent-creation` | Turn a user requirement into a concrete agent: write the target agent's AGENTS.md and install the skills it needs |
 | | `benchmark-design` | Design and calibrate a multi-Case capability Benchmark with repeated independent evaluations and a traceable baseline |
 | | `agent-evaluation` | Run and score exactly one Benchmark Case run, with CLI execution, Trace provenance checks and private Rubric isolation |
@@ -65,6 +65,9 @@ Skill 库以 npm 包 `@prismshadow/penguin-skills` 发布，tarball 直接携带
 | AI 应用开发 | `penguin-sdk` | 基于 SDK 构建 AI 与 RAG 应用：createSession/run 流式循环，外加带可溯源引用的完整检索配方 |
 | | `penguin-cli` | 用 penguin CLI 管理模型 API Key、默认模型与各 Agent 的 Vault 密钥 |
 | | `agenthub-models` | 经 `@prismshadow/agenthub` 调用模型 API：流式文本、图像生成、语音合成与 Embedding |
+| | `vllm` | 用 vLLM 部署与服务 LLM，提供 OpenAI 兼容端点，并为 Agent 负载启用工具调用 |
+| | `ollama` | 用 Ollama 部署与运行本地模型，把 OpenAI 兼容端点接入应用与 Agent |
+| | `llamafactory` | 用 LlamaFactory 微调 LLM：注册数据集、以 YAML 配置训练、合并 LoRA 适配器并部署产物 |
 | Agent 调优 | `agent-creation` | 把用户需求变成具体的 Agent：撰写目标 Agent 的 AGENTS.md 并安装所需 Skill |
 | | `benchmark-design` | 设计并校准多 Case 的能力评测 Benchmark，含重复独立评测与可追溯基线 |
 | | `agent-evaluation` | 隔离执行并评分单个 Benchmark Case:CLI 执行、Trace 溯源检查、Rubric 私有隔离 |
@@ -79,6 +79,9 @@ describe("skills api", () => {
      "penguin-sdk",
      "penguin-cli",
      "agenthub-models",
+      "vllm",
+      "ollama",
+      "llamafactory",
    ]);
    expect(body.groups[3]!.skills.map((s) => s.name)).toEqual([
      "agent-creation",
@@ -10,7 +10,7 @@ Included skills, in the order of the `SKILL_GROUPS` manifest in `src/index.ts` (
 | --- | --- |
 | Office Productivity | `data-analysis`, `firecrawl` |
 | Software Development | `web-design`, `software-engineering` |
-| AI App Development | `penguin-sdk`, `penguin-cli`, `agenthub-models` |
+| AI App Development | `penguin-sdk`, `penguin-cli`, `agenthub-models`, `vllm`, `ollama`, `llamafactory` |
 | Agent Tuning | `agent-creation`, `benchmark-design`, `agent-evaluation`, `agent-optimization` |

 Agent Tuning powers the self-improvement loop: create the Target Agent, design a Benchmark, evaluate it, optimize it to version N+1 with a snapshot before every round.
@@ -0,0 +1,105 @@
+---
+name: llamafactory
+description: Fine-tune LLMs with LlamaFactory — register datasets, train via YAML configs, merge LoRA adapters and serve the result.
+short_description: Fine-tune models with LlamaFactory.
+short_description_zh: 用 LlamaFactory 微调模型。
+version: 1
+updated: 2026-07-22T00:00:00Z
+---
+
+# LlamaFactory Fine-Tuning
+
+LlamaFactory fine-tunes open-weight LLMs (LoRA/QLoRA and full-parameter; SFT, DPO and more) through the `llamafactory-cli` command driven by YAML configs.
+
+## Before you start
+
+If the user's message only invokes this skill (e.g. "use llamafactory skill") without a concrete request, ask the user what they want to fine-tune. Do not run any command until the goal is clear.
+
+Confirm before training:
+
+- GPU memory (`nvidia-smi`) — it bounds the model size and method; LoRA needs far less than full fine-tuning.
+- The base model: a Hugging Face id or a local path.
+- The dataset: where it lives and which format it is in.
+- The goal: SFT with LoRA is the usual starting point.
+
+## Install
+
+```bash
+git clone --depth 1 https://github.com/hiyouga/LlamaFactory.git
+cd LlamaFactory
+pip install -e .
+pip install -r requirements/metrics.txt   # optional: evaluation metrics
+```
+
+## Data
+
+Register every dataset in `data/dataset_info.json`; the alpaca and sharegpt formats are supported. A minimal local entry:
+
+```json
+"my_dataset": { "file_name": "my_dataset.json" }
+```
+
+alpaca rows carry `instruction` / `input` / `output`; sharegpt rows carry a `conversations` list. Put the data file under `data/` next to the registry.
+
+## Train
+
+Training is driven by a YAML config. Start from the shipped example `examples/train_lora/qwen3_lora_sft.yaml`, or save a minimal config as `my_sft.yaml`, e.g. for [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B):
+
+```yaml
+model_name_or_path: Qwen/Qwen3-1.7B
+trust_remote_code: true
+stage: sft
+do_train: true
+finetuning_type: lora
+lora_rank: 8
+lora_target: all
+dataset: my_dataset
+template: qwen3
+output_dir: saves/qwen3-1.7b/lora/sft
+learning_rate: 1.0e-4
+num_train_epochs: 3.0
+bf16: true
+```
+
+```bash
+llamafactory-cli train my_sft.yaml
+```
+
+`llamafactory-cli webui` launches the no-code web UI for the same workflow.
+
+## Merge and export
+
+Merge the LoRA adapter into the base weights for standalone serving. Start from `examples/merge_lora/qwen3_lora_sft.yaml`, pointing `model_name_or_path`, `adapter_name_or_path` and `template` at your run (never merge into a quantized base):
+
+```yaml
+model_name_or_path: Qwen/Qwen3-1.7B
+adapter_name_or_path: saves/qwen3-1.7b/lora/sft
+template: qwen3
+trust_remote_code: true
+export_dir: saves/qwen3-1.7b-sft-merged
+```
+
+```bash
+llamafactory-cli export my_merge.yaml
+```
+
+## Try the result
+
+Both commands take an inference config — derive it from `examples/inference/qwen3_lora_sft.yaml`, again pointing the model, adapter and template at your run:
+
+```yaml
+model_name_or_path: Qwen/Qwen3-1.7B
+adapter_name_or_path: saves/qwen3-1.7b/lora/sft
+template: qwen3
+infer_backend: huggingface
+trust_remote_code: true
+```
+
+```bash
+llamafactory-cli chat my_infer.yaml   # interactive chat with the tuned model
+llamafactory-cli api my_infer.yaml    # OpenAI-compatible API server
+```
+
+## Close the loop
+
+Serve the merged export as a standalone endpoint — vLLM serves the export directory directly, while Ollama needs an import first (a `Modelfile` with `FROM /path/to/export`, then `ollama create`; supported model architectures only) — then register the endpoint with PenguinHarness so agents can build, evaluate and tune AI apps on the fine-tuned model end to end.
@@ -0,0 +1,10 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round">
+  <path d="M3.5 21h17" />
+  <path d="M5 21V10.4a2.1 2.1 0 0 1 4.2 0V21" />
+  <path d="M6.1 8.6V6.9" />
+  <path d="M8.1 8.6V6.9" />
+  <path d="M12 21v-5.6l3.4 2.2v-2.2l3.4 2.2V21" />
+  <path d="M17.6 16.8v-4.2" />
+  <circle cx="18.3" cy="10.5" r="1.5" />
+  <circle cx="20.3" cy="7.9" r="1.1" />
+</svg>
@@ -0,0 +1,91 @@
+---
+name: ollama
+description: Deploy and serve local models with Ollama — pull and run them, then expose the OpenAI-compatible endpoint to apps and agents.
+short_description: Run local models with Ollama.
+short_description_zh: 用 Ollama 运行本地模型。
+version: 1
+updated: 2026-07-22T00:00:00Z
+---
+
+# Ollama Serving
+
+Ollama runs open-weight models locally with automatic GPU detection and an OpenAI-compatible API on `http://localhost:11434`.
+
+## Before you start
+
+If the user's message only invokes this skill (e.g. "use ollama skill") without a concrete request, ask the user what they want. Do not run any command until the goal is clear.
+
+Ask the user which model to run; if they have no preference, recommend the small default [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) (`ollama pull qwen3.5:0.8b`). The model must fit the machine's RAM/VRAM.
+
+Ollama runs everywhere — macOS, Linux and Windows, on CPUs as well as NVIDIA/AMD GPUs — so engine choice follows the user's preference: Ollama is the simple default, while vLLM targets high-throughput GPU serving. Check the current state first:
+
+```bash
+ollama --version   # is Ollama installed?
+ollama ps          # is the service already serving models?
+```
+
+If port 11434 is already serving, reuse that instance — never kill an existing Ollama process.
+
+## Suggested workflow
+
+1. Ask the user which model to run; with no preference, recommend [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) (`qwen3.5:0.8b`).
+2. Pick the engine the user prefers: Ollama is the default; vLLM covers high-throughput GPU serving.
+3. Install Ollama if missing, then `ollama pull qwen3.5:0.8b`.
+4. Verify with `curl http://localhost:11434/v1/models`.
+5. Register the endpoint: `penguin config model add ... --client-type openai --base-url http://localhost:11434/v1` — a pulled Ollama model is not visible to Penguin until added.
+6. Confirm the new entry with `penguin config model list`.
+
+## Install
+
+```bash
+curl -fsSL https://ollama.com/install.sh | sh   # Linux; macOS/Windows use the desktop app
+```
+
+The service then listens on `http://localhost:11434`.
+
+## Pull and run
+
+```bash
+ollama pull qwen3.5:0.8b   # download a model
+ollama run qwen3.5:0.8b    # interactive chat (pulls first if missing)
+ollama list                # downloaded models
+ollama ps                  # models loaded in memory
+ollama stop qwen3.5:0.8b   # unload a model
+```
+
+## OpenAI-compatible endpoint
+
+The endpoint is `http://localhost:11434/v1`; any non-empty API key is accepted (conventionally `ollama`):
+
+```bash
+curl http://localhost:11434/v1/models
+```
+
+## Context length
+
+The default context window is small, and agent sessions need a large one. Raise it in the server's environment:
+
+```bash
+OLLAMA_CONTEXT_LENGTH=32768 ollama serve   # systemd service: set it via `systemctl edit ollama`
+```
+
+Or bake it into a model variant with a Modelfile:
+
+```
+FROM qwen3.5:0.8b
+PARAMETER num_ctx 32768
+```
+
+```bash
+ollama create qwen3.5-32k -f Modelfile
+```
+
+## Register with PenguinHarness
+
+Model configuration is the penguin CLI's job — `penguin config model add` registers an endpoint and `penguin config model list` shows what has been registered. A pulled Ollama model is not visible to Penguin until you add it:
+
+```bash
+penguin config model add --provider custom --client-type openai \
+  --base-url http://localhost:11434/v1 --model-id qwen3.5:0.8b --api-key ollama
+penguin config model list   # the new entry should now be listed
+```
@@ -0,0 +1,9 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round">
+  <rect x="6.8" y="7" width="10.4" height="11" rx="4.4" />
+  <path d="M8.8 7.2V4.6a1.3 1.3 0 0 1 2.6 0v2.2" />
+  <path d="M12.6 6.8V4.6a1.3 1.3 0 0 1 2.6 0v2.6" />
+  <path d="M9.7 11.6h.01" />
+  <path d="M14.3 11.6h.01" />
+  <ellipse cx="12" cy="14.8" rx="2.9" ry="2.1" />
+  <path d="M12 13.9v.9" />
+</svg>
@@ -3,13 +3,13 @@ name: penguin-cli
 description: Manage model API keys, default models and per-agent vault secrets with the penguin CLI.
 short_description: Manage models and secrets with the penguin CLI.
 short_description_zh: 用 penguin CLI 管理模型与密钥。
-version: 3
-updated: 2026-07-21T00:00:00Z
+version: 4
+updated: 2026-07-22T00:00:00Z
 ---

 # Penguin CLI

-The `penguin` CLI manages model credentials, default models and per-agent vault secrets. Configuration goes through the CLI only — never read or hand-edit the underlying hidden files.
+The `penguin` CLI manages model credentials, default models and per-agent vault secrets. Its primary job is model configuration: `penguin config model add` registers a model and `penguin config model list` shows the models currently available. Configuration goes through the CLI only — never read or hand-edit the underlying hidden files.

 ## Before you start

@@ -30,7 +30,10 @@ penguin config model add --provider <group> --model-id <upstream_id> [--api-key
 - For any OpenAI chat-completion compatible endpoint use `--client-type openai --base-url <endpoint>`; omit `--client-type` to auto-route by model id.
 - Prices are USD per million tokens (cache read / cache write / output).
 - `--vision` / `--no-vision` mark whether the model accepts images; omitting both keeps the current value (default is vision-capable).
- All `penguin config model ...` and `penguin config vault ...` commands accept `--root <dir>` to target another data root (default `PENGUIN_HOME`, then `~/.penguin/data`). **When configuring models for an AI app you are building, always pass `--root <dir>` pointing at the app's own data directory inside the current working directory** (e.g. `--root ./penguin_data`, the same path the app gives `createAgent({ root })`); running without `--root` writes to the global `~/.penguin/data`, which belongs to the person running Penguin — not to the app.
+- All `penguin config model ...` and `penguin config vault ...` commands accept `--root <dir>` to target another data root (default `PENGUIN_HOME`, then `~/.penguin/data`). Two configuration targets — treat the difference as a hard rule:
+  - **Penguin's own model** (self-configuration: the model Penguin itself runs on): the default root without `--root` is correct.
+  - **An AI app you are building**: `--root` **must** point at the app's own data directory inside the project (e.g. `--root ./penguin_data`, the same path the app gives `createAgent({ root })`) unless the user explicitly chose another location — never write an app's models or keys into the global `~/.penguin/data`, which belongs to the person running Penguin, not to the app.
+  - While developing an app, review regularly: `penguin config model list --root <app root>` should show the app's entries, and the global list (no `--root`) should stay clean.

 Other model commands:

@@ -0,0 +1,94 @@
+---
+name: vllm
+description: Deploy and serve LLMs with vLLM behind an OpenAI-compatible endpoint, with tool calling enabled for agent workloads.
+short_description: Serve models locally with vLLM.
+short_description_zh: 用 vLLM 部署本地模型服务。
+version: 1
+updated: 2026-07-22T00:00:00Z
+---
+
+# vLLM Serving
+
+vLLM serves open-weight LLMs on local GPUs with high-throughput inference behind an OpenAI-compatible API, ready for chat and agent workloads.
+
+## Before you start
+
+If the user's message only invokes this skill (e.g. "use vllm skill") without a concrete request, ask the user what they want. Do not run any command until the goal is clear.
+
+Ask the user which model to serve; if they have no preference, recommend the small default [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B). Also ask what context length the workload needs.
+
+vLLM needs an NVIDIA or AMD GPU. Engine choice follows the user's preference: Ollama also runs on GPUs and is the simpler default — pick vLLM for high-throughput serving, and Ollama on macOS or CPU-only machines, which vLLM does not serve. Confirm the hardware first:
+
+```bash
+nvidia-smi          # NVIDIA: GPU model and free VRAM (AMD ROCm: rocm-smi)
+python3 --version   # a recent Python is required
+```
+
+The model must fit the available VRAM — model size and context length drive the serve flags below.
+
+## Suggested workflow
+
+1. Ask the user which model to serve; with no preference, recommend [Qwen/Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B).
+2. Pick the engine the user prefers: vLLM for high-throughput GPU serving; Ollama is the simple default and the choice on macOS or CPU-only machines.
+3. Serve on a free port, with the tool-calling flags whenever agents will call it (see below).
+4. Verify with `curl http://localhost:8000/v1/models`.
+5. Register the endpoint: `penguin config model add ... --client-type openai --base-url http://localhost:8000/v1` — a served model is not visible to Penguin until added.
+6. Confirm the new entry with `penguin config model list`.
+
+## Install
+
+Use a fresh virtual environment (or `uv`):
+
+```bash
+python3 -m venv .venv && source .venv/bin/activate
+pip install vllm
+```
+
+## Serve
+
+```bash
+vllm serve Qwen/Qwen3.5-0.8B --port 8000
+```
+
+This exposes an OpenAI-compatible API at `http://localhost:8000/v1`. Key flags:
+
+- `--served-model-name <name>` — the model id clients request (defaults to the model path).
+- `--api-key <key>` — require this bearer token on every request.
+- `--max-model-len <n>` — context window; agent sessions need a large one.
+- `--gpu-memory-utilization <0..1>` — fraction of VRAM to claim (default 0.9).
+- `--tensor-parallel-size <n>` — shard across `n` GPUs.
+- `--dtype <auto|bfloat16|float16>` and `--quantization <awq|gptq|fp8>` — precision and quantized weights.
+
+If the port is taken, pick a free one — never kill a process already listening on it.
+
+## Tool calling — required for agents
+
+Agent harnesses (PenguinHarness included) send `tools` with their requests. vLLM must opt in at startup:
+
+```bash
+vllm serve Qwen/Qwen3.5-0.8B --enable-auto-tool-choice --tool-call-parser hermes
+```
+
+Choose the parser for the model family — e.g. `hermes` for Qwen models, `llama3_json` for Llama models. Without these flags, requests that set tool_choice fail with `400 "auto" tool choice requires --enable-auto-tool-choice and --tool-call-parser to be set`.
+
+## Verify
+
+```bash
+curl http://localhost:8000/v1/models
+```
+
+## Register with PenguinHarness
+
+Model configuration is the penguin CLI's job — `penguin config model add` registers an endpoint and `penguin config model list` shows what has been registered. A served model is not visible to Penguin until you add it:
+
+```bash
+penguin config model add --provider custom --client-type openai \
+  --base-url http://localhost:8000/v1 --model-id <served-model-name> --api-key <key>
+penguin config model list   # the new entry should now be listed
+```
+
+## Troubleshooting
+
+- Out of memory at startup: lower `--gpu-memory-utilization` or `--max-model-len`, or serve a quantized model.
+- Long prompts truncated or context-length errors: raise `--max-model-len` (bounded by VRAM).
+- `400` on tool calls: restart the server with the tool-calling flags above.
@@ -0,0 +1,4 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round">
+  <path d="M3.8 5.5h7.9l-1.9 11.2z" />
+  <path d="M16.4 3.8h4l-4.9 16.4h-4z" />
+</svg>
@@ -154,7 +154,7 @@ export const SKILL_GROUPS: SkillGroupInfo[] = [
    id: "ai-app-development",
    title: "AI App Development",
    titleZh: "AI 应用开发",
-    skills: ["penguin-sdk", "penguin-cli", "agenthub-models"],
+    skills: ["penguin-sdk", "penguin-cli", "agenthub-models", "vllm", "ollama", "llamafactory"],
  },
  {
    id: "agent-tuning",
@@ -104,6 +104,9 @@ describe("loadSkillGroups / groupSkills", () => {
      "penguin-sdk",
      "penguin-cli",
      "agenthub-models",
+      "vllm",
+      "ollama",
+      "llamafactory",
    ]);
    expect(groups[2]!.title).toBe("AI App Development");
    expect(groups[2]!.titleZh).toBe("AI 应用开发");
@@ -159,7 +162,7 @@ describe("loadSkillGroups / groupSkills", () => {
      { id: "software-development", skills: ["web-design", "software-engineering"] },
      {
        id: "ai-app-development",
-        skills: ["penguin-sdk", "penguin-cli", "agenthub-models"],
+        skills: ["penguin-sdk", "penguin-cli", "agenthub-models", "vllm", "ollama", "llamafactory"],
      },
      {
        id: "agent-tuning",