beastgitea2026/gpt_from_scratch
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

c

Browse Source
This commit is contained in:
betterbeast2023
2026-06-10 19:45:12 +08:00
parent 1183d10849
commit a3ebb28ce0
12 changed files with 35469 additions and 491 deletions
Download Patch File Download Diff File Expand all files Collapse all files
opencode_lesson.md
+250
View File
@@ -0,0 +1,250 @@
# Local AI Coding Environment — Reference & Lessons Learned
> Consolidates `local_ai_setup.md` and `local_llm.md`. Those files are superseded by this one.
---
## Hardware
| Component | Spec |
|-----------|------|
| GPU | NVIDIA RTX 5070 Ti, 12 GB VRAM |
| RAM | 32 GB system RAM |
| OS | Windows 11 |
| WSL | WSL2 Ubuntu |
---
## Software Stack
| Tool | Purpose | Notes |
|------|---------|-------|
| Ollama | Local model server | Serves OpenAI-compatible API at `localhost:11434/v1` |
| OpenCode | Primary AI coding agent | Installed via npm (`opencode-ai`) |
| Aider | Alternative AI coding agent | pip: `aider-chat` |
| Claude Code | Cloud AI coding agent | Optional; can point at Ollama |
| VS Code | Editor | With Continue extension for inline chat |
---
## Architecture
```
VS Code
├── Continue Extension ──────────────┐
├── Claude Code CLI ────────────────┤
└── OpenCode / Aider ───────────────┤
▼
Ollama
│
▼
Qwen3-Coder 30B (primary)
```
All code stays local. Cloud models are optional fallback only.
---
## Ollama — Available Models
| Model | Size | Use |
|-------|------|-----|
| `qwen3-coder:30b-fixed` | 18 GB | **Primary** — OpenCode agent (fixed config, see below) |
| `qwen3-coder:30b` | 18 GB | Original pull (unmodified) |
| `gemma4:latest` | 9.6 GB | English-reliable alternative |
| `gemma4:12b` | 7.6 GB | Lighter English-reliable option |
| `qwen2.5-coder:14b` | 9.0 GB | Mid-weight coding model |
| `qwen2.5-coder:7b` | 4.7 GB | Fast, lightweight coding |
| `qwen2.5-coder:1.5b` | 986 MB | Minimal footprint |
| `qwen3:latest` | 5.2 GB | General chat |
| `qwen3:1.7b` | 1.4 GB | Fast general chat |
| `bge-m3:latest` | 1.2 GB | Embeddings |
---
## OpenCode Configuration
**Config file:** `~/.config/opencode/opencode.jsonc`
```jsonc
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"ollama": {
"options": {
"baseURL": "http://localhost:11434/v1"
},
"models": {
"qwen3-coder:30b-fixed": { "name": "Qwen3 Coder 30B" },
"qwen2.5-coder:14b": { "name": "Qwen 2.5 Coder 14B" },
"qwen2.5-coder:7b": { "name": "Qwen 2.5 Coder 7B" },
"gemma4:latest": { "name": "Gemma 4 8B (latest)" },
"qwen3:latest": { "name": "Qwen3 8B" }
}
}
},
"model": "ollama/qwen3-coder:30b-fixed"
}
```
---
## Active Model — qwen3-coder:30b-fixed Modelfile
```
FROM <local blob: sha256-9124a16eb1a7...>
TEMPLATE {{ .Prompt }}
PARAMETER repeat_penalty 1.05
PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>
PARAMETER stop <|endoftext|>
PARAMETER temperature 0.7
PARAMETER top_k 20
PARAMETER top_p 0.8
PARAMETER num_ctx 32768 ← fixed (was missing)
PARAMETER num_predict 8192 ← fixed (was missing)
```
Model specs: 30.5B params, MoE architecture (qwen3moe), Q4_K_M quantization, supports tools + completion.
---
## Issue & Fix — Tool Call JSON Truncation
### Error
```
llama-server returned invalid tool call arguments for "edit":
unexpected end of ... [retrying in 51s attempt N]
```
### Root Cause
Two Ollama parameters were missing from the model's Modelfile:
| Parameter | Default when missing | Required for tool calls |
|-----------|---------------------|------------------------|
| `num_ctx` | **2048 tokens** | Minimum ~8192; 32768 recommended |
| `num_predict` | **128 tokens** | Minimum ~1024; 8192 recommended |
With only 128 output tokens allowed, the model began generating valid JSON for the `edit` tool call but ran out of budget before closing all braces, producing malformed JSON. The 2048-token context also filled up quickly during long sessions, compounding the problem.
### Fix Applied
Rebuilt `qwen3-coder:30b-fixed` with the two parameters added:
```bash
# Export current Modelfile, append fixes, rebuild
ollama show qwen3-coder:30b-fixed --modelfile > Modelfile_fixed
# (add num_ctx 32768 and num_predict 8192)
ollama create qwen3-coder:30b-fixed -f Modelfile_fixed
```
No changes needed to the OpenCode config — same model name, same config file.
### Verification
```bash
ollama show qwen3-coder:30b-fixed
# Should show: num_ctx 32768 and num_predict 8192 in Parameters section
```
---
## Aider Configuration
**Config file:** `.aider.conf.yml` in project root
```yaml
model: ollama/qwen3-coder:30b
openai-api-base: http://localhost:11434/v1
openai-api-key: ollama
edit-format: whole
map-tokens: 2048
no-auto-commits: true
system-prompt: "Always respond in English only. Never use Chinese or any other language."
```
| Option | Why |
|--------|-----|
| `edit-format: whole` | More reliable file writes for local models |
| `map-tokens: 2048` | Limits repo map size for smaller context windows |
| `no-auto-commits: true` | Review changes before they hit git |
| `system-prompt` | Qwen3 defaults to Chinese; forces English |
### Aider Quick Reference
| Command | What it does |
|---------|-------------|
| `/add src/foo.py` | Add file to context |
| `/drop src/foo.py` | Remove file from context |
| `/diff` | See pending changes |
| `/undo` | Revert last change |
| `/run pytest` | Run command, share output with model |
| `/clear` | Clear chat history (keep files) |
| `/settings` | Show loaded config |
| `/model ollama/gemma4:12b` | Switch model on the fly |
---
## Known Issues & Fixes
### Tool call JSON truncated (OpenCode / llama-server)
**Error:** `llama-server returned invalid tool call arguments for "edit": unexpected end of...`
**Cause:** `num_predict` too low (default 128) cuts off JSON mid-generation.
**Fix:** See "Issue & Fix" section above — add `num_ctx` and `num_predict` to Modelfile.
### Model responds in Chinese
**Cause:** Qwen3 is a Chinese model and defaults to Chinese.
**Fix (Aider):** `system-prompt` in `.aider.conf.yml`.
**Fix (OpenCode):** Add a system prompt in OpenCode's project config if supported, or use `gemma4` models which default to English.
### Aider writes code to chat but not to disk
**Cause:** Model didn't follow Aider's edit format.
**Fix 1:** `/add target_file.py` before the request — hints the target.
**Fix 2:** `edit-format: whole` in config (already included above).
### Config changes not taking effect (Aider)
**Fix:** Exit (`/exit`) and restart. Config is read only at startup.
### "File not found" / repo-map warnings for deleted files
**Cause:** File deleted from disk but still tracked in git index.
**Fix:**
```bash
git rm --cached filename.py
# or from inside Aider:
/run git rm --cached filename.py
```
### Model loops with filler text, never executes tool calls
**Symptom:** Model repeatedly says "I'll continue..." but never reads/edits files. Context counter in OpenCode bottom-left shows a number near your `num_ctx` limit.
**Cause:** Context window is full. No tokens left for tool call JSON, so the model outputs short filler and stalls.
**Fix (immediate):** Start a new OpenCode session. Break large tasks (e.g. "translate all files") into one file per session.
**Fix (permanent):** Increase `num_ctx` in the Modelfile (e.g. 65536), rebuild, restart Ollama. Watch RAM usage — each doubling of `num_ctx` roughly doubles KV cache memory.
### Context fills up, model performance degrades
**Cause:** Long session accumulates tokens beyond `num_ctx`.
**Fix (Aider):** `/clear` to reset chat history without losing file context.
**Fix (Ollama model):** Increase `num_ctx` in Modelfile (current: 32768).
---
## Performance Notes
- qwen3-coder:30b is a MoE model — fits in VRAM+RAM split on RTX 5070 Ti (12 GB VRAM).
- `num_gpu 99` in Modelfile maximizes layers on VRAM; remainder spills to RAM.
- Safe `num_ctx` range for this hardware: **8192–32768**. Above 32768 risks RAM pressure.
- Q4_K_M quantization gives best quality/speed/memory balance for this size class.
---
## Environment Verification Checklist
```bash
ollama list # models appear
ollama show qwen3-coder:30b-fixed # num_ctx and num_predict present
curl http://localhost:11434/api/tags # Ollama serving
aider --version # Aider installed
opencode --version # OpenCode installed
nvidia-smi # GPU visible
```