dsh-multimodal — multimodal eyes and hands for DeepSeek Harness

English | 简体中文

Give DeepSeek a pair of eyes and a paintbrush: paste a screenshot or image into the conversation and the GLM vision model first transcribes it verbatim (error messages, code, UI text preserved), then DeepSeek keeps solving your problem — all in the same turn, no extra steps. When an illustration is needed, DeepSeek automatically calls the text-to-image backend and the generated pictures appear right in the conversation.

Compatibility: built for DeepSeek Harness 0.1.0-rc.6 (Web and headless). See CHANGELOG.

Features

Scenario	Behavior
Plain-text chat	Straight to the DeepSeek API (unchanged)
Image + question (e.g. error screenshot)	GLM vision model "looks" first → transcribes to text → DeepSeek continues from the transcription (fix code, explain, propose)
Attaching an image	No more "current model does not support images"
User asks for an image	DeepSeek calls generate_image → Aliyun Wanx / Zhipu cogview-3-flash etc. → images shown in the conversation
Image card	generate_image calls render as a dedicated image card: thumbnail grid, click-to-zoom lightbox, one-click download, prompt & model metadata (injected through the tool.call.toolview slot — no shell changes)
Extracting text from an image	DeepSeek can call extract_text (OCR) → Markdown / plain text / JSON
Other vision platforms	extraProviders accepts any OpenAI-compatible vision endpoint (Xfyun, ModelScope, SiliconFlow, Qianfan, Volcano Engine, …)

Installation

Prerequisites: official DeepSeek Harness (0.1.0-rc.6+) installed and running, Node.js 18+.

# Option 1: install straight from the GitHub repo (requires git)
dsh plugin --profile web add https://github.com/<your-username>/dsh-multimodal

# Option 2: clone and install from a local path
git clone https://github.com/<your-username>/dsh-multimodal.git
dsh plugin --profile web add /path/to/dsh-multimodal

# Also add for headless mode when needed:
dsh plugin --profile headless add /path/to/dsh-multimodal

The repo ships prebuilt artifacts (lib/), so you can install right after cloning; to rebuild from source, see Development.

After restarting dsh web, a 「多模态 / Multimodal」 page appears in the Settings sidebar — everything is configured in one place:

• GLM vision key (see images) and Aliyun Bailian key (draw images) — paste and save, takes effect immediately
• Image backend dropdown: Aliyun Wanx / Zhipu cogview-3-flash (free, watermarked) …
• Custom vision platforms: name, Base URL, key ref, model ID, key → add, shows up in the model picker immediately

Usage

1. In the Web UI model picker choose a DeepSeek (Vision) model — DeepSeek-V4-Flash (Vision) or DeepSeek-V4-Pro (Vision) (the ones tagged Vision are the ones with eyes)
2. Chat normally; paste/drag an image and ask to have it read, or just say "draw me a …" to generate one

API Keys (or write the credentials file directly)

① GLM vision key (see images)

1. Get a free API key from Zhipu: https://open.bigmodel.cn (default vision model GLM-4.1V-Thinking-Flash, free; also glm-4.6v-flash / glm-4v-flash)
2. Write it to the credentials file (takes effect immediately; path is $DSH_HOME/.credentials.yaml, default ~/.dsh/.credentials.yaml):

   GLM_VISION_API_KEY: <your-zhipu-key>
   

② Aliyun Bailian key (draw images)

1. Go to Bailian console: https://bailian.console.aliyun.com → enable Wanx text-to-image (search wan2.2-t2i-flash in the model plaza → enable; requires real-name verification)
2. Create an API key (top-right avatar → API-KEY)
3. Write it to the credentials file (note: environment variables win over the config file; if DASHSCOPE_API_KEY is already set in your environment, edit its value instead):

   DASHSCOPE_API_KEY: <your-aliyun-key>
   

How it works

User image ──▶ host admission (model directory declares image modality → allowed, no popup)
           ──▶ agent/pre-step (plugin): detects images → vision engine (GLM etc.) transcribes
               └─▶ image blocks replaced by "【图片内容转写】…" text
           ──▶ DeepSeek API processes the transcription + your question (image bytes never reach DeepSeek)

• Two built-in routes: deepseek-vision (DeepSeek API, directory advertises image support so the UI allows it; images are actually transcribed, never sent) and glm-vision (Zhipu vision API, inline base64)
• Image bytes are read through the dsh attachment service; the transcription instruction defaults to "transcribe verbatim + extract key error info" and can be customized in the dsh-multimodal: section of ~/.dsh/settings.yaml
• On transcription failure it degrades to a 【图片转写失败: 原因】 placeholder — DeepSeek keeps going instead of hanging
• Settings UI: Settings → Multimodal (settings.section slot, native look & feel like the Models page)

Configuration (optional, ~/.dsh/settings.yaml)

dsh-multimodal:
  providers:
    glm:
      model: glm-4.1v-thinking-flash  # default (stable in practice); or glm-4.6v-flash / glm-4v-flash
  vision:
    transcribeProvider: glm-vision    # vision engine (default GLM; any extra provider works)
  ocr:
    provider: glm-vision              # OCR tool engine
    model: glm-4v-flash               # or glm-ocr (note: glm-ocr only supports PDF/JPG/PNG, ≤10MB)
  image:
    activeBackend: dashscope          # image backend: dashscope / glm-cogview (free) / custom
  transcribePrompt: ...               # custom transcription instruction
  transcribeTimeoutMs: 90000          # transcription timeout

  # Any OpenAI-compatible vision platform (just add the key; shows up in the model picker & Models page)
  extraProviders:
    xfyun-vision:                     # Xfyun MaaS
      displayName: 讯飞星辰视觉
      baseURL: https://maas-api.cn-huabei-1.xf-yun.com/v2
      apiKeyEnv: XFYUN_API_KEY
      models:
        - id: xoppaddleocrv16         # free OCR (PaddleOCR-VL-1.6)
          name: PaddleOCR-VL-1.6
    siliconflow-ocr:                  # SiliconFlow (free hosted DeepSeek-OCR)
      displayName: 硅基流动 OCR
      baseURL: https://api.siliconflow.cn/v1
      apiKeyEnv: SILICONFLOW_API_KEY
      models:
        - id: deepseek-ai/DeepSeek-OCR
          name: DeepSeek-OCR
    modelscope-vision:                # ModelScope (2000 free calls/day)
      displayName: 魔搭视觉
      baseURL: https://api-inference.modelscope.cn/v1
      apiKeyEnv: MODELSCOPE_API_TOKEN
      models:
        - id: Qwen/Qwen3-VL-8B-Instruct
          name: Qwen3-VL-8B

Image backends (OpenAI-compatible /images/generations and DashScope protocols) example:

dsh-multimodal:
  image:
    activeBackend: z-image            # Aliyun Z-Image-Turbo (tested; sk-ws business-space key, generic domain)
    # backends:
    #   z-image:                      # Bailian Z-Image family (new multimodal-generation sync protocol)
    #     kind: dashscope-v2
    #     baseURL: https://dashscope.aliyuncs.com
    #     apiKeyEnv: DASHSCOPE_API_KEY
    #     model: z-image-turbo
    #     defaultSize: 1024*1024      # supports 512*512 ~ 2048*2048, default 1024*1536
    #   modelscope-t2i:
    #     kind: openai-images
    #     baseURL: https://api-inference.modelscope.cn/v1
    #     apiKeyEnv: MODELSCOPE_API_TOKEN
    #     model: <a lightning-badged text-to-image model in the ModelScope plaza>

Troubleshooting

Symptom	Cause & fix
Reply says "【图片转写失败: MISSING_CREDENTIAL…】"	Vision key missing: add it under Settings → Multimodal, or write the credentials file
Reply says "【图片转写失败: RATE_LIMIT…】"	Free-tier rate limit; retry later or switch to a paid tier
No DeepSeek (Vision) in the model picker	Restart dsh web (plugin installed; takes effect after restart)
No "Multimodal" entry in Settings	Verify the plugin is mounted (dsh --profile web --dump-config should list dsh-multimodal) and hard-refresh the browser (Ctrl+F5)

Development

npm install
npm run build      # host: tsc; client: tsc check + tsdown builds lib/client.js

License

MIT