Running Qwen 3.8 27B on an Intel Arc Pro B70 with TensorFold XPU and LLM Router

Intel Arc Pro B70 TensorFold XPU llama.cpp SYCL LLM Router Qwen 3.8 Local LLM Benchmark

Intel's Arc Pro B70 has 32 GB of VRAM. That's enough to run a dense 27B model on one card, with a six-figure context window and a speculative drafter loaded next to it. The hard part on Windows has been the software. You need an inference engine that actually uses the card's XPU backend, and a way to switch between engines without editing launch scripts.

This article shows how to run Qwen 3.8 27B on the B70 with the TensorFold engine on XPU, managed by LLM Router. It then benchmarks that setup against the same model on llama.cpp's SYCL backend, through the same router endpoint. In this setup, TensorFold generated tokens about 2.3× faster and processed prompts about 1.7× faster.

The setup

The router's Devices page picks up the B70 and lists SYCL and Vulkan as its likely compute APIs. That's a quick way to confirm Windows and the oneAPI runtime can see the card before you set anything else up.

LLM Router Devices page listing the Intel Arc Pro B70 with 32 GB VRAM and SYCL and Vulkan APIs

The Devices page: the Arc Pro B70 with 32 GB of VRAM, next to the machine's other GPUs

Step 1: Build the TensorFold XPU engine

TensorFold doesn't publish a Windows release, and its Intel engine only runs on Linux. On Windows, LLM Router runs it in Docker Desktop and reaches the GPU through WSL 2.

Open Engines, switch to the TensorFold tab and find the Intel GPU (XPU) in Docker card. Choose bmg (Arc B-series) as the GPU and click Build image. The router builds an image (llm-router/tensorfold-xpu:managed) that compiles TensorFold over a pinned Intel compute runtime, then checks the result on the GPU. When it finishes, the install appears under Installed builds with the XPU (Level Zero) backend. Here that's TensorFold v1.2.7.

Docker Desktop must be running with the WSL 2 engine. Docker Desktop only runs while its user is signed in, so if LLM Router runs as a Windows service, the TensorFold server can't start while that user is signed out.

LLM Router Engines page on the TensorFold tab showing TensorFold v1.2.7 Intel XPU Docker build and the Build image card

The TensorFold tab: the Intel XPU Docker build installed and in use by one server

Step 2: Create the server

Under Servers → New Server, create a server with the TensorFold engine and bind it to the install from step 1. In this setup it's called xpu-tensorfold and listens on port 8086. It sits next to a sycl-llama.cpp server on port 8080 that targets the same B70.

Both servers share a card with 32 GB of VRAM, so only one of them can be loaded at a time. LLM Router handles this for you: when a request names a preset whose server isn't running, the router starts it. If you put both servers in a memory group, the router also stops the least-recently-used idle server first to make room.

LLM Router Servers list with xpu-tensorfold running and sycl-llama.cpp idle

xpu-tensorfold running on port 8086, with sycl-llama.cpp idle on the same card

Step 3: Download the model from Hugging Face

You don't need to download anything by hand. On the router's Models page, click Browse Hugging Face, search for SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16 and download it into the model library. The page shows progress while it downloads. Then do the same for the drafter, z-lab/Qwen3.8-27B-DFlash2. The drafter is a DFlash2 checkpoint for speculative decoding, and it's a big part of the decode speeds below.

The library recognises both as checkpoints for TensorFold and vLLM, so you can choose them straight from the preset editor. When the server starts, the Docker container mounts their folders read-only. Together they take about 22 GB on disk: 18.2 GB for the model and 3.6 GB for the drafter. If you set a Hugging Face token in the Models page settings, it's also passed on for gated checkpoints.

LLM Router model library showing the Qwen3.8-27B GPTQ Int4 checkpoint and the DFlash2 drafter, both sourced from Hugging Face

Both checkpoints downloaded from Hugging Face into the model library, ready for TensorFold

Step 4: Create the preset

Under Presets → New Preset, create tensorfold-qwen3.8-27b on the xpu-tensorfold server. Choose the model from the library, then fill in the TensorFold section:

Setting

Value

Download the model

Never. The router's model library has already downloaded it, and TensorFold's own pull doesn't run on Windows.

Context length

110,000 tokens

Reasoning effort

medium

Drafter

z-lab_Qwen3.8-27B-DFlash2, chosen from the library

Intel WDDM chunk size

4096 MB. This sets ForceWddmHugeChunkSizeMB for Intel XPU under WSL/Docker. Use 4096 if loading aborts in wddm_memory_manager.cpp.

Images

Enabled (--vision)

Prompt cache

Left at the 2 GiB default

With these settings, the 4-bit 27B model, a 110k-token context, the speculative drafter and vision support all fit on the one B70. The prompt cache uses GPU memory from the context, so lower it if a long context doesn't fit.

TensorFold engine settings in the LLM Router preset editor

The TensorFold section of the preset: 110k context, DFlash2 drafter, WDDM chunk size and vision

Step 5: Call it through the router

Clients only ever talk to the router. To use a preset, put its name in the model field. If the preset isn't loaded yet, the router starts the right server first:

curl http://localhost:5054/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "tensorfold-qwen3.8-27b",
    "messages": [{"role": "user", "content": "Explain how a GPU executes a matrix multiplication."}],
    "stream": true
  }'

The router exposes the same preset through its Claude-compatible (/v1/messages) and Ollama-compatible (/api/chat) endpoints. Tools such as GitHub Copilot in Visual Studio can use the B70 with no other changes.

For a quick test without writing any code, open the Playground, pick the preset and run the built-in Code prompt. Each reply shows its time to first token and its prompt and generation speeds.

LLM Router Playground running the Code prompt on tensorfold-qwen3.8-27b at 116 tokens per second

The built-in Code prompt in the Playground: 126 ms to first token and 116 tok/s generation on the B70

Benchmark: TensorFold XPU vs. llama.cpp SYCL on the B70

For comparison, the Qwen3.8-27B-Q6 preset on the sycl-llama.cpp server runs the unsloth UD-Q6_K GGUF on the same card. Its settings were 128k context, all layers on the GPU, flash attention on, a q8_0 KV cache, MTP speculative decoding (draft-mtp, up to 3 draft tokens), the F16 multimodal projector, and reasoning effort medium.

Method

  • Every request went through the router at http://localhost:5054/v1/chat/completions with streaming on and temperature 0.6.

  • Requests ran one at a time, with no concurrency.

  • Each test ran three times after a warm-up request. The tables show the median.

  • Decode speed was measured on the client, from the first streamed token to the end of the response. It includes reasoning tokens, so it's the speed you actually see.

  • For the cold prefill tests, each prompt started with a unique ID so neither engine could reuse its prompt cache.

  • Only one server was loaded at a time. The router swapped them on the B70 between runs.

Token generation

Test

TensorFold XPU

llama.cpp SYCL

Difference

Short chat (≈768 tokens out)

89.8 tok/s

39.8 tok/s

2.25×

Code generation (1,536 tokens out)

119.7 tok/s

50.8 tok/s

2.35×

Summary of an 8k-token prompt (512 tokens out)

56.9 tok/s

44.6 tok/s

1.27×

Summary of a 16k-token prompt (128 tokens out)

37.0 tok/s

44.0 tok/s

0.84×

On short contexts, TensorFold is much faster. Code generation reached about 120 tokens per second from a dense 27B model on one workstation card, and the 1,536-token code answer finished in 12.9 s, compared with 30.4 s on llama.cpp. Code is predictable, so the drafter's guesses are accepted more often and speculative decoding pays off the most.

The gap narrows as the context grows. At 16k tokens, llama.cpp's decode is slightly faster: it barely slows down as the context fills, while TensorFold drops from about 120 tok/s to under 40.

Prompt processing (cold prefill)

Prompt size

TensorFold XPU

llama.cpp SYCL

Difference

8k tokens

1,606 tok/s (TTFT 5.0 s)

920 tok/s (TTFT 8.8 s)

1.75×

16k tokens

1,507 tok/s (TTFT 10.7 s)

893 tok/s (TTFT 18.1 s)

1.69×

TensorFold processed prompts about 70% faster at both sizes. That matters most for agents and RAG, where every turn sends thousands of tokens of context. When the prompt was already cached, both engines took about 0.16 s to the first token on the 8k prompt, so repeated prompts are fast on either engine.

You don't need your own script to watch this. The router's Statistics page charted both benchmark sessions as they ran.

LLM Router Statistics dashboard charting prompt processing and token generation speed for xpu-tensorfold and sycl-llama.cpp

The Statistics page during the benchmark, with prompt-processing and generation speeds for both servers

Load time

TensorFold XPU

llama.cpp SYCL

Cold start: first request until the server answers

≈3 min 9 s

≈1 min 50 s

llama.cpp loads faster, and TensorFold also has to start its Docker container. With LLM Router that's a one-off cost: the first request waits for the engine to load, and every request after that goes straight to the loaded server.

A fair caveat

This was an experiment to show what's possible on a single B70 card. It isn't a strict apples-to-apples comparison of engines or models. Each preset ran the way it's set up on this machine: the TensorFold preset uses a 4-bit GPTQ model with a DFlash2 drafter, and the llama.cpp preset uses a 6-bit Q6_K GGUF with MTP drafting. The Q6_K model moves more data per token and probably keeps a little more quality, so some of the decode gap comes from the smaller weights and the drafter. Read the numbers as what one B70 can deliver in practice, not as a ranking of the engines.

You can repeat any of these runs in the router's Playground with Record runs turned on, and compare them on the Run History page.

Takeaways

  • The B70 can serve a dense 27B model locally. With 32 GB of VRAM it holds a 4-bit Qwen 3.8 27B with a 110k context, a drafter and vision support on one card. Interactive work ran at 90–120 tok/s.

  • TensorFold on XPU was the faster engine on this card for short and medium contexts: about 2.3× faster generation and about 1.7× faster prompt processing than llama.cpp's SYCL backend.

  • llama.cpp still has its strengths. It runs natively on Windows without Docker, loads faster and holds its decode speed better at long contexts.

  • The router lets you switch between them. Both presets sit behind the same http://localhost:5054/v1 endpoint, and choosing between them only changes the model field. LLM Router manages which one is loaded in the B70's VRAM.

To try it yourself, Windows builds and the full feature list are on the LLM Router project page, and the source is on GitHub.