LogoTensorFusion Docs
LogoTensorFusion Docs
Homepage

Getting Started

OverviewKubernetes InstallVM/Server Install(K3S)Helm On-premises InstallHost/GuestVM InstallTensorFusion Architecture

Application Operations

Create WorkloadConfigure AutoScalingMigrate Existing Workload

Customize AI Infra

Production-Grade DeploymentConfig QoS and BillingBring Your Own CloudManaging License

Maintenance & Optimization

Upgrade ComponentsSetup AlertsGPU Live MigrationPreload ModelOptimize GPU Efficiency

Troubleshooting

HandbookTracing/ProfilingQuery Metrics & Logs

Comparison

Compare with NVIDIA vGPUCompare with MIG/MPSCompare with Run.AICompare with HAMi

Host/GuestVM Install

Run tensor-fusion-worker on the host and deploy tensor-fusion-client inside a virtual machine (VM) to use the host’s physical GPUs from VMs without GPU passthrough, named as TensorFusion HyperGPU.

Terminology

  1. Tensor-fusion-worker (worker): A standalone binary. You can run multiple worker instances on the same host. It receives compute requests from clients and executes the tasks.
  2. Tensor-fusion-client (client): A set of shared libraries that fully export CUDA and NVML APIs. It runs inside your application environment and forwards compute requests to the worker on the host.

Multiple clients can connect to one worker instance. For production, we recommend one client per worker instance for easier resource isolation and management.

Prerequisites

  1. A Linux host with NVIDIA GPUs. Recommended NVIDIA driver version: 570.xx or newer.
  2. One or more VMs (QEMU/MVisor/VMware/Hyper‑V, etc.) that can reach the host either over the network or via shared memory.

Step 1: Download Tensor Fusion

  1. Download the latest release: https://cdn.tensor-fusion.ai/archive-lastest.zip
  2. Unzip the archive:
unzip archive-lastest.zip

Step 2: Install tensor-fusion-worker on the host

Check the worker binary options:

./tensor-fusion-worker -h
Usage: tensor-fusion-worker [option]
Options
  -h, --help            Display this information.
  -v, --version         Display version information.
  -n, --net             Specified network protocol.
  -p, --port            Specified the port for server.
  -l, --load            Specified the file path
  1. Worker supports two protocols: NATIVE and SHMEM.
  2. NATIVE uses TCP, simple to deploy; recommended when VMs and host are in the same network with ping <= ~0.8 ms.
  3. SHMEM uses shared memory, lowest latency; best for latency-sensitive workloads.

Start with NATIVE protocol on port 12345:

TF_ENABLE_LOG=1 ./tensor-fusion-worker -n native -p 12345

Start with SHMEM protocol, creating shared memory at /my_shm with size 256 MB:

TF_ENABLE_LOG=1 ./tensor-fusion-worker -n shmem -m /my_shm -M 256

Environment variables

  • TF_ENABLE_LOG: Enable logging (default: off)
  • TF_LOG_LEVEL: trace|debug|info|warn|error (default: info)
  • TF_LOG_PATH: Log file path (default: empty, i.e., stdout)
  • TF_CUDA_MEMORY_LIMIT: CUDA memory limit in MB (default: unlimited)

Step 3: Install tensor-fusion-client in the VM

  1. Get the tensor-fusion-client directory from the extracted package.

Windows

Place files from tensor-fusion-client/windows (nvcuda.dll, nvml.dll, teleport.dll) either in the system PATH or alongside your application so they can be loaded.

Linux

Use LD_LIBRARY_PATH or LD_PRELOAD to inject the shared libraries from tensor-fusion-client/linux (libcuda.so, libnvidia-ml.so, libteleport.so) into your application process.

OS environments vary. Please ensure the client libraries are actually loaded by your application.

Environment variables

  • TF_ENABLE_LOG: Enable logging (default: off)
  • TF_LOG_LEVEL: trace|debug|info|warn|error (default: info)
  • TF_LOG_PATH: Log file path (default: empty; on Linux logs to console, on Windows view with DebugView)
  • TENSOR_FUSION_OPERATOR_GET_CONNECTION_URL: An HTTP GET endpoint that returns connection info (optional)
  • TENSOR_FUSION_OPERATOR_CONNECTION_INFO: Connection info in the format protocol+param1+param2+version (version is currently 0)
  • TF_MAX_CACHE_REQUEST_COUNT: Maximum cache request count (default: 100)

You can set the connection info directly via TENSOR_FUSION_OPERATOR_CONNECTION_INFO, or provide a GET endpoint via TENSOR_FUSION_OPERATOR_GET_CONNECTION_URL that returns it.

Step 4: Verify the setup

We’ll verify inside a Linux VM using Python with PyTorch CUDA.

Prepare the environment

Enable logs and inject the client libraries

export TF_ENABLE_LOG=1
export LD_PRELOAD=/opt/tensor-fusion-client/linux/libteleport.so:/opt/tensor-fusion-client/linux/libcuda.so:/opt/tensor-fusion-client/linux/libnvidia-ml.so

Assuming the client is installed under /opt/tensor-fusion-client. Adjust paths as needed.

NATIVE protocol

If the worker runs with NATIVE at host IP 192.168.1.100 and port 12345:

  1. Set the connection info (format: native+[host-ip]+[port]+[version]):
export TENSOR_FUSION_OPERATOR_CONNECTION_INFO=native+192.168.1.100+12345+0

Ensure the VM can reach the host IP and port.

SHMEM protocol

If the worker runs with SHMEM, creates /my_shm and sets size to 256 MB, start your VM with QEMU and attach the shared memory via IVSHMEM:

QEMU command example

qemu-system-x86_64 \
  -m 8192 \
  -hda centos8.4.qcow2 \
  -vnc :1 \
  -enable-kvm \
  -cpu host \
  -object memory-backend-file,id=shm0,mem-path=/dev/shm/my_shm,size=256M,share=on \
  -device ivshmem-plain,memdev=shm0 \
  -smp
  1. Locate the IVSHMEM device inside the VM:
lspci -vv | grep -i Inter-VM

Possible output:

00:04.0 RAM memory: Red Hat, Inc. Inter-VM shared memory (rev 01)

Hence the device BDF is 00:04.0.

  1. Set the connection info. (format: shmem+[resource]+[size]+[version]) Use the device resource file (e.g., resource2), match the size (MB), and version 0:
export TENSOR_FUSION_OPERATOR_CONNECTION_INFO=shmem+/sys/devices/pci0000:00/0000:00:04.0/resource2+256+0
  1. The shared-memory size must match the worker’s setting.
  2. Root privileges are typically required to mmap the IVSHMEM BAR resource.

PyTorch validation example

Once the environment variables are set, run the following (PyTorch + Qwen3‑0.6B) in the VM:

pip install modelscope packaging transformers accelerate

cat << 'EOF' >> test-qwen.py
from modelscope import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-0.6B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
	model_name,
	torch_dtype="auto",
	device_map="cuda:0"
)

prompt = "Give me a short introduction to large language model."
messages = [
	{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
	messages,

Validation criteria:

  • The VM prints normal model outputs.
  • On the host, nvidia-smi shows the corresponding inference process and GPU memory usage.

Helm On-premises Install

Deployment to On-premises environments without Cluster Agent and Cloud Console using Helm

TensorFusion Architecture

TensorFusion technical architecture and principles

Table of Contents

Terminology
Prerequisites
Step 1: Download Tensor Fusion
Step 2: Install tensor-fusion-worker on the host
Step 3: Install tensor-fusion-client in the VM
Windows
Linux
Step 4: Verify the setup
Prepare the environment
NATIVE protocol
SHMEM protocol
PyTorch validation example
of
snapshot.
-m, --shmem-file Specified the file path of shared memory.
-M, --shmem-size Specified the size(MB) of shared memory.
4
tokenize=False,
add_generation_prompt=True,
enable_thinking=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
generated_ids = model.generate(
**model_inputs,
max_new_tokens=32768
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
try:
# rindex finding 151668 (</think>)
index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
index = 0
thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")
print("thinking content:", thinking_content)
print("content:", content)
EOF
python3 test-qwen.py