Local AI on your own hardware: from a desktop GPU to Kimi K3

Running AI on your own hardware gives you a different kind of control. You choose where prompts are processed, which model version stays in service, and how much capacity to buy. A document assistant or a coding workflow can become a service you operate alongside your other applications.

The entry point is smaller than a server rack. AMD's Radeon AI PRO R9700 has 32 GB of GPU memory, enough to make a quantized Qwen3.8-27B a realistic local candidate. A Mac mini offers another route through shared CPU and GPU memory. Further up the scale, DeepSeek and GLM need hundreds of gigabytes, while Kimi K3 makes local inference an infrastructure project.

This guide compares those options using specifications and model artifacts checked on 5 September 2026. Memory estimates are planning guidance; we have not benchmarked these configurations. Download sizes use decimal GB/TB as reported by the repositories; hardware memory is quoted as marketed. Prices are explicitly identified vendor reference prices, not live Belgian retail quotes.

Start with memory, then measure speed

Inference means running a trained model to produce an answer. You download its weights—the numerical values learned during training—and load them into a runtime. You do not need to repeat the training process.

A useful first calculation is:

Raw weight storage in GB ≈ parameters in billions × bits per parameter / 8

At 16 bits, a 27-billion-parameter model needs roughly 54 GB for those parameters. At 4 bits, the arithmetic drops to 13.5 GB. Quantization stores weights at reduced precision, trading some numerical fidelity for a smaller footprint. Real formats also store scales, metadata and some higher-precision tensors, so a four-bit download is larger than that simple calculation.

Then add the runtime's working memory, the conversation's attention cache, and any vision components. Longer contexts and simultaneous users can consume the space left after the weights load. On a computer with unified memory, the operating system and other applications need their share too.

There is another distinction in mixture-of-experts (MoE) models. Only some parameters participate in each token, but the other experts still need to be available. Active parameters help explain compute demand; total parameters determine the starting weight budget. Offloading experts to CPU memory changes where that budget sits and can reduce speed.

Memory capacity answers whether a deployment is plausible. Memory bandwidth, compute, runtime support and concurrency determine whether people will enjoy using it. Measure time to the first token, generation speed and queueing with your actual documents and prompts.

Qwen makes a practical desktop starting point

Alibaba's Qwen family spans useful hardware sizes. The smaller Qwen3.5 releases include 0.8B, 2B, 4B and 9B models. These are candidates for extraction, classification, short document work and lightweight assistants; evaluate the smallest one that can do your task reliably. The naming matters: these small releases are Qwen3.5, while the larger desktop example here is Qwen3.8-27B. See the Qwen3.5 collection and Qwen3.8 model card.

For a concrete memory reference, Unsloth's downloadable Qwen3.8-27B GGUF variants include a 16.5 GB UD-Q4_K_M file and a 29 GB Q8_0 file. Those are third-party quantizations of Qwen's weights, and the sizes describe files, not total running memory. Unsloth artifact listing.

That makes the R9700's 32 GB a sensible capacity match for the four-bit version, with room left for runtime allocations. A 29 GB eight-bit file leaves much less headroom; do not assume it supports your desired context or concurrent users simply because the file is smaller than VRAM. Use a compatible llama.cpp build and verify the AMD GPU, operating system and driver combination against the ROCm support matrices.

For text-only experimentation, begin with a short context and one user, then increase the workload. Loading the full-precision checkpoint or enabling vision changes the requirement.

GLM also has a desktop-sized option: GLM-4.7-Flash, with 30B total and 3B active parameters. Its 18.3 GB Q4_K_M download makes it another candidate for a 24–32 GB GPU. It is much smaller than GLM-5.3-Flash discussed below. Official GLM-4.7-Flash card, quantized files.

The hardware worth comparing

These are representative options across the useful memory tiers. Add-in cards need a host computer, suitable power supply, cooling and PCIe slots. Complete systems include those components. A manufacturer's product listing confirms a product option; it does not establish today's local stock or delivery date.

Desktop and workstation GPUs

Intel's B70 reference price comes from its March 2026 announcement. NVIDIA announced the desktop 5090 starting at $1,999. These US reference prices exclude the rest of the computer and are not comparable to a Belgian VAT-inclusive system quote.

The professional NVIDIA card has 96 GB, and the older RTX 6000 Ada has 48 GB. Check the full name when comparing offers. Also distinguish desktop cards from laptop GPUs carrying similar names.

A 5090 does not fit a larger set of weights than an R9700 merely because it costs more. Both have 32 GB. The buying decision turns on measured throughput, supported kernels, model format and the complete system cost. llama.cpp supports CUDA, HIP, Vulkan and SYCL, but support for a backend does not guarantee every model feature works equally well on it.

Complete systems with shared memory

Availability matters for the Mac Studio figures: the M5 models above are preorder options on this article's date. Apple gives 22 September 2026 for initial availability and late October for 512 GB configurations. Those are announced capacities to plan around, not machines already shipping on 5 September. Apple's launch announcement.

Apple silicon's CPU and GPU access the same physical memory pool. This is why a Mac can run a model that exceeds the dedicated VRAM of many desktop cards: it avoids a separate, smaller graphics-memory ceiling. Apple's MLX documentation explains that shared-memory design. The installed capacity still includes macOS, the application, model weights and working memory; a 64 GB Mac is not a 64 GB dedicated GPU.

AMD's Variable Graphics Memory explanation describes a related capacity advantage on Ryzen AI Max+ systems. NVIDIA's Spark combines shared memory with its CUDA environment and is available through NVIDIA and system partners. None of these memory figures is a promise of equal inference speed.

DeepSeek and GLM: where the memory budget changes

Moving beyond desktop Qwen means looking carefully at the specific release and quantization. Flash is a model name, not a promise that it fits in 32 GB.

DeepSeek-V4-Flash has 284B total parameters and 13B active per token. For the current release, use the official DeepSeek-V4-Flash-0731 checkpoint. Its repository is approximately 167 GB, and its weights use mixed low-precision formats. Unsloth's July-release UD-Q4_K_XL is about 155 GB, with a separate optional speculative-decoding model adding roughly 11 GB. Official July files, quantization listing.

Even an idealized four-bit calculation is 142 GB before overhead. That already exceeds a single 96 GB card, a 128 GB Spark or a 128 GB Ryzen system. A 256–512 GB unified-memory computer or a suitable multi-GPU machine enters the capacity conversation, but still needs a compatible runtime and a measured memory budget. The vLLM deployment recipe documents server-oriented configurations; it is not validation of a Mac deployment.

Z.ai's GLM-5.3-Flash has 320B total parameters and 18B active. Its published FP8 repository is approximately 328 GB. Quantization changes the options: Unsloth lists an IQ4_XS version around 157 GB, while its Q4_K_XL version is around 200 GB. A 256 GB machine is therefore a candidate for some quantizations at constrained workloads; a 512 GB configuration offers more capacity. Those are estimates, not tested user-count or context guarantees. Official model and files, quantization artifacts.

The full GLM-5.3 is a different commitment. The vLLM recipe describes roughly 743B total and 39B active parameters, with eight H200 GPUs as a reference configuration and eight B200 GPUs with an FP8 attention cache for full-context serving. The original FP8 repository is roughly 756 GB. Unsloth's approximately 365 GB IQ4_XS conversion makes a 512 GB workstation a capacity candidate, but does not establish server-equivalent performance. Its roughly 467 GB Q4_K_XL leaves much less headroom. GLM-5.3 quantizations.

Two GPUs do not automatically act like one larger GPU. The runtime has to divide the model, and interconnect traffic can become a bottleneck. The RTX 5090 has no NVLink; multi-card plans must account for PCIe topology. CPU offloading can expand what loads, but benchmark the latency before buying around it.

Clustering Mac Studios and DGX Sparks

Several small computers can form a useful local inference cluster. First decide what you want the extra machines to do. Replicas keep a complete model on each machine and distribute requests between them. That serves more users and can improve availability. Sharding divides one model between machines so a larger weight set can fit. Every shard must be available to answer a request.

Each node keeps its own operating system, memory and runtime allocations. Two 256 GB Macs contain 512 GB of installed memory in aggregate, but do not become one Mac with a transparent 512 GB memory pool. The inference software must distribute the model and exchange intermediate results. That communication is why twice the hardware does not promise twice the speed.

Mac Studio: direct Thunderbolt links and MLX

Apple supports RDMA over Thunderbolt on Apple silicon Macs with Thunderbolt 5, starting with macOS 26.2. RDMA reduces the overhead of exchanging data between machines. It requires explicit enablement and compatible software; a Thunderbolt 5 cable does not add this feature to a Thunderbolt 4 Mac. Apple's RDMA technical note.

Two compatible Mac Studios can connect directly. For four machines, a full mesh needs six cables and three suitable ports per Mac: every machine connects to the other three. A ring uses fewer connections, linking each machine to its two neighbors. Apple demonstrates both topology choices in its distributed MLX session.

MLX can partition layers between nodes with pipeline parallelism, or divide computations within layers with tensor parallelism. The latter generally exchanges data more frequently. Match the model, partitioning method and topology to the actual runtime version: the MLX launch guide distinguishes its TCP ring backend from JACCL, while the current JACCL implementation also supports RDMA rings. exo provides a higher-level orchestration option with discovery and model distribution; its own model and topology requirements still apply.

A practical small Mac cluster therefore has two networks: direct Thunderbolt links for the distributed computation, and ordinary Ethernet for management, model downloads and client requests. A MikroTik Ethernet switch belongs on that Ethernet network. It does not replace the Thunderbolt cables. Ethernet-based distributed runtimes are another option, with different bandwidth and latency characteristics.

DGX Spark: direct pairs, a three-node ring or a switch

Each Spark contributes 128 GB of installed memory. NVIDIA documents a direct two-Spark connection over 200 GbE QSFP, without a switch. It also documents three Sparks connected in a ring. Two nodes contain 256 GB in total; four contain 512 GB, before per-node overhead and cache reserves.

For four or more, NVIDIA's switched-cluster playbook demonstrates four machines and specifies at least four 200 Gb/s switch connections, either native or through suitable 400G-to-two-200G breakouts. A pair of Sparks is consequently a capacity candidate for a distributed DeepSeek Flash deployment using the cited 155 GB weights. Whether the chosen runtime and quantization actually run well across that pair must be tested.

Use the ConnectX network for inter-node computation and keep management traffic separate. Spark's QSFP ports expose multiple logical interfaces, so follow the NVIDIA networking guide instead of treating two physical ports as a guaranteed 400 Gb/s host connection. NVIDIA's setup includes network and collective-communication checks before running a distributed workload.

MikroTik lowers the price of the Ethernet network

MikroTik makes the network a relatively modest line item beside several large-memory computers. These are its published suggested USD prices, checked on 5 September 2026; add cables or optics, compatible host interfaces, VAT and local reseller pricing.

The tradeoff is the speed of each complete path. A 100G switch caps a compatible Spark link at 100 Gb/s, below NVIDIA's 200G reference setup. Dividing bits by eight gives raw ceilings of 1.25 GB/s for 10G, 12.5 GB/s for 100G and 25 GB/s for 200G, before protocol overhead. These are network rates, not model generation speeds. A faster switch also cannot turn a Mac's built-in 10GbE connection into a 100G interface.

For RDMA over Ethernet, RoCE, the NICs and switch configuration must work together. MikroTik documents PFC to pause selected traffic classes and ECN to signal congestion, with a device support table and a RoCE example in its RouterOS QoS guide. Port speed alone does not establish a working RDMA network.

These MikroTik products are candidates to validate, not an NVIDIA-certified Spark shopping list. Check the exact firmware, NIC lane mode, cable or optics, forward-error correction and negotiated speed, then run NVIDIA's communication tests. SFP+ cages also need suitable modules or cables: a Mac's RJ45 Ethernet cable does not plug directly into one. Keep those accessories and the time to configure them in the cluster budget.

PCIe switching when the host runs out of lanes

A desktop CPU may expose too few PCIe lanes for several GPUs plus fast networking and storage. A PCIe switch lets several downstream devices share one upstream connection to the host. Broadcom's PEX families provide the switching silicon used in expansion boards, backplanes and servers. Broadcom PCIe switch overview.

Three different products are easily confused:

Supermicro's bifurcation explanation describes the lane split. A switched expansion backplane is different: One Stop Systems' 4UPF Gen4, for example, offers an OSS-538 configuration with one upstream x16 connection and four downstream x16 slots. The topology looks like this:

CPU — PCIe 4.0 x16 — PCIe switch
                      ├── x16 — GPU 1
                      ├── x16 — GPU 2
                      ├── x16 — GPU 3
                      └── x16 — GPU 4

This adds places to attach GPUs and their memory. It does not turn 16 CPU lanes into 64 CPU lanes. The shared PCIe 4.0 x16 uplink has a theoretical bandwidth of about 31.5 GB/s per direction before transaction overhead. All host-bound transfers share that limit. NVIDIA uses the same link-rate figure in its DGX A100 architecture description.

When the GPUs and drivers support peer-to-peer transfers, traffic between GPUs on the same switch can stay inside the switch instead of crossing the host uplink. That can make switching useful for inference where weights stay resident. Frequent CPU offloading or large transfers to a NIC outside that switch can expose the upstream bottleneck. Broadcom's PEX88000 architecture explains the local device-to-device paths.

Buy the complete topology, not just a chip name. Check GPU and driver peer-access support, firmware device enumeration and address-space allocation, switch generation, host-link width, cables, auxiliary power and cooling. NVIDIA's NCCL troubleshooting guide explains why PCIe topology and platform isolation settings affect peer transfers. A switch alone cannot enable a transfer the GPU or driver does not support.

Switched backplanes and expansion chassis are specialist purchases; they can cost enough that a workstation or server platform with more native lanes is the better complete-system deal. Compare both quotes. This is a way to expand a suitable PCIe host, rather than an upgrade path for the soldered GPUs inside a Mac Studio or DGX Spark.

What would it take to run Kimi K3 locally?

Moonshot AI's Kimi K3 takes the same principle much further: 2.8 trillion total parameters, 104B active per token, with native MXFP4 weights. Its official repository is approximately 1.56 TB. The active count does not turn it into a 104B-sized download. Kimi K3 model card, original files.

The vLLM K3 recipe specifies at least eight GB300 GPUs, or at least eight AMD MI355X/MI350X accelerators on its ROCm route, and recommends multiple nodes for production traffic. That is a concrete supported-runtime reference to take to a server integrator.

For scale, eight 96 GB desktop cards provide 768 GB nominally, roughly half the original repository's size. Eight desktop cards and eight server accelerators are very different proposals. A practical design needs supported model parallelism, fast interconnects, host RAM, several terabytes of SSD capacity, and power and cooling sized for the complete system.

Four 512 GB Mac Studios would instead offer about 2 TB of installed aggregate memory. That makes them interesting for capacity experiments, but leaves the K3 runtime, quantization support, per-node headroom and interconnect performance to prove. It is not a demonstrated substitute for the server recipe, and the newly announced M5 Ultra 512 GB configuration is due in late October.

Experimental extreme quantizations can change the answer to “can it load?” Unsloth publishes a roughly 466 GB one-bit variant, putting a 512 GB machine near the weight-capacity boundary. That leaves limited space for everything else and introduces a major quality question. It is not evidence that a 512 GB Mac delivers the capability or performance of the published K3 deployment. Unsloth Kimi K3 artifacts.

For a business, this is the point to commission a deployment benchmark and infrastructure quote. Local can mean a server room or privately operated colocation. Owning the deployment does not require the machine to sit under a desk.

A Chinese model can process your data entirely locally

A model's country of origin and the location of inference are separate questions. Downloading Qwen, GLM, DeepSeek or Kimi weights does not inherently send your prompts to Alibaba, Z.ai, DeepSeek or Moonshot. With downloaded weights and a local runtime, the computation can happen on your machine, including without an internet connection. LM Studio documents offline model and document processing.

That is a concrete privacy benefit. It also needs an accurate boundary: a cloud chat interface, remote embedding service, web-search tool, hosted gateway or telemetry integration can still move data elsewhere. Check the entire application path. Disconnecting external networking after setup is a useful way to verify that the intended local workflow continues to function.

Local execution does not remove model bias, inaccurate answers, unsafe tool use or software supply-chain risk. Evaluate model behavior on your tasks, obtain weights and runtimes from identifiable publishers, review any executable model code, and give tools only the access they need. These are deployment properties to examine for models from any country.

Open weights also do not mean identical licensing. Qwen3.8-27B uses Apache 2.0; GLM-5.3 and Kimi K3 publish their own license terms. Check the license shipped with the exact checkpoint before commercial deployment or redistribution. Qwen model license, GLM-5.3 license, Kimi K3 license.

For example, K3's license requires a separate agreement for certain Model-as-a-Service operators above $20 million in annual aggregate affiliate revenue. Its defined internal-use exception has its own boundaries. That matters when the investment is intended to power a service sold to customers. Kimi K3 commercial terms.

Beyond privacy, ownership lets you retain a tested model version, keep an offline workflow available, schedule batch work when capacity is idle, and adapt the surrounding application without a provider changing its endpoint or withdrawing that model version.

From a token bill to an investment in capacity

With a hosted API, usage commonly creates a variable token bill. Buying local hardware moves part of that spending into CAPEX: an up-front investment in capacity you control. Electricity, maintenance, software, people and replacement capacity remain OPEX.

This matters for a product's business model. If inference is heavily used and predictable, you can plan around the cost of a machine and its useful throughput. More activity can improve utilization without adding a provider charge for every generated token. Once the machine is full, however, more demand means queueing or another capacity purchase.

Consider an illustrative monthly budget, excluding VAT:

These are illustrative planning inputs. The power example assumes the system is off outside those hours. Add idle consumption, financing, cooling, backup capacity and any costs your operations allowance does not cover. The 36-month allocation spreads the purchase for this comparison; the cash outlay happens up front.

For a cluster, include every node, the switch, cables and optics, and any PCIe expansion chassis. Budget for orchestration and spare capacity as well. Distributing one model across several machines makes all those machines part of the cost of serving it.

At a hypothetical blended API price of €5 per million tokens, €435.15 corresponds to roughly 87 million tokens a month. At €0.50 per million, it corresponds to roughly 870 million. The model must actually deliver the required workload within the paid operating hours for that comparison to be meaningful.

Use a fixed, representative mix of input and output tokens, including cached input and batch discounts, when calculating the blended API rate. Then compare equivalent task quality and latency. Cost per successfully completed document or support request is often a better business metric than raw cost per token.

For intermittent use, an API can remain the cheaper choice. For steady private workloads, ownership can make unit costs and model availability more controllable. The opportunity is to choose the cost structure that matches how your business uses inference.

Buy for the workload you can prove

Start with one useful task and a representative evaluation set. Try a small Qwen model on existing hardware. If quality requires Qwen3.8-27B, compare a 32 GB GPU build against a suitably configured Mac. Move to the larger DeepSeek and GLM tiers only when the quality gain justifies the memory, operational effort and complete system price.

Before ordering, ask for the exact model file, quantization, runtime version, context length and concurrency in a benchmark. Record peak memory, wall power and successful task throughput. That turns a memory specification into evidence for a purchase.

When integrating the result with other applications, preserve the same clarity about where requests travel. Our local and public AI gateway guide explains that routing boundary, including the EU-hosted gateway hop. Choose the complete request path to match the privacy requirement that brought you to local inference in the first place.