aliteq.

Type to search every guide, tool and GPU page.

Can you run Inkling 2 Mini locally? Yes, even on a 12 GB GPU, with one long-context catch

Thinking Machines' Inkling 2 Mini is a 26.4B looped MoE that fits a 12 GB card at 2-bit and a 24 GB card at 4-bit. What each GPU needs, the speeds people posted from an 8 GB RTX 4060 to an RTX 5090, and why its memory climbs faster at long context than a 26B model should. Checked Nov 21, 2026.

Ari ValePublished 9 min read
Can you run Inkling 2 Mini thumbnail with the Thinking Machines wordmark and the headline A 26B model on a 12 GB GPU, the Unsloth logo next to 2-bit GGUF 8.5 GB and 4-bit 16.6 GB, and a board of community decode speeds: RTX 4060 8 GB 17, RTX 3060 12 GB 41, RTX 3090 48, RTX 4090 65 and RTX 5090 417 tokens per second, posted on X Nov 21, 2026

Some links on this page are referral links: we may earn a commission, at no cost to you.

Share

Yes, you can run Inkling 2 Mini locally, and you need less GPU than the 26B size suggests. Unsloth's 2-bit GGUF is 8.5 GB, small enough that one owner runs it fully on a 12 GB RTX 3060. The 4-bit Q4_K_M file is 16.6 GB, which wants a 24 GB card like an RTX 3090 or 4090. An 8 GB card still runs the 3-bit file with part of the model in system RAM. The catch is long context: the model runs its middle layers three times, so its memory grows faster with long prompts than a normal 26B model, and today's llama.cpp builds add more on top. Checked November 21, 2026, the day after release, on the model card, the config file, Unsloth's files and the first community runs.

Inkling 2 Mini in four numbers

26.4B

total parameters

model card

3.4B

active per token

5.4B effective with the loop

48

layer passes per token

24 stored layers

8.5 GB

smallest practical GGUF

Unsloth UD-IQ2_XXS

What is Inkling 2 Mini?

Inkling 2 Mini is an open-weights model from Thinking Machines Lab, released on November 20, 2026 under the Apache 2.0 licence. It reads text and images and writes text, with a context window of up to 1M tokens. The model card lists 26.4B parameters with 3.4B active per token, 128 experts with 8 picked for each token, and a single checkpoint of about 20 GB in mixed NVFP4 and MXFP8 precision (NVFP4 is NVIDIA's 4-bit format, explained here).

The unusual part is the loop. The model stores 24 layers of weights, but the middle 12 run three times for every token, which the card counts as "48 layer applications per token". That is why it's small on disk but does more work per token than its 3.4B active size suggests: the card puts the effective active count at 5.4B.

One token through Inkling 2 Mini
  1. Layers 0 to 5, once

    6 passes, one of them global attention

  2. Layers 6 to 17, three times

    same weights, 36 passes, 6 of them global

  3. Layers 18 to 23, once

    6 passes, one of them global

  4. Next token

    48 passes in total

Layer numbers from the model's config.json (loop_config core_start 6, core_end 18, num_passes 3), read Nov 21, 2026.

How much VRAM does Inkling 2 Mini need?

About 9 GB at 2-bit, 14 GB at 3-bit and 17 to 18 GB at 4-bit, before context. These are Unsloth's GGUF file sizes, which is what you download and load. Unsloth's guide gives the same ranges for RAM plus VRAM: 9 GB at 1-bit, 10 to 11 GB at 2-bit, 13 to 15 GB at 3-bit, 17 to 18 GB at 4-bit and 24 GB at 8-bit. The vision projector adds 0.64 GB if you want image input.

Inkling 2 Mini GGUF size by quant (Unsloth, GB)

UD-IQ2_XXS (2-bit)8.5 GB

fits a 12 GB card with room for context

UD-Q2_K_XL9.1 GB
UD-Q3_K_XL (3-bit)13.6 GB

8 GB card plus system RAM

UD-Q4_K_M (4-bit)16.6 GB

24 GB card, or 16 GB with offload

UD-Q4_K_XL (Unsloth's pick)17.6 GB
Official NVFP4 + MXFP820.2 GB

vLLM, SGLang; card recommends 32 GB

UD-Q8_K_XL22.9 GB

Unsloth calls it full precision

Our Inkling 2 Mini cost-to-run page runs the same model through our VRAM engine for every GPU we track. It puts Q4_K_M at 14.9 GB of weights (binary gigabytes, so about 16 GB in the decimal units Hugging Face uses), close to Unsloth's 16.6 GB file; Unsloth's dynamic quants keep some layers at higher precision, so the real files run a little larger. For any other card or context, use the VRAM calculator.

Which GPU runs which Inkling 2 Mini file (Nov 21, 2026)

8 GB (RTX 4060, 3060 Ti, 3070)

Best file
UD-Q3_K_XL, 13.6 GB
What to expect
Runs with 12 expert layers in system RAM; one owner reports 17 tok/s with 16 GB DDR4

12 GB (RTX 3060 12GB, 4070)

Best file
UD-IQ2_XXS, 8.5 GB
What to expect
Fully on the GPU; ~41 tok/s reported. 4-bit needs offload (~18 tok/s)

16 GB (RTX 5060 Ti, 4060 Ti 16GB, 5080)

Best file
IQ4 or Q3 class
What to expect
4-bit slightly overflows; a few experts go to the CPU, ~43 tok/s reported

24 GB (RTX 3090, 4090)

Best file
UD-Q4_K_XL, 17.6 GB
What to expect
Fully on the GPU with room for long context; 48 to 65 tok/s reported

32 GB (RTX 5090)

Best file
Official checkpoint or Q8
What to expect
The card's own recommendation for the 20 GB release; 417 tok/s reported with vLLM

Mac, 32 GB or more

Best file
MLX or GGUF
What to expect
Thinking Machines lists Apple silicon with 32 GB or more
Does Inkling 2 Mini 4-bit fit your card?
Inkling 2 Mini Q4_K_M at 32K context needs ≈17.5 GB
RTX 3060 12GB12 GBover 5.5 GB
RTX 5060 Ti 16GB16 GBover 1.5 GB
RTX 3090 / 4090 24GB24 GBfits
RTX 5090 32GB32 GBfits

Doesn't fit your card?

Rent a 24 GB+ GPU for the hours Inkling 2 Mini Q4_K_M at 32K context needs, then stop it.

No 24 GB card? Rent one by the hour and run the 4-bit file fully on the GPU.

New to Runpod? Sign up with Google through this link and load $10: Runpod adds a one-time bonus credit, $5 in Europe and a random $5 to $500 elsewhere (most people get $5), valid 90 days.

We earn a commission if you buy through this link.

16.6 GB file + about 1 GB of KV cache at 32K (fp16, our engine) + runtime overhead. The 16 GB cards fall just short, so llama.cpp moves a few experts to the CPU; the 12 GB card needs the 2-bit file instead.

Why does Inkling 2 Mini use more memory at long context?

Because each pass through the loop keeps its own KV cache, so the model holds 8 full-context caches where a normal 24-layer model of this design would hold 4. The KV cache is the memory that stores your conversation (our context window explainer shows how it grows). In the config, 20 of the 24 layers use a 1,024-token sliding window, which caps their cache at a small fixed size. Only layers 5, 11, 17 and 23 look at the whole context. Layers 11 and 17 sit inside the loop, so they run three times and each pass computes its own keys and values. Count the passes and you get 8 global caches instead of 4.

Our engine works that out at 32 KB per token at full fp16 precision: 1 GB at 32K, 4 GB at 131K, 8 GB at 262K and 32 GB at the full 1M. The 40 sliding-window passes add a fixed 0.16 GB on top whatever the length. Quantizing the cache to q8_0 halves it and q4_0 cuts it to about a quarter (how KV cache quantization works).

Inkling 2 Mini KV cache by context (our VRAM engine, Nov 21, 2026)

32K tokens

fp16 cache
1.0 GB
q8_0 cache
0.5 GB
q4_0 cache
0.3 GB

131K tokens

fp16 cache
4.0 GB
q8_0 cache
2.1 GB
q4_0 cache
1.1 GB

262K tokens

fp16 cache
8.0 GB
q8_0 cache
4.3 GB
q4_0 cache
2.3 GB

1M tokens (the maximum)

fp16 cache
32 GB
q8_0 cache
17 GB
q4_0 cache
9 GB

That math is why the 12 GB RTX 3060 run works at all. The 8.5 GB 2-bit file plus a q4_0 cache at 262K comes to roughly 11 GB, and Dogukan Urker, who posted that run, set the cache to q4_0 for exactly that reason, writing that it "halves the cache so 262K fits" (he also notes he has only tested down to 64K of real prompt so far).

How fast is Inkling 2 Mini on my GPU?

These are decode speeds that owners posted on X on November 21, 2026, the day after release. They are their runs, not ours, on different engines and settings, so read them as a rough ranking. Loktar, who measured 47 to 48 tokens a second on a single RTX 3090, put it plainly: "Speed is the engine not the model".

Inkling 2 Mini decode speed people posted (tokens per second)

RTX 4060 8 GB, 3-bit, experts in RAM17

@analogalok, llama.cpp, 16 GB DDR4

RTX 3060 12 GB, 4-bit, experts in RAM18

@DogukanUrker, llama.cpp

RTX 3060 12 GB, 2-bit, all on GPU~41

@DogukanUrker, llama.cpp

RTX 5060 Ti 16 GB, IQ4_NL43

@niklaslenz_ai, llama.cpp, some experts on CPU

DGX Spark, official checkpoint47-56

@majewskizby, SGLang

RTX 3090, llama.cpp47-48

@loktar00

RTX 4090, Q4_K_M at 200K context~65

@CrtMods17622

M5 Max 48 GB MacBook Pro125

@exolabs, EXO

RTX 3090, custom engine129.5

@superalesha, UD-Q4_K_XL, FP8 KV

M5 Ultra 96 GB Mac Studio210

@exolabs, EXO

RTX 5090, vLLM, code prompt417

@niklaslenz_ai, official image with speculative drafts

A few details behind those numbers:

  • 8 GB RTX 4060: Alok ran Unsloth's UD-Q3_K_XL with --n-cpu-moe 12, which keeps 12 expert layers in system RAM (what that flag does), and a q8_0 cache. Cutting the context setting from 24,000 to 18,000 tokens took decode from 7.43 to 17.11 tokens a second on the same prompt, so leave headroom rather than maxing the context.
  • 12 GB RTX 3060: the 2-bit file sits fully on the GPU at about 41 tokens a second. The 4-bit file, Dogukan Urker replied, "needs cpu offload, and offload is slow on this model", about 18 tokens a second with all experts in RAM.
  • 24 GB RTX 3090: llama.cpp gives 47 to 48 tokens a second. Alexey Fateev wrote a custom engine that reaches 129.5 on the same card (against 57 in his llama.cpp run), and Loktar reports 147 to 179 tokens a second on two 3090s with vLLM. Engines are moving fast, so these numbers will climb.
  • RTX 5090: 416.6 tokens a second on code with vLLM in the official image, using speculative decoding, per Niklas Lenz. His 16 GB RTX 5060 Ti managed 43.1 with llama.cpp, because the 4-bit file doesn't fully fit.

If you'd rather compare against what you already run, the closest rival is Qwen3.8-27B (what that one needs, and its cost-to-run page). Lasse measured Inkling decoding about 1.75 times faster than Qwen3.8-27B at nearly the same file size on his 32 GB card, because only 3.4B parameters work per token (why MoE models are fast).

Is Inkling 2 Mini good enough to switch from Qwen3.8-27B?

For instruction following, maybe; for agent work with tools, not yet. Lasse's paired tests are the most careful early numbers we found. On IFBench, Inkling scored 7.3 points higher than Qwen3.8-27B over 300 prompts. In a follow-up on multi-turn tool calling (BFCL), he found Inkling clearly behind his comparison model: 48 of 100 tasks against 70 in his 3-bit llama.cpp pair, 49 against 59 at 4-bit, and 46.7% against 58.7% with the official checkpoint. His own verdict: "Not a reason to replace an optimised Qwen setup yet."

Inkling 2 Mini vs Qwen3.8-27B in one owner's paired tests

Inkling 2 Mini

26.4B MoE, 3.4B active

vs

Qwen3.8-27B

27.8B dense

about 1.75x faster
Decode speed, same GPU and file size
baseline
+7.3 points
IFBench, 300 prompts
baseline
31.2 GiB
Memory at 131K context, day-0 llama.cpp
20.9 GiB
Mini wins 2wins 1 Qwen3.8-27B

Thinking Machines' own card is frank about the rough edges too. It says the model can get stuck in "long reasoning loops", is less reliable with tools than larger models and may take destructive actions without confirmation, so cap output tokens and agent turns. The card's benchmark table puts it ahead of Qwen3.8-27B on several coding tests; treat those as the vendor's numbers.

How do I run Inkling 2 Mini?

  1. Update your runtime. llama.cpp added Inkling 2 Mini on November 20, 2026 (pull request 32694) and merged a fix for Unsloth's quants on November 21 (pull request 32704), so use a build from after that. vLLM, SGLang, Unsloth, Ollama on Apple silicon and EXO are listed on the model card.
  2. Pick the file for your VRAM from the table above: UD-IQ2_XXS for 12 GB, UD-Q3_K_XL for 8 GB with offload, UD-Q4_K_XL for 24 GB.
  3. Use the recommended sampling from Unsloth's guide: temperature 1.0, top-p 1.0, top-k 0, min-p 0.0, with --jinja for the chat template.
  4. Start with 32K context, quantize the cache with --cache-type-k q8_0 --cache-type-v q8_0 (or q4_0 to go long), and raise the context only after it loads.

Renting is the cheap way to test before buying a card. Our cloud GPU price tracker lists the cheapest 24 GB and 32 GB cards right now, and the Inkling 2 Mini cost-to-run page ranks the cards that fit by live price.

Which GPU should I buy for Inkling 2 Mini?

If you already own a 12 GB card, try the 2-bit file before buying anything. If you're shopping, a 24 GB card is the comfortable size: the 4-bit file fits with room for long context, and a used RTX 3090 is the usual value pick (3090 buying guide). For other models in the same size class, see the best local LLMs for 24 GB, for 16 GB and for 12 GB, and which MoE models run best on 12 to 16 GB.

Inkling 2 Mini questions

Can Inkling 2 Mini run on an 8 GB GPU?
Yes, slowly. An owner ran Unsloth's 13.6 GB UD-Q3_K_XL file on an 8 GB RTX 4060 with 16 GB of DDR4, keeping 12 expert layers in system RAM, and got 17.11 tokens a second (posted Nov 21, 2026).
How much VRAM does Inkling 2 Mini need at 4-bit?
Unsloth's Q4_K_M file is 16.6 GB and its recommended UD-Q4_K_XL is 17.6 GB, plus about 1 GB of KV cache at 32K context. A 24 GB card runs it fully on the GPU; a 16 GB card needs a few experts on the CPU.
Why does Inkling 2 Mini need more memory at long context than other 26B models?
Its middle 12 layers run three times per token, and two of the four full-attention layers sit inside that loop, so it keeps 8 full-context KV caches instead of 4. Day-0 llama.cpp builds also lack flash attention for this design, which adds working memory that grows with context.
Is Inkling 2 Mini free to use commercially?
Yes. It is released under Apache 2.0, per the model card on Hugging Face.
What is the maximum context length of Inkling 2 Mini?
Up to 1M tokens per the model card. Community runs so far go to 262K on a 12 GB card with a q4_0 cache and about 344K on a 24 GB RTX 3090 with a custom engine.

Found this useful? Share it

Share

Sources

What this guide was researched from. How we check facts and compute numbers: methodology.

Founder & chief editor

Ari Vale

Founder and chief editor of Aliteq (a PC hardware shop from 2019, now a US-based publication). Ari Vale is my pen name: I stay anonymous, the work doesn't. I research the questions people actually ask, check every price and limit against the vendor's own pages, and write down what I'd choose and why. I use the cloud-GPU and AI-video services I write about.

Work out the hardware

GPUs in this article

Specs, the AI models each one runs, and where to buy or rent it.

Email list

Local AI, only the changes that matter

A new model your GPU can actually run, a GPU price move worth acting on, or a fix that saves you an evening. Nothing in between.

Written and sent by hand. One-click unsubscribe in every email. Emails may contain referral links.

Keep reading