How many tokens is my codebase? Claude counts 58% more than GPT

How many tokens is my codebase? If you count it with the usual tools and plan to send it to Claude, the answer is probably too low. I measured real Claude token counts for 33 files from five public repos, and on source code the current Claude models (Opus 4.7 and later, including Haiku, Sonnet and Opus 5.5) used 1.58 tokens for every token of OpenAI’s o200k_base, the encoding that Repomix and most counters use by default. A repo that counts 900,000 tokens there is about 1.4 million for Claude, which no longer fits in a 1M context window.

So I built a small counter that says so: it counts a repo with o200k_base, estimates Claude from ratios I measured per file type, lists the files that eat the budget, and prices one full send on eight models. On two repos I had not measured, its Claude estimate landed within 7.3% and 0.2% of the real count.

How many tokens is my codebase in Claude?

Claude has no offline tokenizer for current models. Anthropic’s answer is the token counting API, which needs an API key I don’t have. What I do have is Claude Code: claude -p --output-format json reports the input tokens of every run. So I sent each file followed by a short question, subtracted a run with the question alone, and got the file’s real token count, for five Claude models. The sample is 4 code files from each of five repos (requests, express, ky, cobra and ripgrep, picked by a hash of the path), plus Markdown, config files, a lockfile, a CSV and an SVG.

File type	Files	o200k_base	Claude 4.7 and later	Ratio	Characters per Claude token
Go	4	11,975	20,021	1.67	2.39
Rust	4	8,892	14,328	1.61	2.47
Python	4	13,868	21,420	1.54	2.65
TypeScript	4	7,200	11,061	1.54	2.44
JavaScript	4	3,071	4,500	1.47	2.67
All code	20	45,006	71,330	1.58	2.51
Markdown and reST	5	44,621	61,523	1.38	2.29
YAML, JSON, TOML	5	3,391	4,829	1.42	2.33
Cargo.lock	1	5,024	7,157	1.42	1.89
CSV	1	35,207	53,922	1.53	1.62
SVG	1	35,199	38,119	1.08	1.04

Single code files ranged from 1.43 to 1.84. The 2.51 characters per token for code matches Anthropic’s own rule of thumb that 1M tokens is about 2.5 million characters on the current tokenizer (models overview). The gap is specific to Claude: over all 742 text files of the five repos, OpenAI’s older cl100k_base, Qwen3’s tokenizer and DeepSeek-V3’s tokenizer all came within 7% of o200k_base on code.

Do Haiku, Sonnet and Opus count tokens the same way?

Yes, for every model on the new tokenizer. Haiku 5.5, Sonnet 5.5, Opus 5.5 and Opus 4.7 returned the same count for all 33 files, token for token. Sonnet 4.6, the last model on the previous tokenizer, used 193,794 tokens where the others used 236,880. So one ratio table covers the whole current lineup, and a second one covers 4.6 and earlier.

Is the new Claude tokenizer really 30% more tokens?

Anthropic’s pricing page says the tokenizer introduced with Opus 4.7 “produces approximately 30% more tokens for the same text”. On my files:

Content	Claude 4.7+ / Claude 4.6
20 code files	1.28
5 Markdown and reST files	1.25
5 config files	1.24
SVG logo	1.07
itsdangerous, whole repo	1.16
hexyl, whole repo	1.22

For code, “about 30%” holds. For a whole repo it was less, because repos carry lockfiles, images in SVG and docs, which grow less. Either way, the same prompt is more tokens on any current model than it was on Sonnet 4.6.

Is “characters divided by 4” good enough?

For OpenAI, on code, it is: the five repos averaged 3.99 characters per o200k_base token on code files. For Claude it is not. I sent two whole repos that were not in the sample to Sonnet 5.5 and Sonnet 4.6 and compared:

Repo	Model	Real count	My estimate	o200k_base as is	Characters / 4
hexyl (Rust, 21 files)	Sonnet 5.5	86,540	86,726 (+0.2%)	57,493 (-33.6%)	48,295 (-44.2%)
hexyl	Sonnet 4.6	71,089	69,787 (-1.8%)	57,493 (-19.1%)	48,295 (-32.1%)
itsdangerous (Python, 50 files)	Sonnet 5.5	144,753	155,311 (+7.3%)	110,978 (-23.3%)	70,635 (-51.2%)
itsdangerous	Sonnet 4.6	124,272	132,298 (+6.5%)	110,978 (-10.7%)	70,635 (-43.2%)

Characters divided by 4 did worst, partly because both repos hold files that tokenize badly (a lockfile and SVGs). But even plain o200k_base was a quarter to a third short for current Claude models. That is what Repomix reports by default (its config schema sets o200k_base), and its maintainer says in issue #476 that counts may not match Claude or Gemini. Counters that load the old Xenova/claude-tokenizer from Hugging Face did no better: on my code files it produced 0.71 tokens for every real Claude token.

If you are trimming a repo to fit, Repomix’s --compress is the other lever. I measured what it saves and what Claude can still answer on three of these repos: 12% to 61% fewer tokens, no wrong answers, but some Python functions silently disappear.

My estimate was 7.3% high on itsdangerous, almost certainly because of its uv.lock. My only lockfile sample was a Cargo.lock, and the estimate applies its ratio to every lockfile.

What fills a repo’s token count?

The biggest surprise was not the tokenizer. It was which files hold the tokens:

• requests: ext/requests-logo.svg is 563,459 of the repo’s 731,288 o200k_base tokens, 77%. With it, the Claude estimate is 858,236 tokens; with --exclude '*.svg', 248,010.
• itsdangerous: uv.lock is 67% (74,518 of 110,978). Without lockfiles, the Claude estimate drops from 155,311 to 49,123.
• ripgrep: three benchmark CSV files from 2016 to 2020 are 17.6% of the repo.

None of those files help a model understand the code. The counter lists the five largest files and, when one of them is more than a quarter of the total, suggests the flag that leaves it out:

$ python repotokens.py ~/src/itsdangerous
...
Largest files (o200k tokens):
     74,518   67.1%  uv.lock  (lockfile)
      6,984    6.3%  docs/_static/itsdangerous-name.svg
      3,661    3.3%  src/itsdangerous/serializer.py
      3,406    3.1%  docs/_static/itsdangerous-logo.svg
      2,171    2.0%  src/itsdangerous/signer.py
  One file is over a quarter of the total; if a model does not need it, try --exclude 'uv.lock' or --no-lockfiles.

How much does it cost to send a repo to Claude, GPT or Gemini?

The counter multiplies its counts by input prices from the official pages of Anthropic, OpenAI and Google, checked on 2026-10-16 and stored with their sources in prices.json. For ripgrep (228 text files, 3.2 million characters, 911,550 o200k_base tokens), one full send as input:

Model	Tokens used for the price	Input cost	Fits in context?
Claude Opus 5.5	1,410,572 (estimate)	$5.64	No, over 1M
Claude Sonnet 5.5	1,410,572 (estimate)	$2.82	No, over 1M
Claude Haiku 5.5	1,410,572 (estimate)	$0.71 (over-100k price)	No, over 1M
Claude Sonnet 4.6	1,123,330 (estimate)	$3.37	No, over 1M
GPT-5.4	911,550	$4.56 (over-272k price)	not checked
GPT-5.4 mini	911,550	$0.68	not checked
Gemini 3.1 Pro Preview	911,550 (approximation)	$3.65 (over-200k price)	not checked
Gemini 3.8 Flash	911,550 (approximation)	$0.68	not checked

Two price tiers bite here. Haiku 5.5 costs $0.10 per million input tokens up to 100,000 tokens and $0.50 above that, so the Claude count matters most exactly where o200k_base says you are just under the line. GPT-5.4 and Gemini 3.1 Pro double their input price above 272,000 and 200,000 tokens. The GPT and Gemini rows use o200k_base, which tiktoken 0.12.0 maps to the GPT-5 family; Google and the newest OpenAI models don’t publish an offline tokenizer, so those rows are approximations. Repeated sends with prompt caching cost far less than these one-off prices.

How I counted Claude tokens without an API key

The core of measure_claude.py is one function (shown without its retry loop). It replaces Claude Code’s system prompt with one line, turns all tools off, and adds up the three input fields of the usage report:

def claude_input_tokens(model, prompt):
    p = subprocess.run(
        ["claude", "-p", "--model", model, "--system-prompt", "You reply with OK.",
         "--tools", "", "--strict-mcp-config", "--setting-sources", "",
         "--output-format", "json"],
        input=prompt, capture_output=True, text=True, env=ENV, cwd="/tmp", timeout=300)
    u = json.loads(p.stdout)["usage"]
    return (u["input_tokens"] + u.get("cache_creation_input_tokens", 0)
            + u.get("cache_read_input_tokens", 0))

base = claude_input_tokens(model, PREFIX + QUESTION)
tokens = claude_input_tokens(model, PREFIX + text + "\n\n" + QUESTION) - base

The baseline was 450 to 606 tokens depending on the model, and it was identical before and after every batch of files. ENV sets CLAUDE_CODE_DISABLE_CLAUDE_MDS=1 and CLAUDE_CODE_DISABLE_AUTO_MEMORY=1, because my last benchmark of Claude Code showed that personal instructions leak into claude -p runs otherwise. The two measurement passes took 12 and 14 minutes. The clean pass made 173 measuring runs that sent 1.66 million input tokens, about $4.52 of input at API prices plus a few small baseline runs; it all ran through Claude Code on a subscription.

The PREFIX line is there because of the first pass.

What broke and what I’d do differently

• A prompt that starts with / changes what Claude Code sends. My first pass gave six files absurd counts: a 1,000-character Go file came out at 122 tokens, and its license header alone at minus 87. All six start with a // comment. With a leading slash, claude -p sends about 300 fewer tokens of its own context (244 input tokens for // hello there against 548 for hello there), while the model still sees the whole text. The baseline no longer matched, so the subtraction broke. Every prompt now starts with “Here is a file.”, and I re-ran all 145 measurements; the Go file is 427 tokens. The bad pass is kept in the repo.
• The sample is small. 33 files, two held-out repos. The per-type ratios are averages, and the SVG and lockfile rows rest on one file each. A different SVG (the requests logo) had 1.57 characters per o200k_base token against 1.13 for the one I measured, so treat SVG estimates loosely.
• Lockfiles need their own measurements. uv.lock, package-lock.json and Cargo.lock are different formats, and the estimate treats them as one. I later measured nine lockfiles and whether Claude Code actually reads them: it greps about 1 KB, and .claudeignore does not keep it out.
• I didn’t measure Gemini or the newest GPT models. Their counts in the table are o200k_base, labeled as an approximation.
• The machine was busy. The load average was around 6 on a 4-thread laptop CPU during the CLI timings, so 3.4 s for ripgrep is an upper bound, not a benchmark.

If you have Claude Code, you can check a single file the same way with two claude -p calls (about 5 seconds each here); if you have an API key, the token counting API is the exact answer. For a whole repo before you paste it, the estimate gets you within a few percent, and the list of the largest files is often the more useful half. The code, the prices, every measurement and the discarded pass are in the project folder.