LLM Text Chunker: Split Text by Tokens, Free

LLM Text Chunker

Split long text into chunks that fit a context window or a retrieval pipeline, by tokens, characters or words.

Your text stays in your browser. Nothing is uploaded or saved.

Chunk settings

The end of each chunk is repeated at the start of the next one. It must be less than half the chunk size. Use 0 for none.

A paragraph or sentence that is too big for one chunk is cut at the next smaller unit.

Working. Large texts can take a few seconds.

The exact tokenizer could not be loaded, so token counts are estimates (about 4 characters per token). Reload the page to try again.

Save all chunks

Token counts use o200k_base, the tokenizer OpenAI publishes for its GPT models. Other models count differently, so leave some room below their limit.

An LLM text chunker splits long text into smaller, numbered pieces that fit a model's context window or a retrieval index. Paste text or drop a file, choose a size in tokens, characters or words, set an overlap, and you get copy-ready chunks you can export as JSON, JSONL, CSV or a plain text file.

What Is Text Chunking for LLMs?

Text chunking is the process of cutting a long document into smaller segments so a language model or a search index can handle each one. Models accept only a limited number of tokens per request, and retrieval systems work better when each stored passage covers one idea.

A token is the unit a model reads. It is often a short word or part of a longer word, and English text averages roughly four characters per token. Because models count in tokens, not words, a 1,000-word article can land anywhere from about 1,300 to 1,500 tokens depending on vocabulary and formatting.

Chunking has two settings that matter most:

  • Chunk size: the largest a single piece may be, measured in tokens, characters or words.
  • Overlap: the tail of one chunk that is repeated at the start of the next, so a sentence cut near a boundary still appears whole somewhere.

How to Use the LLM Text Chunker

You can split a document in a few clicks. Here is the full flow, using the labels you see on the page.

  1. Paste your text into Text to split, or drop a .txt or .md file on the box. You can also use Or choose a text file (up to 10 MB). Click Load sample if you want to see how it behaves first.
  2. Under Chunk settings, pick a unit in Measure chunks in: Tokens, Characters or Words.
  3. Set Largest chunk size by typing a number, or choose a preset from Common token sizes (256 up to 16,000 tokens). Then set Overlap between chunks, or use 0 for none.
  4. Choose Try to cut at: paragraphs, sentences, words, or the exact size.
  5. Tick or untick Keep a Markdown heading with the text under it and Start each chunk with a label such as [Part 2 of 7].
  6. Click Split text. Copy a single chunk with Copy chunk, or use Copy all, JSON, JSONL, Text file or CSV under Save all chunks.

Worked Example: Splitting a 2,400-Token Guide

Say you have a 2,400-token Markdown guide that you want to index for a question-answering bot. You choose Tokens, 512 from Common token sizes, an overlap of 50, paragraph-first cutting and the heading option turned on.

Each chunk after the first begins with the last 50 tokens of the one before it. That means every chunk adds at most about 460 new tokens. Divide 2,400 by roughly 460 and you get a little over five, so expect about six chunks. Because the tool prefers to cut at paragraph ends, chunks usually come in a bit under 512 rather than exactly at it.

The summary might show an average chunk of around 430 tokens and a largest chunk just under 512. That is a healthy result: nothing exceeds your limit and the pieces are fairly even. If a heading such as "Feeding the starter" sits above a paragraph, the heading stays attached to that paragraph instead of being stranded at the end of the previous chunk.

If you need the chunks in a pipeline, click JSONL to get one chunk per line, which is a common format for batch jobs and embedding scripts.

How to Read and Act on the Results

The results panel shows the chunk count, the average chunk and the largest chunk, followed by each chunk as a card. Use these numbers to check whether your settings did what you intended.

Largest chunk close to your limit

This is normal and means the splitter used the space available. Remember that a model also needs room for your instructions and its reply. If you plan to paste a chunk into a chat with a 4,000-token prompt budget, choose a size below that, such as 3,000, not 4,000.

Average chunk far below the largest

A big gap usually means short paragraphs or headings forced early cuts. Try Sentences, then words for even pieces, or raise the size so more paragraphs fit together.

Too many tiny chunks

Very small pieces lose context and bloat an index. Increase the size, or reduce overlap. Overlap that is too high repeats a lot of text, which inflates your total token count.

Sentences cut in odd places

Switch Try to cut at from the exact size to a paragraph or sentence option. The exact setting cuts at the limit, even in the middle of a word, so use it only when a strict length matters more than readability.

Suggested starting points

These are common guidelines, not rules. Test with your own content and questions.

GoalChunk sizeOverlap
Retrieval (RAG) passages256 to 512 tokens10 to 20%
Pasting long text into a chat2,000 to 4,000 tokens0 to 5%
Summarizing in large passes8,000 to 16,000 tokensSmall or none

Who Needs an LLM Text Chunker?

Anyone who works with text that is longer than a model can read in one go will benefit. These are the situations where it earns its place:

  • RAG builders: prepare documentation, help articles or transcripts for an embedding step with consistent passage sizes.
  • Chat users: feed a long report into a chatbot in numbered parts. The [Part 2 of 7] label tells the model and you where each piece fits.
  • SEO and content teams: break a site's articles into passages to test how an AI answer engine might quote them. Convert a page first with the URL to Markdown converter, then chunk the result.
  • Developers: export JSON, JSONL or CSV to feed scripts, spreadsheets or batch jobs without writing a splitter.
  • Researchers and editors: divide a long interview or paper into sections for summarizing, translating or fact-checking.

Best Practices and Common Mistakes

Good chunks come from matching the settings to the job, not from picking the biggest number. These habits prevent most problems.

  • Clean the text first. Hidden characters, odd spaces and stray citation marks waste tokens and can break cuts. Run messy copy through the AI text cleaner before you split.
  • Check your total first. If you only need to know whether a document fits in one request, count it with the token calculator. Chunk only when it does not.
  • Keep headings with their text. Leave the Markdown heading option on for structured documents. A passage with its heading is easier for a retriever to match.
  • Use the label for manual workflows. The part label helps when you paste chunks into a chat one by one. Skip it for embeddings, where the label becomes noise in the stored text.
  • Leave headroom. Labels and headings add a few tokens, so set the size slightly below the true limit of your target model.
  • Do not use half-size overlap. Overlap must stay below half the chunk size, and in practice 10 to 20 percent is plenty.

Troubleshooting messages

If the tool shows that the chunk size was raised or lowered, your number fell outside the allowed range. The minimum is 20 for every unit. The maximum is 100,000 for tokens and words, and 1,000,000 for characters. If the overlap was reduced, it was too large for the chunk size, so lower it or increase the size. If a file is rejected, it is either larger than 10 MB or could not be read, so paste the text directly or split the file first.

Limits and Accuracy Notes

The chunker reports token counts with o200k_base, the tokenizer OpenAI publishes for its GPT models. Other model families count tokens differently, so the same chunk may be a little larger or smaller elsewhere. Keep a safety margin of 5 to 10 percent below the limit of any non-OpenAI model.

If the exact tokenizer cannot load, the page shows a note and counts become estimates of about four characters per token. Reload the page to try again before relying on tight limits.

When a text produces many chunks, the page previews them as cards and may not list every one on screen. Use the export buttons under Save all chunks to get the complete set. Also note that the tool splits text only. It does not judge meaning, so very different topics inside one paragraph can still end up in the same chunk.

Once your content is chunked, you can check how crawlers see the original page with the AI crawler view, or run the AI search readiness checker. For more related utilities, browse the AI SEO tools hub.

Paste your text into the box above, pick a size and overlap that suit your model, and click Split text to get clean, numbered chunks you can copy or export in seconds.

Frequently Asked Questions

What does an LLM text chunker do?
An LLM text chunker splits long text into smaller, numbered pieces that fit a model's context window or a retrieval index. You pick a size in tokens, characters or words, plus an overlap, and the tool returns chunks you can copy or export as JSON, JSONL, CSV or text.
How do I split text by tokens for ChatGPT?
Paste your text, keep Measure chunks in set to Tokens, choose a size such as 4,000 from Common token sizes, then click Split text. Turn on the part label option so each piece starts with a marker like [Part 2 of 7]. Keep the size below the model limit to leave room for your prompt.
What chunk size and overlap should I use for RAG?
A common starting point for RAG is 256 to 512 tokens per chunk with 10 to 20 percent overlap. Smaller chunks match more precisely, while larger ones keep more context. Test with your own questions, then adjust the size and run Split text again.
Can I upload a file to the LLM text chunker?
Yes, you can drop a .txt or .md file on the text box or use the file chooser. Files up to 10 MB are accepted. If a file is larger or cannot be read, paste the text directly or split the file into parts first.
Why was my overlap or chunk size changed?
The chunker shows a note when it adjusts values that fall outside the allowed range. Size has a minimum of 20, and overlap must be less than half the chunk size. Lower the overlap or change the size, then click Split text again.
Are the token counts exact, and is the tool free?
The LLM text chunker is free and needs no sign-in. Token counts use o200k_base, the tokenizer OpenAI publishes for its GPT models. Other models count differently, so keep some room below their limits. If the exact tokenizer cannot load, counts are estimates.
Recommended

Explore VPN options

A VPN changes your network exit. It does not guarantee anonymity.

Hide Your IP Encryption Network coverage
Protect My IP Now