Chunking website content for retrieval

Chunking is the step where you cut content into retrievable pieces. It is usually treated as a parameter to tune, and the tuning discussion is almost always about size. Size matters less than where the cuts land.

The failure mode that size tuning cannot fix

Consider a paragraph explaining what happens when you hit a plan limit. A fixed 400 character window might cut it after "when you reach your monthly limit, the assistant", leaving the actual consequence in the next chunk.

Now retrieval on "what happens at the limit" returns the first chunk, which contains the question's vocabulary but not its answer. The model sees a truncated sentence and either says it does not know or completes the thought itself, which is worse.

No chunk size fixes this. The problem is that the boundary ignored the structure of the text.

Split on structure first

HTML gives you a document outline for free and most chunking implementations throw it away. Headings mark topic boundaries that a human author already decided on.

A better default: split at heading boundaries, then split further only if a section exceeds your size ceiling, and when you do, split at paragraph boundaries rather than character counts.

def chunk(html_doc, max_chars=1200):
    sections = split_at_headings(html_doc)   # h2, h3
    out = []
    for sec in sections:
        if len(sec.text) <= max_chars:
            out.append(sec)
        else:
            out.extend(split_at_paragraphs(sec, max_chars))
    return out

Carry the heading path into the chunk

A chunk lifted out of the middle of a document loses its context. The text might say "this is included on all paid plans" with no indication of what "this" refers to, because the answer was in a heading two levels up.

Prefixing each chunk with its heading path fixes this cheaply and improves both retrieval and the model's ability to use the chunk.

Pricing > Plan limits > Overage

When you reach your monthly credit limit, ...

This costs a few tokens per chunk and it is one of the highest return changes available.

Overlap, and why it is often overrated

Standard advice is to overlap chunks so that content near a boundary appears in both. This helps with fixed size chunking because boundaries are arbitrary.

With structural chunking, boundaries are meaningful, and overlap mostly adds duplicate content to the index. Duplicates compete with each other in retrieval and consume context budget. If you split structurally, start with no overlap and add it only if you can measure a problem it solves.

Content that resists chunking

Three types need special handling:

• Tables. Splitting a table separates rows from headers. Keep small tables whole or convert each row to a sentence.
• Lists where every item depends on the introduction. Keep the introduction with the list.
• Navigation, footers and cookie banners. Strip them before chunking or every chunk carries the same noise and similarity scores get flattened.

That last one is the most common preprocessing bug and it is invisible unless you look at your actual chunks. Print ten of them and read them. If they all start with your main navigation, that is your first fix and it is worth more than any parameter tuning.

How to know it is working

Take twenty real questions, retrieve for each, and read the top chunk. Not the answer, the chunk. If a human reading only that chunk could answer the question, retrieval is doing its job. If they could not, no amount of prompt engineering downstream will save it.