I built a retrieval system over forty thousand Indian court judgments. The retrieval part gets the attention, but most of the engineering went into getting the documents in. Ingestion is where distributed systems stop being a textbook topic and start costing you sleep.
Here is what I would tell myself before starting.
One machine is fine until it isn't
The first version was a Python script. It read a judgment, split it into chunks, called an embedding model, wrote rows to Postgres. Forty thousand documents at roughly two seconds each is about a day of wall clock. That's fine once. It's not fine when you change the chunking strategy and have to redo everything, which happened four times.
The moment you need to rerun work, you need a queue. I moved the pipeline to Kafka with Go consumers. Each judgment became a message. Each stage (parse, chunk, embed, index) became a consumer group. The script became a topology.
Partitions are a promise about ordering, nothing else
Kafka gives you ordering within a partition. It gives you nothing across partitions. I keyed messages by judgment ID so all chunks of one judgment land on the same partition and arrive in order. That's the only ordering guarantee I actually needed, and it cost nothing.
What I got wrong at first was partition count. I started with three partitions and twelve consumers. Nine consumers sat idle. Partitions are your parallelism ceiling. Pick more than you think you need, because adding them later reshuffles keys.
Every consumer must be idempotent
Consumers crash. Kafka redelivers. If your consumer writes a chunk row and then crashes before committing the offset, it will write that chunk again on restart. I found this out with duplicate chunks polluting search results.
The fix is boring: derive a deterministic ID for every write. Chunk ID became a hash of judgment ID plus chunk index. Writes became upserts. Once every stage was safe to replay, retries stopped being scary and I could restart consumers whenever I wanted.
chunkID := fmt.Sprintf("%s:%04d", judgmentID, index)
_, err := db.Exec(`
INSERT INTO chunks (id, judgment_id, text, embedding)
VALUES ($1, $2, $3, $4)
ON CONFLICT (id) DO UPDATE SET text = EXCLUDED.text, embedding = EXCLUDED.embedding
`, chunkID, judgmentID, text, vec)
Exactly-once delivery is mostly a marketing phrase. Idempotent consumers give you the same result with far less machinery.
Backpressure is the feature you forget
The embedding stage was the slow one. Parsing and chunking could produce thousands of messages a minute. Embedding could consume a few hundred. Without limits, the embed topic grew without bound and the embedding API started rate limiting me, which made it slower, which made the backlog grow faster.
Two things fixed it. Consumers now pull with a bounded in-flight count instead of grabbing everything available. And the producer for the embed stage checks lag and slows down when the topic falls behind. Neither is clever. Both were missing.
Cache the hot path, not everything
Redis sits in front of retrieval. The temptation is to cache every query. In practice, legal search has a long tail; most queries are unique. What repeats is the second step: given a chunk hit, fetch the full judgment metadata and neighbouring paragraphs. That lookup is identical across many queries and it's what I cache. Hit rate went from near zero to over eighty percent by caching the right thing.
Observability before optimisation
I added per-stage lag, throughput and error counters to a small dashboard before I touched performance. Every optimisation after that was a response to a number I could see. The ones I guessed at before the dashboard existed were mostly wasted.
The short version
- Queue early. The first rerun pays for it.
- Key by the thing that needs ordering. Ignore global order.
- Make every write an upsert with a deterministic ID.
- Bound in-flight work. Let slow stages slow down fast ones.
- Cache the repeated lookup, not the unique query.
- Measure first.
None of this is new. All of it is easy to skip when the first script works. The script always works. It's the fourth rerun that teaches you.