Why 2.5 MB Mongo Document Became a 5.3 MB Kafka Message in Debezium

5 min read

Here is what we were running when we hit this:

Debezium 2.7.4.Final
Debezium MongoDB connector, with the MongoEventRouter outbox SMT (still marked incubating)
Kafka 3.7.0
producer.override.max.request.size set to 4 MB (4194304 bytes)
transforms.outbox.collection.expand.json.payload set to false

The problem

We stream a MongoDB outbox collection to Kafka through the Debezium MongoDB connector. One document was about 2.5 MB in Mongo. When Debezium tried to publish it, the task failed:

org.apache.kafka.common.errors.RecordTooLargeException: The message is 5323164
bytes when serialized which is larger than 4194304, which is the value of the
max.request.size configuration.

Two things did not add up. First, the document is around 2.5 MB in Mongo, but Debezium reports 5,323,164 bytes, more than double. Second, when we published the same document to Kafka directly with a plain producer (same 4 MB limit), it went through without any trouble.

Digging in

The exported document, minified, is about 2.75 MB of JSON. That lines up with the 2.5 MB of BSON stored in Mongo, so the 4 MB limit is plenty for the real data. Something in Debezium’s path was inflating it.

The shape of the document was the first hint. It is not big because of one large field. It is big because it is deeply nested, with roughly 97,900 fields, almost all of them small strings. Keep that field count in mind.

The cause: the outbox router pretty-prints the payload

We had expand.json.payload set to false. With that setting, when the payload field is a nested document, MongoEventRouter turns it into a string using these writer settings:

// MongoEventRouter.java
private final JsonWriterSettings jsonWriterSettings = JsonWriterSettings.builder()
        .outputMode(JsonMode.EXTENDED)   // verbose, canonical extended JSON
        .indent(true)                    // pretty-print
        .newLineCharacters("\n")         // newline per field
        .build();

afterStruct.put(fieldPayload, entry.getValue().asDocument().toJson(jsonWriterSettings));

The part that hurts is indent(true) together with newLineCharacters("\n"). This pretty-prints the payload, so every field gets a newline plus some indentation spaces. With about 97,900 fields, that whitespace, nearly all of the growth, inflates the payload by 1.93 times, from roughly 2.75 MB to about 5.32 MB.

The numbers match closely. Running the same serialization over the document gives 5,318,367 bytes, against the 5,323,164 bytes Debezium reported. That is a difference of about 5 KB.

Extended JSON adds a bit more

JsonMode.EXTENDED is MongoDB’s canonical Extended JSON. It always wraps typed BSON values in explicit envelopes so the type is never lost. A 32-bit int 5 becomes {"$numberInt": "5"}, a long becomes {"$numberLong": "5"}, a date becomes {"$date": {"$numberLong": "1784244711736"}}, and binary data becomes {"$binary": {"base64": "...", "subType": "00"}}.

For our document this added only about 5 KB, since it had just 61 numeric values and one date. But for a document full of numbers, dates, or binary fields, this wrapping can grow the payload a lot on its own.

This only happens on the expand.json.payload=false path. The connector’s normal document serialization uses empty indent and newline characters, so no pretty-printing there.

Can you configure the JSON writer, or is it fixed in a newer version?

We checked. These writer settings are hardcoded in MongoEventRouter, and there is no config field for the output mode, indentation, or pretty-printing. The only related option, collection.expand.json.payload, just switches between the two code paths and does not affect formatting.

We confirmed this against the latest stable release, v3.6.0.Final, as well as the current main branch: both still hardcode EXTENDED mode with indent(true). So there is no version, from 2.7.4 through the latest 3.x, where you can turn off the indentation or switch to RELAXED mode through config. Changing it would need a patch to the SMT or a feature request upstream. At the time of writing we did not find an existing Debezium issue tracking this, so filing one would be a good next step.

Why the direct publish worked and Debezium did not

It was never really the same message. The bytes Kafka measures are the output after Debezium transforms it, not the original document.

When you publish directly, you send the compact 2.5 MB document, which stays under 4 MB and gets accepted. When Debezium publishes, the SMT re-serializes the payload with pretty-printing and canonical Extended JSON, pushing it to about 5.32 MB, which is over 4 MB and fails.

One more thing worth knowing: max.request.size is a producer-side limit on the whole serialized record, so value plus key plus headers plus overhead, and it is checked before compression. So the raw, inflated size is what trips it.

Fixing it

A few options, from quickest to cleanest:

Raise the size limits. Increase producer.override.max.request.size, and make sure the broker message.max.bytes and the topic max.message.bytes are at least as large. (Consumer settings like max.partition.fetch.bytes are soft limits since Kafka 0.10.1 / KIP-74, so consumers can still read oversized records; raising them just helps throughput.) This is the fastest unblock, but it only buys headroom for genuinely large documents.
Add producer compression alongside option 1, for example producer.override.compression.type=zstd. On its own it will not fix this error: max.request.size is checked on the uncompressed size, so a 5.32 MB record fails against a 4 MB limit no matter how well it compresses. What compression does buy you is on the broker side, where message.max.bytes is checked against the compressed batch. Repetitive JSON whitespace compresses very well, so you can raise max.request.size without raising the broker and topic limits nearly as much.
Set expand.json.payload=true. This takes the other code path, where the payload is expanded into a proper Connect Struct instead of a pretty-printed Extended-JSON string, which avoids both the indentation and the type wrappers. It does change the value schema your consumers see, so test it against them first.
Patch the SMT if you build a custom connector: drop indent(true) and the newlines, and optionally switch to RELAXED mode.

Wrapping Up

A “message too large” error from Debezium is not always about your data being too large. The connector and its SMTs can reshape the payload before it reaches the producer. In our case, pretty-print indentation in the MongoDB outbox router nearly doubled a 2.75 MB payload to 5.32 MB, enough to cross a 4 MB limit that the original data never got close to.

Note: Links point to the versions we used. It is worth confirming them against the docs for your own version before relying on them.

Thanks for reading. Have a great day!