Protobuf Decoder

Reading protobuf without the schema

Protocol Buffers serialize structured data into a compact binary form. Normally you need the .proto definition to interpret it, but the wire format is self-describing enough to decode partially without one. Each field is encoded as a tag (a field number plus a wire type) followed by its value, so a raw decoder can walk the bytes and reconstruct the shape even if it cannot recover the original field names.

How the wire format is laid out

Every field starts with a varint key. The low 3 bits are the wire type (0 varint, 1 fixed64, 2 length-delimited, 5 fixed32) and the rest is the field number. Length-delimited fields (type 2) hold strings, bytes, or nested messages, which is why a raw decoder often cannot tell a string from a sub-message without guessing.

Example

bytes:  08 96 01 12 03 61 62 63

08    -> field 1, wire type 0 (varint)
96 01 -> value 150
12    -> field 2, wire type 2 (length-delimited)
03    -> length 3
61 62 63 -> "abc"

Field 1 is the integer 150 and field 2 is the string abc. Without the schema you learn the numbers and types but not that field 1 was named, say, user_id.

The ambiguities to expect

A varint could be an int32, int64, bool, or an enum; the decoder cannot know which. Signed integers using zigzag encoding look like large unsigned values unless you decode them as sint. A length-delimited blob might be a valid nested message or just raw bytes that happen to parse, so treat inferred nesting as a best guess.

FAQ

Why do my negative numbers look huge? Plain int32/int64 encode negatives as very large varints. Fields declared as sint use zigzag encoding; decoding with the wrong interpretation produces the mismatch.

Can it recover field names? No. Names live only in the .proto file and are never present on the wire. You get field numbers and types.