Parquet Data Processing Pipeline
Original Column Data
Example:
"Apple"
"Apple"
"Banana"
Encoding Phase
Dictionary: {0: "Apple", 1: "Banana"}
Encoded: [0, 0, 1]
+ Page Statistics
Compressed
Binary Data
Reduced
Size
Encoding Schemes
Dictionary
Maps repeated values
to integer keys
Best for low cardinality
RLE/Bit-Packing
Run Length Encoding +
Efficient bit storage
For repeated values
Delta
Stores differences
between values
For sorted numbers
Plain
Direct value storage
No special encoding
Fallback option
Compression Algorithms
Speed Optimized
Snappy | LZ4
Balanced
ZSTD | LZO
Compression Optimized
Gzip | Brotli