Duplex & Transform Streams
Overview
If Readable Streams only allow reading, and Writable Streams only allow writing, what happens when you need to do both? A Duplex Stream implements both interfaces simultaneously (the classic example is a TCP Network Socket: you receive incoming messages and send outgoing messages on the exact same connection). A Transform Stream is a specialized Duplex stream designed to explicitly mutate data on the fly as it passes through the pipeline (e.g., Zipping/Compressing a file, or Encrypting a stream before saving it).
Syntax
// --- TRANSFORM STREAM EXAMPLE ---
const { Transform } = require('stream');
// Create a custom stream that forces all text to UPPERCASE on the fly!
const upperCaseTransform = new Transform({
// The magical transform method runs on every single incoming chunk!
transform(chunk, encoding, callback) {
// 1. Read the raw binary chunk and convert it to a string
const text = chunk.toString();
// 2. Mutate the data
const upperText = text.toUpperCase();
// 3. Push the mutated data back down the pipeline!
this.push(upperText);
// 4. Signal that this chunk is completely finished
callback();
}
});
// Using it in a pipeline:
// readStream -> upperCaseTransform -> writeStream
process.stdin.pipe(upperCaseTransform).pipe(process.stdout);Common Pitfalls
- Forgetting to call the
callback()in a Transform stream. The stream engine will permanently freeze on the very first chunk. The callback is the mandatory signal to libuv that the stream is ready to receive the next 64KB chunk of data. - Mutating the chunk directly in memory without realizing it's a shared Buffer. In high-performance streams, Node sometimes re-uses the exact same physical memory buffer to save allocation time. Always use
Buffer.from()ortoString()to clone the data if you are doing complex manipulation.
Interview Questions
Duplex stream and a specialized Transform stream?A generic Duplex stream (like a TCP socket) has entirely independent Read and Write pipelines; the data you receive is completely unrelated to the data you send. A Transform stream logically binds the two: the data you Write into it is mathematically mutated, and that mutated data is what you Read out of it.
Real-World Example
Compressing (Zipping) a massive file entirely on the fly using the native zlib Transform stream, saving 90% of disk space.
const fs = require('fs');
const zlib = require('zlib'); // Native Node C++ compression
const readStream = fs.createReadStream('massive_log.txt');
const writeStream = fs.createWriteStream('massive_log.txt.gz');
// Create a native GZIP Transform Stream
const gzipTransform = zlib.createGzip();
// Read -> Compress On The Fly -> Write to Disk!
readStream.pipe(gzipTransform).pipe(writeStream);Check Your Knowledge
Test your understanding of Duplex & Transform Streams with these quick questions.