Processing Pipelines

A processing pipeline decides what happens to a source as it is added: validate it, transform it, leave it untouched, or fan it out into variants. This guide covers building pipelines, routing by file type so a single upload endpoint can accept images and videos, and the property switches that control buffering and change detection.

The building blocks

A FileProcessingPipeline is an ordered list of FileProcessor steps. Each processor receives a FileProcessingContext and returns a FileProcessingResult. A pipeline both provides itself (IFileProcessingPipelineProvider) and selects itself (IFileProcessingPipelineSelector), so a single pipeline can be passed straight to AddAsync without wrapping it.

The Empty Pipeline

Empty stores the source as-is with no processing. Use it when you only need durable, transactional storage and the content has already been validated or comes from a trusted source.

// Storing a generated PDF report - already known to be a valid PDF.
var added = await txn.AddAsync(reportStream, ".pdf", leaveOpen: true, FileProcessingPipeline.Empty);
Caution

Do not use Empty for untrusted uploads. Anything that originates from a user (or an external API) should go through a pipeline with at least a FileFormatValidationProcessor so the bytes are confirmed to match the declared extension. See Validating File Formats.

Building a Pipeline

Construct a pipeline from one or more processors. Processors run in order, each receiving the previous step's output, which is what lets you compose two transformations into a single step. A common case is producing a compact JPEG thumbnail of a video: VideoThumbnailProcessor extracts a full-resolution PNG poster frame, then ImageProcessor resizes that frame and converts it to JPEG.

// Extract a poster frame, then resize/convert it to a 256x256 JPEG thumbnail.
var pipeline = new FileProcessingPipeline(
    new VideoThumbnailProcessor(VideoThumbnailProcessingOptions.Standard),
    new ImageProcessor(new ImageProcessingOptions {
        Formats = [new ImageFormatMapping(ImageFormat.Png, ImageFormat.Jpeg)],
        Resize = new ImageResizeOptions(ImageResizeMode.FitDown, 256, 256),
    }));

A single processor can also produce a pipeline directly with ToPipeline, which is convenient when there is exactly one step:

var pipeline = new ImageProcessor(ImageProcessingOptions.Preserve).ToPipeline();

Routing by File Type

When different extensions need different handling, build a FileProcessingPipelineSelector from several pipeline providers. It routes each source to the first provider whose leading processor declares the file's extension in AllowedFileExtensions. At most one provider may act as a catch-all default.

This is the natural shape for a media upload endpoint that accepts both images and videos:

var selector = new FileProcessingPipelineSelector(
    new ImageProcessor(imageOptions),
    new VideoProcessor(VideoProcessingOptions.StandardizedH264AACMP4));

// The selector picks the image pipeline for .jpg/.png and the video pipeline for .mp4/.mov.
var added = await txn.AddAsync(source, leaveOpen: true, selector);

If no provider matches the extension and there is no default, GetPipeline throws FileProcessingException, which the upload handler can translate into a "file type not supported" error. See Exception Handling.

Writing a Custom Processor

Derive from FileProcessor, declare the extensions it accepts in AllowedFileExtensions, and implement the protected ProcessAsync method. Return a FileProcessingResult describing the output:

  • File when the output is a file on disk. Use this when the processor invokes an external tool that writes to a temp file.
  • Stream when the output is a stream, optionally with a new extension. Use this when the processor produces its result in memory.

Each result carries a hasChanges flag. Reporting false means the step did not alter the source, which feeds the change-detection switches below (alias-when-unchanged for variants, throw-when-unchanged for the main file).

The FileProcessingContext exposes FileId, VariantId, Extension, IsLastProcessStep, and a CancellationToken.

Tip

Custom processors are usually unnecessary because the built-in image and video processors cover most needs. Reach for one when integrating an external tool (a PDF flattener, an OCR step that embeds extracted text into the PDF, a virus scanner that fails the add on detection, etc.) that has to participate in the same transaction as the storage step.

Buffering and Change Detection

A few pipeline properties tune behavior:

Source buffering

SourceBufferingMode controls whether the source is buffered before processing. The default is automatic, which buffers when a processor needs random access or rewinds. Override only if you know the upload stream is already seekable and want to skip the extra copy, or you want to force buffering for a forward-only source that an early processor needs to re-read.

Aliasing unchanged variants

On a variant pipeline, set AliasWhenVariantSourceUnchanged to true so that when processing reports no changes, an alias to the source is stored instead of a duplicate copy. This is what makes a "256x256 thumbnail" pipeline storage-efficient when the source image is already 256x256 or smaller. This property applies only to variant pipelines. See Variant Aliasing.

Rejecting unchanged main sources

On a main-file pipeline, set ThrowWhenMainSourceUnchanged to true to throw FileSourceUnchangedException when processing leaves the main source unchanged. This is useful when an unchanged result indicates a caller error, for example a re-encode endpoint where seeing the same bytes out as in means the upload was already in the target format and the call should be rejected as a no-op rather than silently succeeding.

Next Steps