Video Playback

Images are still frames. Shaders are generated frames. But live performance and installations need video — pre-recorded content, loops, backgrounds, media playback. The texture pipeline supports all of it, but it needed a node that turns a video file into a stream of textures.

VideoPlayer is that node.

The node

• Inputs: path (String), rate (Number, 1.0), loop (Bool, true), seek (Number, -1.0 = disabled)
• Outputs: texture (TextureHandle), duration (Number, seconds), position (Number, seconds)

Point it at a video file. Get a texture that updates every frame at the video’s native rate. The output plugs directly into any downstream node — filters, composites, feedback loops, custom shaders.

Rate controls playback speed. 1.0 is normal, 2.0 is double speed, 0.5 is slow-mo, -1.0 plays backwards. The node tracks position internally: position += dt * rate each frame, where dt is the frame time delta.

Loop wraps position around when it hits the end: position %= duration. Turn it off and playback stops at the last frame.

Seek overrides position when set to >= 0. Wire a slider to it for scrubbing. Set it to -1 to return to normal playback. This is the “jump to time” control — useful for performance cue triggers or interactive installations where user input drives video position.

FFmpeg under the hood

Video decoding is done by FFmpeg via the ffmpeg-next Rust bindings. This gives Lux support for essentially every video format in existence — MP4, MOV, AVI, WebM, MKV, and whatever else FFmpeg can demux and decode.

The VideoDecoder wrapper handles:

1. Opening — finds the best video stream, extracts metadata (width, height, duration, FPS, time base).
2. Decoding — reads packets sequentially, decodes frames on demand. Uses FFmpeg’s SwsContext to convert any pixel format to RGBA8.
3. Seeking — flush-based seeking via input_ctx.seek(). Flushes the decoder, seeks the demuxer, and decodes forward to the target frame.
4. Caching — stores the last decoded RGBA frame. If the position hasn’t advanced past the current frame’s timestamp, returns None (no new data), and the node skips the texture upload.

The frame cache is important for performance. At 60fps render with 30fps video, half the frames will be skipped — the node checks, sees the video hasn’t advanced, and does zero work. No redundant decodes, no redundant uploads.

Feature gating

Here’s the pragmatic decision: FFmpeg is a heavy system dependency. It requires shared libraries (libavcodec, libavformat, libswscale, libavutil) installed on the build machine. CI environments don’t have them. Not every user needs video.

So video decoding is behind a Cargo feature flag:

[features]
default = []
video = ["ffmpeg-next"]

When the video feature is disabled (the default), the VideoPlayer node still exists — it registers, it appears in the node browser, it has all the right pins. But it outputs TextureHandle::INVALID and logs a warning: “Video support not enabled. Build with –features video.”

This keeps the build clean for development and CI while letting users opt in to video when they have FFmpeg installed. The node is never invisible — you always know it’s there and what it needs.

# Build with video support
cd app && cargo build --features video

# Run with video
cargo run --bin lux --features video

The decoder details

Sequential decoding is a deliberate choice. Video codecs use temporal compression — P-frames and B-frames reference other frames. Random access requires seeking to the nearest keyframe and decoding forward. For smooth playback, sequential decode is dramatically cheaper than random access.

When the user does seek (the seek input goes >= 0), the decoder flushes its internal state, seeks the demuxer to the nearest keyframe before the target timestamp, and then decodes forward until it reaches the desired frame. It’s not instant — seeking to the middle of a long GOP can require decoding dozens of frames — but it’s correct.

The time base conversion is handled carefully. FFmpeg uses stream-specific time bases (e.g., 1/12800 for some codecs). The decoder converts all timestamps to seconds on output, and converts seek positions from seconds back to stream time base on input.

Error handling is comprehensive — Ffmpeg, NoVideoStream, and SwsInit error types cover the failure modes. All errors are logged and the node degrades gracefully to an invalid texture output.

What this enables

VideoPlayer completes the source node lineup:

Every kind of visual content that a creative coder or live performer might need is now available as a TextureHandle. Load a video, run it through ChromaKey to remove the green screen, blend it with a live NoiseTexture, apply Bloom, and output to a projector.

VideoPlayer completes Phase 7 — the full texture processing pipeline. From the GPU foundation through sources, filters, layer bridging, feedback, custom shaders, extended processing, and now video — that’s 34 nodes, 28 WGSL shaders, and roughly 7,500 lines of code. Lux went from a vector graphics tool to a GPU image processing engine in eight sessions.

One node, 747 lines, and Lux can play video.