SOP: Video Frame Processing with WebGPU + WebCodecs

Fresh 🌱

Version: 1.0 | Chrome: 113+

Overview

WebGPU integrates natively with the WebCodecs API for GPU video processing. Unlike JavaScript/Canvas approaches, WebGPU processes video frames directly on the GPU — no CPU round-trips, minimal latency, full parallelism.

Flowchart

Step-by-Step

Step 1 — Access Camera Stream

const constraints = { video: { frameRate: 60 } };
const stream = await navigator.mediaDevices.getUserMedia(constraints);

const video = document.createElement('video');
video.srcObject = stream;
await video.play();

Step 2 — Initialize WebGPU for Video

const adapter = await navigator.gpu.requestAdapter();
const device = await adapter.requestDevice();

const format = navigator.gpu.getPreferredCanvasFormat();
const context = canvas.getContext('webgpu');
context.configure({ device, format });

Step 3 — WGSL Shader for Video Processing

// Grayscale filter: left half grayscale, right half color
@vertex
fn vertexMain(@builtin(vertex_index) i : u32) -> @builtin(position) vec4f {
  const quadPos = array(
    vec2f(-1,  1), vec2f(-1, -1),
    vec2f( 1,  1), vec2f( 1, -1)
  );
  return vec4f(quadPos[i], 0, 1);
}

@group(0) @binding(0) var videoTexture: texture_external;

@fragment
fn fragmentMain(@builtin(position) position : vec4f) -> @location(0) vec4f {
  let texCoord = vec2u(position.xy);
  let color = textureLoad(videoTexture, texCoord);

  // Right half: color | Left half: grayscale
  if (position.x > f32(textureDimensions(videoTexture).x / 2)) {
    return color;
  }
  let gray = dot(color.xyz, vec3f(0.299, 0.587, 0.114));
  return vec4f(gray, gray, gray, 1.0);
}

Step 4 — Create Render Pipeline

const shaderModule = device.createShaderModule({ code: wgslCode });

const pipeline = device.createRenderPipeline({
  layout: 'auto',
  vertex: { module: shaderModule, entryPoint: 'vertexMain' },
  fragment: {
    module: shaderModule,
    entryPoint: 'fragmentMain',
    targets: [{ format }]
  },
  primitive: { topology: 'triangle-strip' }
});

Step 5 — Per-Frame Processing Loop

function applyFilter(videoFrame) {
  // Import video frame as external GPU texture
  const texture = device.importExternalTexture({ source: videoFrame });

  // Create bind group with this frame's texture
  const bindGroup = device.createBindGroup({
    layout: pipeline.getBindGroupLayout(0),
    entries: [{ binding: 0, resource: texture }]
  });

  const commandEncoder = device.createCommandEncoder();
  const renderPass = commandEncoder.beginRenderPass({
    colorAttachments: [{
      view: context.getCurrentTexture().createView(),
      loadOp: 'clear',
      storeOp: 'store'
    }]
  });

  renderPass.setPipeline(pipeline);
  renderPass.setBindGroup(0, bindGroup);
  renderPass.draw(4); // Full-screen quad
  renderPass.end();

  device.queue.submit([commandEncoder.finish()]);

  // CRITICAL: always close the frame when done
  videoFrame.close();
}

// Animation loop
(function render() {
  const videoFrame = new VideoFrame(video);
  applyFilter(videoFrame);
  requestAnimationFrame(render);
})();

Zero-Copy Check (Developer Feature)

const externalTexture = device.importExternalTexture({ source: video });

if (externalTexture.isZeroCopy) {
  console.log('Optimal: Video frame accessed directly by GPU (no copy)');
} else {
  console.log('Suboptimal: Intermediate copy created');
}

Zero-Copy Requirement

Zero-copy requires enabling the WebGPU Developer Features flag at chrome://flags/#enable-webgpu-developer-features.

Performance Tips

1. Always call videoFrame.close() — failure to close causes memory leaks
2. Create bind group per frame — external textures are single-use
3. Use triangle-strip topology — more efficient for full-screen quads (4 vertices vs 6)
4. Match canvas size to video — prevents scaling overhead

Verification Checklist

• [ ] Video is playing before first VideoFrame creation
• [ ] videoFrame.close() called after every frame
• [ ] Canvas configured with device and correct format
• [ ] External texture binding is texture_external in WGSL (not texture_2d)
• [ ] New bind group created every frame (external textures expire)

See Also

• Create Render Pipeline
• Developer Features — Zero-Copy Check
• From WebGL to WebGPU — Video Processing section