Custom effects

A custom effect has two halves. An EffectSpec is data that describes it, shared across platforms. An EffectResolver turns that data into something the current platform can render.

The spec

data class SoftFocus(
  val radius: Float,
) : EffectSpec {
  override val id: String get() = "acme.softfocus"

  override val stage: EffectStage get() = EffectStage.Spatial
}

fun EffectsBuilder.softFocus(radius: Float): EffectsBuilder = add(SoftFocus(radius))

id has to be stable. Namespace it like a package, so two libraries can’t collide, and never derive it from the class name.
stage defaults to EffectStage.Color. Your effect lands at the end of whichever stage it names.
scope defaults to EffectScope.Everywhere. An effect whose result depends on where a frame sits in its run overrides it with ClipOnly, and a plan refuses it anywhere else.
requiresSdr defaults to false. An effect that needs SDR frames says true, and an HDR source tone maps first. HdrMode.Auto reports that as a concession naming your effect, and HdrMode.KeepHdr is refused by name.
isInert defaults to false. Say true when the spec as configured leaves every pixel alone, and the plan leaves the effect out and lists it on its inert notices, so no backend is handed it.
beneathOverlays defaults to false. Say true on an EffectStage.Composite effect that runs over the fill and under every overlay at its own scope, like vhs. Outside that stage it changes nothing.

A spec has to compare by value, which a data class gives you. filmstrip compares specs to decide if a composition changed and if a cached thumbnail is still good, so one left on identity equality rebuilds the graph and misses the cache every time.

Use your extension like any other effect.

val composition =
  compositionOf {
    clip(MediaSource.of("/path/to/clip.mp4")) {
      effects { softFocus(radius = 0.01f) }
    }
  }

The resolver

A resolver answers one spec at a time. Return null to decline it, and the next resolver gets a look.

class SoftFocusResolver : EffectResolver {
  override fun resolve(
    spec: EffectSpec,
    capabilities: RenderCapabilities,
    attributes: Attributes,
  ): EffectResolution? {
    if (spec !is SoftFocus) return null

    return softFocusOn(spec, capabilities, attributes)
  }
}

A non-null answer claims the spec, which ends the chain.

EffectResolution.Resolved renders it as declared.
EffectResolution.Degraded renders it with named causes, which plan reports before any work starts.
EffectResolution.Unsupported refuses it by name, and no later resolver can substitute another rendering.
EffectResolution.Failed refuses the export with its error when a read the effect needs failed, like a decode that ran out of time. A later call may succeed, so withoutUnsupported never leaves the effect out.

Each platform’s answer is a PlatformEffect, in that platform’s own terms. Gate on capabilities.api and decline what you didn’t write for. attributes holds what the pipeline already worked out, so multiply a fraction by inputSize to get pixels, the frame size entering this effect. Return a fresh PlatformEffect each time, since each plan holds its own.

The resolver’s softFocusOn is an expect function you declare yourself, whose actual on each platform calls that platform’s function below.

A media3 Effect that runs as its own pass, or a function that builds a TextureOverlay. It runs once for every graph that draws the effect, so answer a new overlay each time, since an overlay holds the texture of the graph it drew in.

fun softFocusOnAndroid(
  spec: SoftFocus,
  capabilities: RenderCapabilities,
  attributes: Attributes,
): EffectResolution? {
  if (capabilities.api != RenderApi.OpenGlEs) return null

  val sigma = spec.radius * attributes.inputSize.height
  return EffectResolution.Resolved(PlatformEffect(GaussianBlur(sigma)))
}

A step in a Core Image chain, or a single CIFilter. A step is told about the frame, and everything scale-dependent has to be read from there.

fun softFocusOnIos(
  spec: SoftFocus,
  capabilities: RenderCapabilities,
  attributes: Attributes,
): EffectResolution? {
  if (capabilities.api != RenderApi.Metal && capabilities.api != RenderApi.CoreImage) return null

  val radius = spec.radius * attributes.inputSize.height
  val step =
    CoreImageEffect { image, _ ->
      val blur = CIFilter.filterWithName("CIGaussianBlur") ?: return@CoreImageEffect image
      blur.setValue(image, forKey = "inputImage")
      blur.setValue(radius.toDouble(), forKey = "inputRadius")
      blur.outputImage ?: image
    }
  return EffectResolution.Resolved(PlatformEffect(step))
}

A fragment of an ffmpeg filter graph. It’s plain data, so a test can assert on it with no toolchain installed.

fun softFocusOnJvm(
  spec: SoftFocus,
  capabilities: RenderCapabilities,
  attributes: Attributes,
): EffectResolution? {
  if (capabilities.api != RenderApi.FilterGraph) return null

  val sigma = spec.radius * attributes.inputSize.height
  val blur = FilterNode("gblur", "sigma" to sigma.toString())
  return EffectResolution.Resolved(PlatformEffect(FilterFragment(chain = listOf(blur))))
}

A WebGlPass that brings its own fragment source through WebGlShader, or a refusal by name, which is what this one does.

fun softFocusOnWeb(
  spec: SoftFocus,
  capabilities: RenderCapabilities,
  attributes: Attributes,
): EffectResolution? {
  if (capabilities.api != RenderApi.WebGl) return null

  return EffectResolution.Unsupported(spec.id, "The browser has no pass that can draw a blur.")
}

An effect that has to do async work first, like decoding an image in a browser, overrides prepare, which runs before each negotiation of a composition.

Register it

Resolvers are asked newest registration first, so add yours after the backends.

val filmstrip =
  Filmstrip {
    allBackends()
    addEffectResolver(SoftFocusResolver())
  }

A spec no resolver claims is refused by name at plan time.