Avalonia with Fragment Shaders

Introduction

Avalonia is a powerful cross-platform UI framework that, when combined with fragment shaders, opens up a world of possibilities for creating stunning visual effects in your applications. This comprehensive guide will explore how to use shaders in Avalonia, focusing on creating a custom control that can render and animate them. We'll start with the basics and build our way up to a fully functional ShaderAnimationControl.

Basic Concepts and Prerequisites

Before we dive into the implementation, let's review some key concepts:

• Shaders/Fragment Shaders: These are programs that compute the color of each pixel on the screen.

• Shader Languages: GLSL (OpenGL Shading Language) is commonly used for fragment shaders. In this guide, we'll focus on converting GLSL to SkSL (Skia Shading Language), which is understood by Skia, the graphics engine used by Avalonia.

• Uniforms: These are variables that remain constant for all pixels during a single draw call, often used for passing data from the application to the shader.

To follow this guide, ensure your project uses Avalonia 11.2.1 or newer. We'll assume you have basic knowledge of Avalonia and fragment shaders moving forward.

Converting Shadertoy Shaders to SkSL

Shadertoy is a popular platform for creating and sharing exciting and visually captivating fragment shaders. To use these shaders with Avalonia, we need to convert them to SkSL. The conversion process involves several key steps:

Uniform declarations:

• SkSL supports most of the Shadertoy uniforms directly.

• You may need to adjust the types of some uniforms (e.g., vec3 to float3) depending on your SkiaSharp version.

Main function signature:

• Change void mainImage(out vec4 fragColor, in vec2 fragCoord) to vec4 main(vec2 fragCoord)

• Return the fragColor at the end of the function instead of using an out parameter.

Adjust uniform and variable types:

• Change vec2, vec3, vec4 to float2, float3, float4 respectively.

• Replace int uniforms with float (e.g., uniform int iFrame becomes uniform float iFrame). Cast to int in the shader when needed: int currentFrame = int(iFrame);

• Other type adjustments may be necessary.

Update coordinate system:

• Shadertoy uses a bottom-left origin, while Avalonia and SkiaSharp use a top-left origin.

• Flip the y-coordinate at the beginning of the main function:

Normalize coordinates:

• Update how you calculate normalized coordinates:

Here's an example of the default Shadertoy shader and its SkSL conversion:

Original Shadertoy shader:

uniform vec3      iResolution;           // viewport resolution (in pixels)
uniform float     iTime;                 // shader playback time (in seconds)
uniform float     iTimeDelta;            // render time (in seconds)
uniform float     iFrameRate;            // shader frame rate
uniform int       iFrame;                // shader playback frame
uniform float     iChannelTime[4];       // channel playback time (in seconds)
uniform vec3      iChannelResolution[4]

Converted SkSL version:

Key differences and changes:

• The main function signature changed from void mainImage(out vec4 fragColor, in vec2 fragCoord) to float4 main(float2 fragCoord).

• We flip the y-coordinate at the beginning of the main function to account for the different coordinate systems.

• Uniform types are adjusted (e.g., vec3 to float3, vec4 to float4, int to float).

• Instead of setting the fragColor out parameter, we now return it at the end of the function.

• We use float2 and float3 instead of vec2 and vec3 in the shader code.

• We simplified the uniforms to the actual ones that are used by the shader code. We also adjusted iResolution from float3 to float2 to account for the fact that the control is only going to pass width and height values.

When using these shaders with the ShaderAnimationControl, you'll need to ensure that you're passing the correct uniforms from your C# code. The ShaderAnimationControl we're building will handle passing iResolution and iTime. If your shader uses other uniforms like iMouse, you'll need to extend the control to pass these values as well.

Important Notes

Remember that some more complex fragment shaders may require additional modifications and that this article is not exhaustive of all the modifications you need to make, especially if they use features not directly supported in SkSL.

Always test your converted shaders thoroughly and be prepared to make additional adjustments as needed.

Please be mindful of the licensing of any shader code while converting and using them from Shadertoy or other similar websites & sources to your project. Make sure to properly attribute the author and compensate them according to the licensing terms of the said shader code.

Understanding and Building the ShaderAnimationControl

The ShaderAnimationControl is a custom Avalonia control that allows for the rendering and animation of fragment shaders. We'll be building it step by step to understand its structure and functionality.

1. Class Definition and Private Structures

Let's define the ShaderAnimationControl class and its private structures as a starting point:

using Avalonia;
using Avalonia.Controls;
using Avalonia.Media;
using Avalonia.Platform;
using Avalonia.Rendering.Composition;
using Avalonia.Skia;
using SkiaSharp;
using System;
using System.Diagnostics;
using System.IO;
using System.Numerics;

public class ShaderAnimationControl : UserControl
{
    private record struct ShaderDrawPayload(
        HandlerCommand HandlerCommand,
        Uri? ShaderCode = default,
        Size? ShaderSize = default,
        Size? Size = default,
        Stretch? Stretch = default,
        StretchDirection? StretchDirection = default);

    private enum HandlerCommand
    {
        Start,
        Stop,
        Update,
        Dispose
    }

    private const float DefaultShaderLength = 512;
    
    private CompositionCustomVisual? _customVisual;

    // ... (rest of the code will be explained in subsequent sections)

This structure sets up the foundation for the ShaderAnimationControl. The ShaderDrawPayload record struct and HandlerCommand enum work together to facilitate communication between the main control and its rendering logic in _customVisual.

The use of private nested types (the record struct and enum) encapsulates these implementation details within the ShaderAnimationControl class.

2. Avalonia Properties

The ShaderAnimationControl class defines several Avalonia properties to allow customization of the shader's appearance and behavior.

public class ShaderAnimationControl : UserControl
{
    public static readonly StyledProperty<Stretch> StretchProperty =
        AvaloniaProperty.Register<ShaderAnimationControl, Stretch>(nameof(Stretch), Stretch.Uniform);

    public static readonly StyledProperty<StretchDirection> StretchDirectionProperty =
        AvaloniaProperty.Register<ShaderAnimationControl, StretchDirection>(
            nameof(StretchDirection),
            StretchDirection.Both);

    public static readonly StyledProperty<Uri?> ShaderUriProperty =
        AvaloniaProperty.Register<ShaderAnimationControl, Uri?>(nameof(ShaderUri));

    public static readonly StyledProperty<double> ShaderWidthProperty =
        AvaloniaProperty.Register<ShaderAnimationControl, double>(
            nameof(ShaderWidth), 
            DefaultShaderLength);

    public static readonly StyledProperty<double> ShaderHeightProperty =
        AvaloniaProperty.Register<ShaderAnimationControl, double>(
            nameof(ShaderHeight), 
            DefaultShaderLength);

    public Stretch Stretch
    {
        get => GetValue(StretchProperty);
        set => SetValue(StretchProperty, value);
    }

    public StretchDirection StretchDirection
    {
        get => GetValue(StretchDirectionProperty);
        set => SetValue(StretchDirectionProperty, value);
    }

    public Uri? ShaderUri
    {
        get => GetValue(ShaderUriProperty);
        set => SetValue(ShaderUriProperty, value);
    }

    public double ShaderWidth
    {
        get => GetValue(ShaderWidthProperty);
        set => SetValue(ShaderWidthProperty, value);
    }

    public double ShaderHeight
    {
        get => GetValue(ShaderHeightProperty);
        set => SetValue(ShaderHeightProperty, value);
    }

    static ShaderAnimationControl()
    {
        AffectsRender<ShaderAnimationControl>(ShaderUriProperty,
            StretchProperty,
            StretchDirectionProperty,
            ShaderWidthProperty,
            ShaderHeightProperty);

        AffectsMeasure<ShaderAnimationControl>(ShaderUriProperty,
            StretchProperty,
            StretchDirectionProperty,
            ShaderWidthProperty,
            ShaderHeightProperty

Let's break down each property and its purpose:

• StretchProperty: Determines how the shader content is stretched to fill the control's bounds.

• StretchDirectionProperty: Specifies in which directions the shader content can be stretched.

• ShaderUriProperty: Holds the URI of the shader code file to be loaded and rendered. In this control's case, it should be in avares:// format and the shader must be an Avalonia Resource. This is explained further in section Using the ShaderAnimationControl in Your Avalonia Project.

• ShaderWidthProperty and ShaderHeightProperty: Defines the width and height of the shader's rendering area. Defaults to 512 x 512 when not set.

The static constructor of the class uses AffectsRender and AffectsMeasure to indicate that changes to these properties should trigger re-rendering and re-measurement of the control. This ensures that the control updates appropriately when these properties change.

3. Layout and Measurement

The control overrides methods for measuring and arranging itself:

private Size GetShaderSize()
{
    return new Size(ShaderWidth, ShaderHeight);
}

protected override Size MeasureOverride(Size availableSize)
{
    var source = ShaderUri;
    var result = new Size();

    if (source != null)
    {
        result = Stretch.CalculateSize(availableSize, GetShaderSize(), StretchDirection);
    }

    return result;
}

protected override Size ArrangeOverride(Size finalSize)
{
    var source = ShaderUri;

    if (source == null) return new Size();

    var sourceSize = GetShaderSize();
    var result = Stretch.CalculateSize(finalSize, sourceSize);
    return result

These methods ensure that the control sizes itself correctly based on the shader size and stretch settings.

4. Visual Tree Attachment and Detachment

The ShaderAnimationControl overrides methods for attaching to and detaching from the visual tree. These methods are crucial for initializing the custom visual when the control is added to the UI and cleaning up resources when it's removed.

protected override void OnAttachedToVisualTree(VisualTreeAttachmentEventArgs e)
{
    base.OnAttachedToVisualTree(e);

    var elemVisual = ElementComposition.GetElementVisual(this);
    var compositor = elemVisual?.Compositor;
    if (compositor is null)
    {
        return;
    }

    _customVisual = compositor.CreateCustomVisual(new ShaderCompositionCustomVisualHandler());
    ElementComposition.SetElementChildVisual(this, _customVisual);

    LayoutUpdated += OnLayoutUpdated;

    _customVisual.Size = new Vector2((float)Bounds.Size.Width, (float)Bounds.Size.Height);

    _customVisual.SendHandlerMessage(
        new ShaderDrawPayload(
            HandlerCommand.Update,
            null,
            GetShaderSize(),
            Bounds.Size,
            Stretch,
            StretchDirection));

    Start();
}

protected override void OnDetachedFromVisualTree(VisualTreeAttachmentEventArgs e)
{
    base.OnDetachedFromVisualTree(e);
    LayoutUpdated -= OnLayoutUpdated;

    Stop();
    DisposeImpl();
}

private void OnLayoutUpdated(object? sender, EventArgs e)
{
    if (_customVisual == null)
    {
        return;
    }

    _customVisual.Size = new Vector2((float)Bounds.Size.Width, (float)Bounds.Size.Height);

    _customVisual.SendHandlerMessage(
        new ShaderDrawPayload(
            HandlerCommand.Update,
            null,
            GetShaderSize(),
            Bounds.Size,
            Stretch,
            StretchDirection

Let's break down some of the key parts of the OnAttachedToVisualTree method:

var elemVisual = ElementComposition.GetElementVisual(this);
var compositor = elemVisual?.Compositor;
if (compositor is null)
{
    return;
}

_customVisual = compositor.CreateCustomVisual(new ShaderCompositionCustomVisualHandler());
ElementComposition.SetElementChildVisual(this, _customVisual

This block of code is responsible for setting up the composition visual elements:

• It retrieves the ElementVisual for the control and its associated Compositor.

• It creates a new CustomVisual using ShaderCompositionCustomVisualHandler.

• The custom visual is then set as a child composition visual of the control.

This code is crucial for enabling high-performance rendering of the shader, as it allows the shader to be rendered independently of the main UI thread.

The control subscribes to the LayoutUpdated event to handle layout changes. Then the custom visual's size is set based on the control's bounds, and an initial update message is sent to the handler. The Start() method is then called to begin the shader animation.

The OnDetachedFromVisualTree method performs cleanup operations:

• It unsubscribes from the LayoutUpdated event.

• It stops the shader animation.

• It disposes of any resources held by the control.

These methods ensure that the ShaderAnimationControl properly initializes when added to the UI and cleans up when removed, preventing resource leaks and ensuring proper integration with Avalonia's visual tree management.

Lastly, the OnLayoutUpdated method sends the latest layout bounds to the custom visual handler.

5. Shader Control Methods

The class includes methods for starting, stopping, and disposing of the shader:

private void Start()
{
    _customVisual?.SendHandlerMessage(
        new ShaderDrawPayload(
            HandlerCommand.Start,
            ShaderUri,
            GetShaderSize(),
            Bounds.Size,
            Stretch,
            StretchDirection));
    
    InvalidateVisual();
}

private void Stop()
{
    _customVisual?.SendHandlerMessage(new ShaderDrawPayload(HandlerCommand.Stop));
}

private void DisposeImpl()
{
    _customVisual?.SendHandlerMessage(new ShaderDrawPayload(HandlerCommand.Dispose

These methods send appropriate messages to the custom visual handler to control the shader's lifecycle.

6. Custom Visual Handler

The ShaderCompositionCustomVisualHandler is a class defined inside ShaderAnimationControl that is responsible for managing the shader's lifecycle, handling animation frame updates, and performing the actual fragment shader rendering using SkiaSharp and Composition API's. Let's break it down further:

It is based around the CompositionCustomVisualHandler class and Avalonia's Composition API's that provides mechanisms to process custom visual rendering that is independent of the app's UI Thread.

This allows for better performance and visual responsiveness and also allows for some kind of visual feedback to the user if the UI Thread is blocked for some reason.

6.1. Class Definition and Fields

public class ShaderAnimationControl : UserControl
{
    // ... (rest of the prior code here)
    
    private class ShaderCompositionCustomVisualHandler : CompositionCustomVisualHandler
    {
        private bool _running;
        private Stretch? _stretch;
        private StretchDirection? _stretchDirection;
        private Size? _boundsSize;
        private Size? _shaderSize;
        private string? _shaderCode;
        private readonly object _sync = new();
        private SKRuntimeEffectUniforms? _uniforms;
        private SKRuntimeEffect? _effect;
        private bool _isDisposed;
        
        // ... (methods will be explained in subsequent sections)

Let's break down these fields:

• _running: A boolean flag indicating whether the shader animation is currently running.

• _stretch and _stretchDirection: These fields store the current stretch settings for the shader.

• _boundsSize: Represents the current size of the control's bounds.

• _shaderSize: Stores the size of the shader itself.

• _shaderCode: Holds the actual GLSL code of the shader.

• _sync: An object used for thread synchronization.

• _uniforms: An instance of SKRuntimeEffectUniforms used to pass uniform values to the shader.

• _effect: An instance of SKRuntimeEffect representing the compiled shader.

• _isDisposed: A flag indicating whether the handler has been disposed.

These fields are crucial for managing the state of the shader and ensuring proper rendering and lifecycle management. The use of nullable types (e.g., Stretch?) allows for cases where these values might not be set.

6.2. Message Handling

The OnMessage method is responsible for handling messages sent from the main control:

public override void OnMessage(object message)
{
    if (message is not ShaderDrawPayload msg)
    {
        return;
    }

    switch (msg)
    {
        case { HandlerCommand: HandlerCommand.Start, ShaderCode: { } uri, ShaderSize: { } shaderSize, Size: { } size, Stretch: { } st, StretchDirection: { } sd }:
        {
            using var stream = AssetLoader.Open(uri);
            using var txt = new StreamReader(stream);
            
            _shaderCode = txt.ReadToEnd();

            _effect = SKRuntimeEffect.Create(_shaderCode, out var errorText);
            if (_effect == null)
            {
                Console.WriteLine($"Shader compilation error: {errorText}");
            }

            _shaderSize = shaderSize;
            _running = true;
            _boundsSize = size;
            _stretch = st;
            _stretchDirection = sd;
            RegisterForNextAnimationFrameUpdate();
            break;
        }
        case { HandlerCommand: HandlerCommand.Update, ShaderSize: { } shaderSize, Size: { } size, Stretch: { } st, StretchDirection: { } sd }:
        {
            _shaderSize = shaderSize;
            _boundsSize = size;
            _stretch = st;
            _stretchDirection = sd;
            RegisterForNextAnimationFrameUpdate();
            break;
        }
        case { HandlerCommand: HandlerCommand.Stop }:
        {
            _running = false;
            break;
        }
        case { HandlerCommand: HandlerCommand.Dispose }:
        {
            DisposeImpl();
            break

This method handles different commands:

• Start: Loads and compiles the shader, sets up initial parameters, and starts the animation.

• Update: Updates the shader's size and stretch parameters.

• Stop: Stops the animation.

• Dispose: Cleans up resources.

6.3. Animation Frame Update

The OnAnimationFrameUpdate method is called for each animation frame:

public override void OnAnimationFrameUpdate()
{
    if (!_running || _isDisposed)
        return;

    Invalidate();
    RegisterForNextAnimationFrameUpdate

This method invalidates the visual (triggering a redraw) and registers for the next frame update, creating a continuous animation loop.

6.4. Shader Drawing

The Draw method is responsible for actually rendering the shader:

private void Draw(SKCanvas canvas)
{
    if (_isDisposed || _effect is null)
        return;

    canvas.Save();

    var targetWidth = (float)(_shaderSize?.Width ?? DefaultShaderLength);
    var targetHeight = (float)(_shaderSize?.Height ?? DefaultShaderLength);

    _uniforms ??= new SKRuntimeEffectUniforms(_effect);

    _uniforms["iTime"] = (float)CompositionNow.TotalSeconds;
    _uniforms["iResolution"] = new[] { targetWidth, targetHeight };

    using (var paint = new SKPaint())
    using (var shader = _effect.ToShader(false, _uniforms))
    {
        paint.Shader = shader;
        canvas.DrawRect(SKRect.Create(targetWidth, targetHeight), paint);
    }

    canvas.Restore

This method sets up the shader uniforms (including time for animation) and draws the shader onto the canvas. This is the place where you can add more uniforms to the shader as needed.

6.5. Rendering

The OnRender method is called when the control needs to be redrawn:

public override void OnRender(ImmediateDrawingContext context)
{
    lock (_sync)
    {
        if (_stretch is not { } st
            || _stretchDirection is not { } sd
            || _isDisposed)
        {
            return;
        }

        var leaseFeature = context.TryGetFeature<ISkiaSharpApiLeaseFeature>();
        if (leaseFeature is null)
        {
            return;
        }

        var rb = GetRenderBounds();
        var size = _boundsSize ?? rb.Size;
        var viewPort = new Rect(rb.Size);
        var sourceSize = _shaderSize!.Value;

        if (sourceSize.Width <= 0 || sourceSize.Height <= 0)
        {
            return;
        }

        var scale = st.CalculateScaling(rb.Size, sourceSize, sd);
        var scaledSize = sourceSize * scale;
        var destRect = viewPort
            .CenterRect(new Rect(scaledSize))
            .Intersect(viewPort);
        var sourceRect = new Rect(sourceSize)
            .CenterRect(new Rect(destRect.Size / scale));

        var bounds = SKRect.Create(new SKPoint(), new SKSize((float)size.Width, (float)size.Height));

        var scaleMatrix = Matrix.CreateScale(
            destRect.Width / sourceRect.Width,
            destRect.Height / sourceRect.Height);

        var translateMatrix = Matrix.CreateTranslation(
            -sourceRect.X + destRect.X - bounds.Top,
            -sourceRect.Y + destRect.Y - bounds.Left);

        using (context.PushClip(destRect))
        using (context.PushPostTransform(translateMatrix * scaleMatrix))
        {
            using var lease = leaseFeature.Lease();
            var canvas = lease.SkCanvas;
            Draw(canvas

This method handles:

• Acquiring the SkiaSharp API lease and the underlying SkCanvas from Avalonia's ImmediateDrawingContext.

• Calculating the correct scale and position for the shader within the control.

• Applying clipping to ensure the shader only renders within its designated area.

• Applying transformations to scale and position the shader correctly.

• Calling the Draw method to render the shader on the canvas.

6.6. Disposal

private void DisposeImpl()
{
    lock (_sync)
    {
        if (_isDisposed) return;
        _isDisposed = true;
        _effect?.Dispose();
        _uniforms?.Reset();
        _running = false

This ensures that the SkiaSharp assets in the custom visual handler are properly disposed of, that the handler stops playback, and avoids memory leaks.

Using the ShaderAnimationControl in Your Avalonia Project

Now that we've built our ShaderAnimationControl, let's see how you can use it in your Avalonia project.

Add the control to your XAML:

<Window xmlns:controls="using:YourNamespace.Controls">
    <!-- other content -->
    <controls:ShaderAnimationControl x:Name="ShaderHost"
                                     ShaderUri="avares://YourProject/Assets/your_shader.sksl"/>
</Window>

Ensure your shader file is included in your project:

• Add your SkSL shader file to your project's assets folder. For example, if the folder is in (Project Root Folder)/Assets then make sure that the project file has the following entry:

  <AvaloniaResource Include="Assets\**" />

• Alternatively, you can set the Build Action to "Avalonia Resource" in the shader file's properties in your IDE.

Customize the control:

• Adjust the Stretch, StretchDirection, ShaderWidth, and ShaderHeight properties as needed.

Handle playback events:

• If necessary, you can start and stop the shader animation in your code-behind:

protected override void OnLoaded(RoutedEventArgs e)
{
    base.OnLoaded(e);
    ShaderHost.Start();
}

protected override void OnUnloaded(RoutedEventArgs e)
{
    base.OnUnloaded(e);
    ShaderHost.Stop

Conclusion

With the ShaderAnimationControl, you can now easily incorporate stunning shader effects into your Avalonia app. This control provides a flexible way to load, render, and animate fragment shaders, opening up a world of creative possibilities for your app's visuals.

Remember to optimize your shaders for performance, especially on mobile devices or when using complex effects. The ShaderAnimationControl provides a solid foundation for shader integration, but you may need to extend it further for more complex use cases, such as passing additional uniforms or handling user interactions.

As you experiment with different shaders and effects, keep in mind the balance between visual appeal and performance. Complex shaders can be resource-intensive, so always test your application on various devices to ensure a smooth user experience.

Happy coding, and enjoy creating captivating visual effects with Avalonia and fragment shaders!

Special thanks to Wiesław Šoltés for giving us the permission to use his EffectsDemo project as an inspiration to the article's ShaderAnimationControl code.