Frame

The Frame Subsystem

The Nebula Frame system provides a scriptable interface for defining the passes and resources in a frame. Where most engines define their passes, draw batches, computes, copies and such, the frame system uses a JSON-script which is loaded at runtime, and which can be modified and customized for each game, without changing a single line of C++ code. The point is to have a modular and interpretable rendering routine, which allows the engine to properly syncrhonize resources automatically, define resizing behavior, and to quickly add, remove and modify sections of the frame without rewriting huge swats of code or to have to think about unforseen side effects. As such, the frame script provides a JSON-based language which we will go through in this document.

Declarations

The Nebula frame script system allows rendering to be scripted, rather than it being hard-coded. Frame scripts allows us to create resources needed for rendering, them being:

• Textures, either used for rendering, or shader writing, or for sampling.
• ReadWrite Buffers, used for non-texture type data reads and writes.
• Plugins, code hooks which allows explicit code to be run at some point during the frame.

Textures are declared like so:

textures: [
    {...}
]

And contains a series of texture objects. A texture object has a certain set of fields, these are:

• name - Name.
  
• format - Pixel format, available options can be found in CoreGrapics::PixelFormat.
  
• usage - List of flags, combined with |, allowed values are: Render, ReadWrite, Copy, Immutable and Mapable found in CoreGraphics::TextureUsage.
  
• relative - Bool value, true means the texture will scale with the window the script is attached to, false means it wont.
  
• width, height, depth - Texture dimensions, percent of screen size if relative is true, otherwise absolute size in pixels.
  
• mips, layers - Specify how many mips a texture has, and how many layers. Layers are only relevant if the texture is of an array type.
  
• type - Textures can be different types, Texture1D, Texture1DArray, Texture2D, Texture2DArray, TextureCube, TextureCubeArray and Texture3D CoreGraphics::TextureType.
  An example texture declaration would look like this:

{
  name: "AlbedoBuffer",
  format: "R8G8B8A8",
  usage: "Render",
  relative: true,
  width: 1.0,
  height: 1.0,
  type: "Texture2D"
}

Read write buffers can also be created in the framescript, although there is usually very little reason to do so, since they don't necessarily map very well to frame operations. However, if wanted, one can do it like so:

read_write_buffers: [
    {...}
]

And each buffer will have the following fields:

• name - Name of buffer.
  
• size - Size of buffer in bytes, absolute size if relative is false, otherwise the size will be a % of the screen size.
  
• relative - True if buffer should scale with the size of the screen.
  An example can look like this:

{
    name: "ClusterBuffer",
    size: "4096",
    relative: true
}

This will make the buffer 4096 * screen width * screen height in size.

Operations

The operations allowed in the 'global' scope of a frame script are:

• blit copies a texture to another with a filtering and/or image conversion method, such as like RGB -> sRGB, BGR->RGB.
• copy performs a flat copy without any filtering or conversion.
• mipmap generates a mip chain on a texture, be sure to specify the 'mips' field with the value 'auto' first.
• compute runs a compute kernel using a non-optional shader state.
• plugin performs an algorithm outside of a render pass.
• pass triggers a rendering pass, within which only subpasses and subpass-related operations are allowed. In a pass, we are inside a 'render' scope, wherein only and exclusively render related commands are allowed.
• barrier puts an explicit synchronization point in the script, which has some rare uses, for example when synchronizing a texture before presenting, as the present system knows nothing of the layout changes from the framescript.

Dependencies

Operations may and usually should define a set of resources which they need to be in a certain state before being operated on. This is a necessity since Vulkan, DX12 and Metal requires explicit synchronization for resources, and texture require a transition of layouts depending on their intended usage. Therefore, all operations share two fields which are to be used for this type of behavior. The benefits are that if two operations are rearranged, the frame script system will ensure the synchronization is kept valid, which would otherwise require a lot of hassle.

inputs: [
    {...}
],
outputs: [
    {...}
]

The inputs and outputs actually behave the same way, but the point of their declarations is to know what resources are actually being written by the operation, and which ones are only being read. An example resource dependency can look like this:

inputs: [
    {
        name: "ZBuffer",
        access: "DepthAttachmentRead",
        layout: "DepthStencilRead",
        stage: "ComputeShader",
        aspect: "Depth"
    }
]

This requires the operation to wait for the ZBuffer to finish any operation leading up to being able to read depth attachments, and will transition the resource to the state where the depth values can be read. It will do so for the computer shader stage, and it will only wait for depth data to finish. Here are the possible values and what they mean:

• access - The resource stage in the pipeline, CoreGraphics::BarrierAccess.
  
• layout - The image layout, CoreGraphics::ImageLayout.
• stage - The execution stage in the pipeline, CoreGraphics::BarrierStage.
  
• aspect - The image bits, CoreGraphics::ImageAspect.
  
• mip - The start mip level being used.
• mip_count - The amount of mips to depend on.
• layer - The start layer being used.
• layer_count - The amount of layers to depend on.

Pass

Within a pass, we must list some color-renderable textures and a single (or none) depth-stencil renderable texture. If a clear value is provided for any of these textures, the texture will be cleared with said value when the pass begins. We call color or depth-stencil renderable textures inside passes as attachments, since they are used for attaching to a fragment shader in order to receive color outputs.

Attachments are by default considered to NOT be stored when the pass ends, meaning that the implementation may decide that the result of an attachment is null and void after the pass is done. To avoid this, an attachment must be provided with a store flag, making sure the implementation knows this attachment will be needed after the render pass is done. Try avoid using store as much as possible, since it could improve performance for high-resolution images if the implementation doesn't need to perform a write-back on every attachment.

Likewise, if an attachment should be retained and loaded when the pass enters, the load flag must be provided. Otherwise every attachment is considered to be made up of invalid data, which the implementation may chose to reallocate, resulting in no guarantee the old data can be read.

A typical pass would look like this:

pass: {
    name: "GBuffer",
    attachments: [
        {
            name: "AlbedoBuffer",
            clear: [ 0.7, 0.7, 0.7, 1 ],
            store: true
        },
        {
            name: "NormalBuffer",
            clear: [ 0, 0, 0, 0 ],
            store: true
        },
        {
            name: "SpecularBuffer",
            clear: [ 0, 0, 0, 0 ],
            store: true
        },
        {
            name: "SSSBuffer",
            clear: [ 0.5, 0.5, 0.5, 1 ],
            store: true
        }
    ],
 
    depth_stencil: {
      name: "ZBuffer",
      clear: 1,
      clear_stencil: 0,
      store: true
    },
 
    ...
}

After this pass initial setup, of attachments and optional depth_stencil, we add elements of type subpass. More about that later. In this example, we declare that we will update AlbedoBuffer, NormalBuffer, SpecularBuffer and SSSBuffer at any time during the pass. This means some subpass might write to, for example AlbedoBuffer, and some to SpecularBuffer. This syntax allows for subpasses to communicate dependencies and do rendering operations independently! The values accepted for attachments, which are supposed to be color attachments, are:

• name - Name of the attachment to be used by subpasses.
• clear - If specified, is a FLOAT4 clear color value, if not specified, no clear will be done.
• store - If true, the pixel values will be kept after this pass ends, otherwise, there is no such guarantee.
• load - If true, the pass operations may load values, and perform blending on this attachment.

For depth_stencil, the following fields are accepted:

• name - Name of the attachment to be used by subpasses.
• clear - If specified, is a single FLOAT clear depth value, otherwise, depth will not be cleared.
• clear_stencil - If specified, is a single INT clear stencil value, if not, stencil will not be cleared.
• store - If true, the pixel values will be kept after this pass ends, otherwise, there is no such guarantee.
• load - If true, the pass operations may load values, and perform blending on this attachment.

Subpass

Subpasses use a subset of the color attachments, and optionally the depth-stencil attachment in the pass. Only within a subpass is it legal to perform rendering calls. The following list of operations are available in a subpass:

• name - Name of the subpass, this is important to remember, because it is used to setup subpass dependencies.
• dependencies - A STRING LIST of other subpasses this subpass dependends on, which means this subpass has to wait on some other subpass to finish.
• attachments - A STRING LIST of textures to render to.
• inputs - A STRING LIST of attachments from previous subpasses to read from, these will be InputAttachments in the shading system.
• depth - If true, this subpass will use Z-testing, otherwise the shaders in this subpass can't use z-testing.
• resolve - If true, this subpass will resolve any multisampling attachments being used.
• resources - Just like inputs and outputs from ordinary operations, this subpass declares dependencies on resources using the same syntax as in Dependencies
• viewports - A FLOAT4 LIST of viewports to use instead of the viewport inferred from the attachments.
• scissors - A FLOAT4 LIST of scissor rectangles to use instead of the viewport inferred from the attachments.
• plugin - A plugin callback for rendering.
• batch - A material sorted rendering batch, only takes a single string as the material name to render.
• sorted_batch - An unsorted rendering batch, which allows for arbitrary sorting. Not yet implemented.
• fullscreen_effect - A full screen quad using a specific shader, and texture to inherit its size from.

Full screen effects provide an additional piece of syntax, an example can look like this:

fullscreen_effect: {
    name: "Finalize",
    shader_state: {
        shader: "finalize",
        variables: [
            {
                semantic: "LuminanceTexture",
                value: "AverageLumBuffer"
            },
            {
                semantic: "DepthTexture",
                value: "ZBuffer"
            },
            {
                semantic: "ColorTexture",
                value: "LightBuffer"
            },
            {
                semantic: "BloomTexture",
                value: "BloomBufferBlurred"
            },
        ]
    },
    size_from_texture: "ColorBuffer"
},

The shader_state member defines an object encapsulating a shader and a set of variables to apply when rendering. The semantic corresponds to a shader variable name, and the value is the texture or buffer from the frame script. size_from_texture determines the size to use when rendering the post effect.

Script Interface

A frame script then executes actions, and all actions are inherited from the class FrameOp. FrameOp provides a couple of default functions used to handle rendering. FrameOp is the base class, and is meant as the template from which a FrameOp::Compiled is constructed. Compiled frame ops are the finished product, constructed automatically when we run FrameScript::Compile, and will construct Compiled operations only for the operations which are enabled. This allows for the frame script to always run a minimal update, and arrange the internal synchronization correctly.

 
virtual void Run(const IndexT frameIndex);
virtual void OnWindowResized();