How To Create 3D Materials With Shader Programming Using FireMonkey

Shaders give your programs the ability to take full advantage of the GPU for graphical effects and more. Some common use cases are creating materials applied to 3D objects, and other general filters or effects. The specifics of shader programming are not unique to the application programming framework you are using. While this post will focus mostly on creating materials in 3D with the , you can easily use what you learn in your programming language and framework of choice. We using Delphi FireMonkey to get all six samples working on Windows, macOS, iOS, Android, and Linux. Delphi FireMonkey framework 3D for 2D You may assume that 3D forms and applications are only useful for specific use cases where you need to work with 3D models or other types of 3D content. However, 3D forms can also be used to create a purely 2D user interface, by using plane-like objects without a camera so they look like regular 2D rectangles. A reason you may want to do this is so you can utilize GPU shaders to render those rectangles. For example, our product looks like a 2D application, but the main form is actually a 3D form built with FireMonkey. This enables us to very efficiently render multiple live videos and other content to the screen using specialized GPU shaders, thereby freeing the CPU to perform other tasks (such as decoding those videos). Lumicademy About GPUs and Shaders The GPU or video card is responsible for rendering 2D and 3D content to the screen. When it comes to 3D content, the GPU only works with 3D triangles (and sometimes points and lines). It converts the coordinates of these triangles to screen coordinates, clips the triangles to the screen, and renders them using colors, textures, or custom effects. In the early days, video cards and APIs used a fixed graphics pipeline. This means that the GPU would take care of most of the hard work for you: You would give the API a list of triangles, colors, texture coordinates, and other data, and the GPU would figure out how to render it all. If the GPU wasn’t capable of certain functionality, the API (eg. DirectX) would use software emulation if configured to do so. Nowadays, virtually all GPUs and APIs use a programmable graphics pipeline. It could be said that in some ways this is a step back since the GPU and API do less work for you. Instead, you need to tell the GPU how triangles (vertices) must be converted to screen coordinates, and how each pixel must be rendered. This is a lot more work for the developer, but also offers much more flexibility and enables functionality that is simply not possible with a fixed pipeline. The newest graphics APIs (DirectX 12, Metal, and Vulkan) shed even more abstraction layers to give developers even more control over the GPU hardware. This is achieved at the cost of increasingly complicated APIs but allows for even more efficient rendering. Depending on the GPU and API, the graphics pipeline can have multiple programmable stages. Every programmable GPU has at least two programmable stages: a vertex transformation stage and a pixel shader stage. Newer GPUs and APIs may add additional stages, such as a tessellation stage. But unless you are creating a sophisticated game engine, you only need the first two stages. These are also the only stages you can customize in FireMonkey. The following image shows a possible graphics pipeline with 4 stages, of which the first and last ones are programmable and the two middle ones are fixed. In the vertex transformation stage, individual triangle vertices are transformed from 3D world space to 2D screen space. You program this stage by creating a vertex shader. This shader is also used to pass data that is needed for rendering to the pixel shader stage. The vertex shader is called once for each vertex. The next two stages are fixed. The shape assembly stage converts the transformed vertices to shapes (triangles). The rasterization stage then rasterizes these triangles into a set of pixels. Finally, the pixel shader stage determines the color of each pixel. This is the second programmable stage, for which you write a pixel shader (aka fragment shader in OpenGL and Metal). This shader calculates the final pixel color based on some algorithm and/or textures supplied to the shader. It can receive input data from the vertex shader if needed. The pixel shader is called once for each pixel. Shader Languages Although most concepts of vertex and pixel shaders are the same across all graphics APIs, each API has its own shader language that is used to write these shaders: On Windows, the DirectX API is used, which uses the High-Level Shader Language (HLSL). Windows also supports OpenGL, but DirectX is the native API. On Android, Linux, macOS, and iOS, the OpenGL API is used, which comes with the GL Shader Language (GLSL). OpenGL is deprecated on macOS and iOS (although still supported at the time of writing). You should use the Metal API on these platforms and write your shaders in the Metal Shader Language (MSL). One of my shows how to enable Metal on these platforms. previous blog posts There is also a new cross-platform API called Vulkan, which is the official successor of OpenGL. Unfortunately, since this API is not supported by Apple, it hasn’t gained much traction yet. If it becomes popular in the future, then Delphi will add support for it. This would also mean another shader language to learn (although it is very similar to GLSL). All APIs provide the option to compile the shader source code on the fly in your app. Some APIs also provide the option to compile the shader off-line. In that case, you pass the compiled bytecode to the API. About 3D in FireMonkey Although most developers probably use FireMonkey as a 2D application framework (much like the Visual Component Library or VCL), FireMonkey had support for 3D applications from the beginning. This post will not go into the details of creating 3D applications with FireMonkey. That would leave little room to talk about the actual topic of shaders (and it is long enough already). If you want to learn more about 3D programming in FireMonkey you can consult the , , blog posts like , or . Delphi documentation Delphi sample projects Bruce McGee’s 3D article Andrea Magni’s excellent FireMonkey book FireMonkey uses compiled bytecode for DirectX shaders and source code for OpenGL and Metal shaders. To compile DirectX shaders, you need to use the Direct3D Shader Compiler tool (fxc.exe), which ships with the DirectX SDK. I also included this tool in the that accompanies this post. GitHub repository 2 Ways to do 3D in FireMonkey Here are two ways you can add 3D content to your Delphi FireMonkey application: You can use a 3D form (TForm3D) and add your 3D controls to the form. If you also need to use 2D controls, you will need to add a TLayer3D (or TBufferedLayer3D) to the form and add your 2D controls to that layer. You usually want to change the Projection property of that layer to Screen so it doesn’t get distorted by the camera. You can use a regular (2D) form and add a TViewPort3D to it. You then add your 3D controls to the viewport. Which option you should use depends on the situation. In general, if most of your form contains 3D content, you should use the first option. Otherwise, the second option is more efficient. FireMonkey Materials In FireMonkey, the vertex shader and pixel shader are encapsulated in a material (derived from ). However, you cannot apply a material directly to a 3D object. Instead, FireMonkey uses the concept of a material source (derived from ), which is used to link a material to a 3D object. You usually add a component derived from to your form and set the property of a 3D control to this component. The material source will then create the corresponding descendant when it is needed to render the control. TCustomMaterial TMaterialSource TMaterialSource MaterialSource TMaterial So materials usually come in pairs: a material source and a material. For example, Delphi’s is used at design time. It creates a at runtime for rendering. TColorMaterialSource TColorMaterial The remainder of this post focuses mostly on creating some custom material sources and materials for rendering various kinds of effects. Demo Applications To demonstrate the use of custom materials and shaders, I have added a GpuProgramming folder to our with six sample projects. We start simple, with a shader that renders just blue pixels, and then work our way up to (slightly) more advanced scenarios that use textures and finally a simple plasma effect (which is also used for the title image of this post). JustAddCode GitHub repository Most sample applications just show a spinning control with our custom material (source) applied to it. Since GPUs work exclusively with points, lines, and triangles, a rectangular plane is represented with two triangles. FireMonkey takes care of creating these triangles for you and passing them to the GPU. TPlane You can install your material sources into a design-time package so you can place them on a form and link them to controls visually. However, to keep the samples simple and avoid dependencies on packages, the material sources are created and linked in code. This is done in the event of the form. For example, for the first demo app, this method looks like this: OnCreate FMaterialSource := TBlueMaterialSource.Create(Self); Plane.MaterialSource := FMaterialSource; ; . ; procedure TFormMain Form3DCreate (Sender: TObject) begin end The code simply creates a material source (for a solid blue material in this example) and links it to the plane. We will look at how to create this material next. Alternative Backends By default, the sample applications use the default graphics backend for the platform. This is DirectX 11 on Windows and OpenGL on all other platforms. However, you can also build each sample with an alternative backend by choosing the “Debug_AlternativeBackend” configuration. This configuration uses DirectX 9 on Windows and Metal on macOS and iOS. Each sample app has a header that shows the name of the context class (and thus graphics backend) that is currently used (for example, when DirectX 11 is used). TDX11Context Sample 1: Getting Started This is the easiest example, that just renders every pixel using a blue color. Even so, this example takes up the most space in this article due to the fact that there is a lot of scaffolding to put up. Before we look at the Delphi side, let’s take a look at what shader source code looks like. Vertex Shader – HLSL Let’s start with the HLSL (DirectX) code of the vertex shader (in the file VertexShader.DX.txt): 4x4 MVPMatrix; 4 main( 4 position: POSITION0): SV_Position0 { mul(MVPMatrix, position); } float float float return All shader languages are C-like languages, and HLSL is no exception. The sole purpose of this vertex shader is to convert a vertex coordinate (a 4D vector) from world space to screen space. This is done by multiplying the with a model-view-projection matrix ( ). This 4×4 matrix (of type ) is calculated for you by FireMonkey based on the camera and viewport. It is a combination of three matrices: position position MVPMatrix float4x4 The Model matrix converts the model (vertex) coordinates from local space to world space The View matrix converts the result from world space to camera space And finally, the Projection matrix converts from camera space to screen space, taking things like lens size and perspective projecting into account The variable is a so-called uniform input, which means that its value is constant for multiple invocations of the shader (that is, its value is the same for each vertex that is transformed). FireMonkey calculates this matrix in Delphi code and passes it to the shader. MVPMatrix The shader must have a function called that returns the transformed position (a 4D vector of type ). Its input is the source position in the local space. This type of variable is called a varying input in HLSL (or attribute in GLSL), meaning that its value is unique to each invocation of the shader. These inputs must be marked with a semantic, which is a name used to link the input and output of parts of the graphics pipeline. The semantic used here means that this parameter represents position data. main float4 POSITION0 The entire function is also marked with a (system value) semantic called , meaning that the function result represents the transformed position. SV_Position0 FireMonkey requires that HLSL shares are compiled to bytecode. This is done with the following command line: fxc /T vs_3_0 /E main /O3 /FoVertexShader.DX9 VertexShader.DX.txt fxc /T vs_4_0 /E main /O3 /FoVertexShader.DX11 VertexShader.DX.txt The parameters of interest are: to specify the target profile. For our purposes, the following values are used: /T : a version 3.0 vertex shader (used by DirectX 9) vs_3_0 : a version 4.0 vertex shader (used by DirectX 11) vs_4_0 : a version 3.0 pixel shader (used by DirectX 9) ps_3_0 : a version 4.0 pixel shader (used by DirectX 11) ps_4_0 to specify the name of the entry point, which is main in our case /E to specify the name of the output file /Fo This is followed by the name of the input file. Each sample project has a Shaders directory with the source code of all shaders, as well as a batch file ( ) that compiles the shaders and generates a resource file. Build.bat Vertex Shader – GLSL The GLSL (OpenGL) version looks a bit different: _MVPMatrix[ ]; a_Position; main() { .x = (_MVPMatrix[ ], a_Position); .y = (_MVPMatrix[ ], a_Position); .z = (_MVPMatrix[ ], a_Position); .w = (_MVPMatrix[ ], a_Position); } uniform vec4 4 attribute vec4 void gl_Position dot 0 gl_Position dot 1 gl_Position dot 2 gl_Position dot 3 The main differences compared to the HLSL version are: The data types are named differently (eg. instead of and instead of ). vec4 float4 mat4 float4x4 Uniform inputs (constants) must be marked with a keyword. These variables must start with an underscore (as in ). This is not a GLSL rule but a FireMonkey requirement. uniform _MVPMatrix The per-vertex inputs are not passed as a parameter to the function, but must be declared with an keyword instead. Attributes can have any name, but FireMonkey requires fixed names so it knows what these attributes represent (in HLSL, this is not required since the semantic is used for this purpose). So position attributes must be named (the prefix is also a FireMonkey requirement). main attribute a_Position a_ GLSL uses some predefined variable names (starting with a prefix) for common variables in the pipeline. The output vertex coordinate is always stored in the predefined variable . gl_ gl_Position GLSL also supports matrix multiplication, so the variable could be of type . However, FireMonkey requires that matrices are passed as arrays of vectors instead. This also means that the matrix multiplication has to be split into four separate vector dot operations. Fortunately, this part of the vertex shader is boilerplate, and you will use the same code in most GLSL vertex shaders you write. _MVPMatrix mat4 There is no need to compile this shader off-line because FireMonkey will compile it for you when needed. Vertex Shader – MSL Finally, we have the MSL (Metal) version: using namespace metal; struct Vertex { }; struct ProjectedVertex { float4 position ; }; vertex ProjectedVertex vertexShader( constant Vertex *vertexArray , const unsigned int vertexId , constant float4x4 &MVPMatrix ) { Vertex = vertexArray[vertexId]; ProjectedVertex out; out.position = float4( .position[ ], .position[ ], .position[ ], ) * MVPMatrix; out; } [[position]] [[buffer(0)]] [[vertex_id]] [[buffer(1)]] in in 0 in 1 in 2 1 return This looks a bit more complicated: The part means that the code can use standard types and functions from the namespace (like a uses clause in Delphi). using namespace metal metal Next, it declares two structs (records): The struct represents the type of input vertex. FireMonkey will fill this in for you by replacing with the source code needed to represent the vertex (this tag is not a Metal feature). Vertex The structure represents the output vertex. In this case, it only contains an output , but we will add more in later examples. The attribute is like a semantic in HLSL and indicates that this field represents a vertex position. ProjectedVertex position [[position]] Finally, we have the main function, which starts with the keyword vertex to indicate this is a vertex shader (not to be confused with Vertex within an uppercase V, which is the type of the input vertices). It returns the transformed vertex of type ProjectedVertex. This shader has three parameters: The parameter is an array of input vertices (of type Vertex). It has a constant qualifier, meaning that the vertex array is stored in the constant (read-only) address space (not to be confused with the const qualifier). Parameters in the constant address space must have a [[buffer(index)]] attribute, where the index is the location of the buffer. This index is later used in Delphi code to link this parameter to the vertices supplied by FireMonkey. vertexArray The parameter contains the index of the vertex in the parameter that is currently being processed. This parameter has a const (not constant) qualifier meaning it is read-only (much like the const qualifier for parameters in Delphi). It must have a [[vertex_id]] attribute to tell Metal what the parameter is used for. vertexId vertextArray Finally the parameter contains the 4×4 model-view-project matrix, also stored in the constant address space. It uses a different buffer index than the vertexArray parameter. MVPMatrix The body of the function just extracts the vertex with the given Id from the array and multiplies it with the transformation matrix. Again, there is no need to compile this shader off-line. Pixel Shader – HLSL The pixel shader is pretty simple since it always returns just a blue color: 4 main(): SV_Target0 { 4( , , , ); } float return float 0.0 0.0 1.0 1.0 The function returns a 4D vector, which is not only used for positions (X, Y, Z, W), but also for colors (R, G, B, A). Color components range from 0.0 (fully off) to 1.0 (fully on). Alpha components also range from 0.0 (fully transparent) to 1.0 (fully opaque). If you use values outside of this range, the GPU will automatically clip (or saturate) them. The body of the function sets the Blue and Alpha components of the color to 1.0. Remember to always set the Alpha value as well. Note that the function is marked with an semantic, meaning it returns the target pixel color. SV_Target0 Pixel Shader – GLSL main() { = ( , , , ); } void gl_FragColor vec4 0.0 0.0 1.0 1.0 Here, the predefined variable must be set to the output color. gl_FragColor Pixel Shader – MSL Finally, the Metal version looks like this: fragment 4 fragmentShader() { 4( , , , ); } float return float 0.0 0.0 1.0 1.0 Here, the function must start with the keyword to indicate that this is a pixel (or fragment) shader. fragment Shader Material Now we are finally ready to use these shaders to create a material source and material (remember, these always come in pairs). The material code is specific to Delphi's FireMonkey and will be different if you choose to use a different framework. Since these materials have no configurable properties, the implementation is pretty simple. Let’s start with the material source: = (TMaterialSource) TMaterial; ; ; TMaterial; Result := TBlueMaterial.Create; ; type TBlueMaterialSource class protected : function CreateMaterial override end . : function TBlueMaterialSource CreateMaterial begin end You must always override the method and create the actual material instance in its implementation. CreateMaterial If you look at the source code of materials that ship with Delphi (in the FMX.Materials unit), you will see that an offline tool is used to convert the (byte) code of HLSL and GLSL shaders to Delphi byte arrays. The MSL shaders are included as Delphi strings of MSL source code. In the sample projects, we use a different approach that does not require an external tool: we link the shaders into the executable as resources and use a to load them. The interface of the class looks like this: TResourceStream TBlueMaterial = (TCustomMaterial) FShaderArch: TContextShaderArch; FVertexShaderData: TBytes; FPixelShaderData: TBytes; FMatrixIndex: Integer; FMatrixSize: Integer; ; TBytes; ; ; ; type TBlueMaterial class private class var private class ; procedure LoadShaders static class : function LoadShader ( AResourceName: ) const String static protected ; procedure DoInitialize override end The class defines five static class variables (because we do not need different values of these variables for each instance): contains the shader architecture that is currently used, based on the graphics backend. It will have the values DX9, DX11, GLSL, or Metal. FShaderArch and contain the byte code or source code of the shaders, as read from the resource. FVertexShaderData FPixelShaderData contains the index of the variable in the shader, which is needed to link the Delphi matrix to the corresponding matrix in the shader. This value can be different depending on the shader architecture. FMatrixIndex MVPMatrix contains the size of a matrix. This also depends on the shader architecture. For DX11, this is the size of a matrix in bytes (which is 4 x 4 x 4 = 64). For other architectures, this is the size of a matrix as the number of 4D vectors it contains (which is 4). FMatrixSize You must always override the method to register the shaders: DoInitialize ; (FShaderArch = TContextShaderArch.Undefined) LoadShaders; FVertexShader := TShaderManager.RegisterShaderFromData( , TContextShaderKind.VertexShader, , [ TContextShaderSource.Create(FShaderArch, FVertexShaderData, [TContextShaderVariable.Create( , TContextShaderVariableKind.Matrix, FMatrixIndex, FMatrixSize)]) ]); FPixelShader := TShaderManager.RegisterShaderFromData( , TContextShaderKind.PixelShader, , [ TContextShaderSource.Create(FShaderArch, FPixelShaderData, []) ]); ; . ; procedure TBlueMaterial DoInitialize begin inherited if then 'blue.fvs' '' 'MVPMatrix' 'blue.fps' '' end If this is the first time this material is used, then the shaders are loaded from the resource using the static class method LoadShaders. Next, it registers the vertex shader and pixel shader. Note that FVertexShader and FPixelShader are fields of the TCustomMaterial class, from which TBlueMaterial derives. The method has these parameters: TShaderManager.RegisterShaderFromData A name that uniquely identifies that shader. You can use any name you want as long as it is unique in your code. A convention is to use a name based on the material class name, with an .fvs suffix for vertex shaders and .fps suffix for pixel (fragment) shaders. The kind of shader (either VertexShader or PixelShader). An optional string containing the original shader source code for reference. This is usually left empty . An array of shader sources (each of type TContextShaderSource, which is a record type). When you look at the built-in materials that ship with Delphi, you will notice that it registers shaders for all languages here. However, since we already know which graphics backend is being used, we only use a single TContextShaderSource element with the following parameters: The shader architecture (DX9, DX11, GLSL or Metal) The shader bytecode or source code An array of shader (uniform) variables that are passed from Delphi to the shader This simple shader only has a single shader variable: the model-view-projection matrix. The TContextShaderVariable record has a constructor with the following parameters: The name of the shader variable ( in this case) MVPMatrix The type of the variable ( here) TContextShaderVariableKind.Matrix The index of the variable in the shader. For DirectX and OpenGL shaders, this is the zero-based index of the variable as it appears in the source code (that is, the first variable has index 0, the second variable has index 1, etc.). For Metal shaders, this is the index as specified in the attribute discussed above. In our sample, we pass the value of FMatrixIndex here, which is set in the LoadShaders method. [[buffer(index)]] The size of the variable in units that depend on the shader architecture. As mentioned earlier, DirectX 11 uses a size in bytes, and other architectures use the number of 4D vector units as size . We use , which is also set in the LoadShaders method. FMatrixSize The LoadShaders method detects the current shader architecture, sets the size of the and fields accordingly, and loads the corresponding vertex and pixels shaders from the resource: FMatrixIndex FMatrixSize Suffix := ; ContextClass := TContextManager.DefaultContextClass; (ContextClass.InheritsFrom(TCustomDX9Context)) FShaderArch := TContextShaderArch.DX9; FMatrixIndex := ; FMatrixSize := ; Suffix := ; (ContextClass.InheritsFrom(TCustomDX11Context)) FShaderArch := TContextShaderArch.DX11; FMatrixIndex := ; FMatrixSize := ; Suffix := ; ; (ContextClass.InheritsFrom(TCustomContextOpenGL)) FShaderArch := TContextShaderArch.GLSL; FMatrixIndex := ; FMatrixSize := ; Suffix := ; ; (ContextClass.InheritsFrom(TCustomContextMetal)) FShaderArch := TContextShaderArch.Metal; FMatrixIndex := ; FMatrixSize := ; Suffix := ; ; (FShaderArch = TContextShaderArch.Undefined) EContext3DException.Create( ); FVertexShaderData := LoadShader( + Suffix); FPixelShaderData := LoadShader( + Suffix); ; TBytes; Stream := TResourceStream.Create(HInstance, AResourceName, RT_RCDATA); SetLength(Result, Stream.Size); Stream.ReadBuffer(Result, Length(Result)); Stream.Free; ; ; class . ; procedure TBlueMaterial LoadShaders begin var '' var {$IF Defined(MSWINDOWS)} if then begin 0 4 'DX9' end else if then begin 0 64 'DX11' end {$ELSE} if then begin 0 4 'GL' end {$ENDIF} {$IF Defined(MACOS)} if then begin 1 4 'MTL' end {$ENDIF} if then raise 'Unknown or unsupported 3D context class' 'VERTEX_SHADER_' 'PIXEL_SHADER_' end class . : function TBlueMaterial LoadShader ( AResourceName: ) const String begin var try finally end end Now we can finally apply this material to a 3D control and run the application: I know, pretty boring for that much work. But we have to take baby steps before we can run. Fortunately, with most of the scaffolding setup now, building on this in the next examples will be faster. What's Next? As promised I have 6 samples ready for you but to keep this post short I'll not paste everything here. You can jump out to the or my for all 6 samples, or I'll be back next week with . original blog post GitHub repository part 2 Lead Photo by James Bak on Unsplash