Ramblings on writing game engines

[enlynn]

Intro

I do a lot of thinking about game engines - perhaps too much. Part of it is because it relates to my day job, but also, I just really enjoy working on problems that come with creating little engines. I don’t usually share what I am thinking outside of stream since I am not a fan of short form social media (i.e. Bluesky, Mastodon, etc.). However, I want to try out posting my ideas/thoughts on this forum as I work on my own little engine.

I happen to want to take a fresh look at how I design my engines, so for this first little post, I am going to ramble a bit on my chosen language and build system for game engines.

The Language

I like to learn different programming languages. Each language has its own unique design and way of designing programs, so I often find myself writing a little renderer in different languages to see how the language might change my approach to programming.

However, I really like C. Of all the languages I have used over the years (C++, Rust, Odin, Zig, Go), C remains my favorite language and I always find myself returning to it. The spec is simple. The language is simple. The build systems are atrocious. I can program without feeling like the language is in my way. So, for this little engine, I am going to use C - or more specifically C99.

There’s also the decision of which compiler’s I want to support. For now, I am going to support the main compilers - GCC, Clang, and Microsoft’s Visual Compiler. There is a part of me that wants to just use GCC so I can have my own little defer macro, but getting GCC on Windows is…interesting.

The Build System

The build system ecosystem in C (and C++) is probably the worst of any language I have ever learned. Here are the tools people like to use (in various combinations):

• Makefiles
• NMake (Microsoft Makefiles)
• Ninja
• Autotools
• CMake
• Meson
• Bazel
• Build2
• vcpkg
• Conan
• …and probably many more.

There are many build tools in this ecosystem, and honestly, they all have their own flavor of suck.

Makefiles, NMake, Ninja, and Build2 are your actual “build tools” - the programmer provides a set of targets/recipes, and invoke the compiler with the correct flags.

Unfortunately, Makefiles and co. are written in the most obtuse languages possible, so folks created build tool generators so you don’t have to write them! That’s where CMake, Autotools, Meson, and Bazel come in. The programmer now writes a set of targets/rules in the generator’s language, which will then generate the build tool, which then invokes the compiler. Easy peasy, right? (/s)

Okay, but any massive project is going to have dependencies, which probably have their own dependencies, and none of the build generators are particularly good at resolving nested dependencies (they can do it, but it might get complex). This is where package managers like Conan and vcpkg come in! They’ll handle all of your deps for you - you need to just write some config files specific to the package manager.

Let’s see, we have:

• A C Compiler
• A build tool language
• A build generator language
• A package manager language

Sheesh, just to start programming, I need to use (learn) 4 languages! It’s no wonder C++ is so bloody hard to learn.

takes deep breadth

I (mostly) learned C by watching Casey Muratori work on Handmade Hero, and his approach to build his game is with a batch script. The script is like 10 lines code. The source is organized around a “unity” build, where all source files are compiled into a single translation unit (as opposed to the traditional one translation unit per source file), so compilation is dead simple.

To this day, a batch script + unity builds are the most elegant approach to compiling code I’ve come across. They are inherently simple - the script only cares about the unity source file. The downside is that you have to compile all of your code at once, but honestly, compiling my entire codebase in C is often faster than incremental builds in most other languages.

If I were to use a batch (or in my case, a bash script), a user (me) only needs the compiler to get started. Bash is supported on just about every platform, and if you have Git on Windows, you have a bash interpreter. Building the code base is simply:

# assuming a compiler is already installed
sh bin/init_local.sh # setup basic local filters
sh bin/build_all.sh  # build the codebase

as opposed to:

# assuming a compiler is already installed
pacman -S cmake ninja git # install deps

sh bin/bootstrap_vcpkg.sh  # fetch vcpkg if not installed
cmake --preset debug       # generate build files for debug preset
cmake --build build-debug/ # build the codebase

My approach to writing build scripts has always been cumbersome when I try to add in multiple platforms and compilers, so I always end up just using CMake/Ninja.

However, I just so happened to stumble across Mr4th’s (the creator of 4coder - one of my favorite editors) gitea, where he was rebuilding his personal code base. He had written a set of bash scripts allowing his build system to support multiple platforms/compilers through the use of filters. It was the missing piece I needed for my own set of bash scripts, so I decided to test out his code with my codebase and it worked perfectly!

Going forward, I think I am going to keep using his scripts as part of my build system. If it continues to work out, I might write another post on it!

Conclusion

I had planned to talk about a few more topics in this post, but I think it is long enough. I might go into more detail on the build system in my next post, once I’ve had a few more days to play around with it or maybe the general architecture/goals I am aiming towards.

I’m curious to know if anyone is interested in my ramblings and might want more of them or have any game engine specific topics you might want me to discuss (although rendering architecture is where most of my experience is).

Until next time,
Enlynn

[ZeikJT]

Super cool! I 100% agree that simplicity is often best, and I’d also rather have a little compile script that does everything in a couple of lines and sacrifices some “performance” that I don’t need because I’m not compiling a trillion lines of code.

Whenever I’ve seen Casey speak he’s usually doing a good job of being practical and pragmatic, which I super appreciate. Guess I should watch the Handmade Hero stuff…

Oh also, getting gcc on windows is not that bad anymore, at least via subsystem where you can just apt-get install it. Also has the benefit that if you have subsystem you can just run bash scripts too. At least that’s how it was last I tried it.

I am also definitely interested in hearing more ramblings!

[exodrifter]

Could you link Mr4th’s repository where you found the bash scripts you decided to use? I found their gitea repository but there sure is a lot of stuff there so I’m not sure where the scripts you’re talking about are exactly.

The bad build system ecosystem is one reason I dislike working with C and C++, it’s always so tedious and annoying. I am always so thankful when I’m using a language that has a relatively nice all-in-one build system that manages fetching dependencies and compiling like Rust’s cargo run or Haskell’s cabal run.

For your engine, are you planning on providing some kind of way for code written in other languages to be used? I’ve always been interested in building a modular game engine where different subsystems can be freely glued together to make an engine that does exactly what you want it to do, and I feel like supporting code written in other languages is important for this, so you have access to more ecosystems.

[Choto]

@enlynn

but getting GCC on Windows is…interesting.

This part in particular really stood out to me. I am someone who’s tried and failed numerous times to get GCC working right on my Windows PC, I appreciate the mention. Very good read, I hope to see more coming down the way 8D

[fuguesoft]

Thanks for writing this! Not quite there yet myself but this will be useful to come back to!

[enlynn]

I knowww, Exo, - everytime I have to do something C/C++ build system related, I’m just like “Why can’t it be as easy as Rust or similar language?”

The scripts I am basing my build system on are here: mr4th/bin at main - mr4th - Mr. 4th Git

The system itself is relatively simple. You create a few line-delimited txt files under the options directory that specify compiler flags/options combined with “filters”. For example, I might have:

cmp:cl>mode:debug>-Zi

Filters are delimited by the “>” character. In the above snippet, there are two filters:

1. cmp:cl - which translates to the CL compiler.
2. mode:debug - which translates to compiling in Debug mode

Filters can also be used to filter subprojects! For example, the gpu-vulkan submodule requires the VK_NO_PROTOTYPES compiler flag, so I can have the compiler option:

gpu-vulkan>-DVK_NO_PROTOTYPES

and this flag will only be added when compiling a module that specifies the filter “gpu-vulkan”.

The user has a local directory (it should be added to .gitignore) that provides a set of project wide filters (like the compiler and build modes). For my little engine right now, the local_vars file is just:

#!/bin/bash
compiler="gcc"
compile_mode="debug"
assembler="ml64"
arch="x64"
linker="gcc"
ctx_opts=""
os_type="darwin"

For your engine, are you planning on providing some kind of way for code written in other languages to be used?

If I were to make the engine public for others to use then this should be possible! The engine API would be compliant with the C ABI, so it should technically be usable in other languages.

[nmorales]

No one likes writing cmake
But at least as far as open source goes, I think there is no other viable alternative. While I’m sure the build scripts click for you specifically, they make code really difficult to use for other people. With something like CMake which has plenty of flaws, the main advantage is its de-facto standardization.

Let’s say I clone your repository – all I have to do is:

cmake -B ./build -S .
cmake --build ./build
ctest --test-dir ./build

Want a different compiler? Just change the first line:

cmake -B ./build -S . -DCMAKE_C_COMPILER=clang

Let’s say your project has a dependency on GLFW:

cmake -B ./build -S . -DCMAKE_C_COMPILER=clang -Dglfw3_ROOT=/path/to/glfw/install

I would have no idea how to do that with those scripts. If you plan on making your engine open source as someone who consumes a lot of open source project’s I’d really suggest using CMake.

Even if it’s just for your own use, I think bash scripts can only scale so much. They may be good for small projects but become incredibly unwieldy for large ones. I have a pretty large C++ graphics application I’m working on; I have separated out the match components, utility libraries, and rendering components into separate projects (as they can be used for other stuff like game engines). Moreover, like your project it is cross platform and compiles with GCC, Clang, and MSVC. I would go insane trying to scale a bash script to track all of this.

With just a pure CMake approach, all I would have to do to add a dependency across platforms is add the following two lines to my cmake:

find_package(foo 1.0 REQUIRED)
target_link_libraries(mylib PUBLIC foo::foo)

In practice I also use Conan as a package manager due to the sheer number of dependencies I have. So It’s a bit more complex than that as I have a whole CI/CD pipeline to build package binaries. I can go into more detail on how it works (that part is a bit complex), but in the end you can clone my code and basically do:

conan install .
cmake --preset build-release
cmake --build --preset build-release
ctest --test-dir ./build

And it will download all the binaries of the dependencies and compile and test the project.

Furthermore if you are developing your game engine as a library for other people to link to, without something that generates a libraryConfig.cmake other people using standard build systems will have trouble linking to your library. A good example of this was the Boost C++ library which is very popular; CMake even had to ship their own custom FindBoost.cmake script because Boost used their own custom build system b2. Eventually the Boost developers finally wrote some build system code to generate the correct BoostConfig.cmake so downstream applications could use it properly, and now they even have CMake build system support (it was so much worth it for them that they are maintaining two build systems for the project because there is so much demand for CMake).

It’s similar with something like Godot – they use a horrible build system generator called SCons that no one else uses, and it got bad enough to the point where godot-cpp now maintains a CMakeLists.txt in addition to a SConstruct file.

One final point – bash scripts don’t tend to work well with editors. With CMake, I have nice autocomplete integration out of the box with CLion, Visual Studio, VSCode (ugh I know) and variants, QTCreator, etc. Even with something like vim and clang-tidy CMake has an easy way of exporting the compile_commands.json it needs.

As someone developing game-related projects in C++, I’m very interested in seeing your thoughts in the future. I didn’t mean this reply to be negative at all and I hope it wasn’t – build systems are something I feel pretty strongly about because they are so integral to interoperating with other code. I entirely agree with your conclusion that the C and C++ build ecosystem is immensely flawed, and I know CMake is not the uh… most ideal language to write a build system in. But pragmatically, I feel like it’s the de-facto standard. In principle Meson is supposed to be more elegant, but so far I’ve not had enough motivation to try it so I can’t comment too much on it.

[Dminshd]

This is exactly what I’ve been working on recently. I’m a little beyond this point but this is still great. I’m using clang since it seems to work on all the desktop platforms. I looked at build systems but that only put me on the path of looking in how to implement my own.

[enlynn]

I didn’t think you were being too negative at all @nmorales ! This might seemingly contradict my post, but I’d actually largely agree with you!

CMake is definitely the de-facto build system at the moment, and if I were to want to get something off the ground quickly, then I would use CMake. It just works (usually).

Something I didn’t really mention in my post is that my problem with CMake and similar system is a philosophical one - why do we need a build system generator? Why not just generate the cache and build the application?

There are likely some historical reasons behind this that I am not familiar with, but when I look at other languages, none of them have adopted a similar complexity (partially because they don’t have to deal with the many compiler problem C/C++ has).

For dependencies, I usually go very lite on dependencies, and the ones I do have, I integrate directly into the source tree. While FetchContent is a super useful tool, I just don’t find myself needing it all that often!

Another difference in workflow I have is that I don’t use clangd/clang-format or similar tools all that much. When I do, usually a simple compile_commands.json does what I want. Although, this definitely is not going to work for a larger open source project with multiple contributors.

After sitting on the build scripts for a few days and writing a bunch of engine code, the limits I have seen are this:

• Bash is just plain hard to read (at least for me)
• While Bash is technically multiplatform, it does have compatibility issues. This is currently the biggest problem for me, honestly.

I’m currently in the process of porting the bash scripts to C, but I can’t really comment on it at the moment since it will be a while before I will determine it is was the “right choice”.

(I had forgotten to mention SCons before! I must’ve tried to forget about it for the short time I was using it to compile GDExtensions before swapping to CMake lmao)

[Haru]

Good read! Gamedevving in C brings back good memories I started with DOS edit and DJGPP and for some time I thought the only way to organize code is to have one .h file and one .c file…because that’s how all the tutorials were. Soo..my arkanoid clone had ~100kb header file and I spent more time searching stuff than coding. My memory might fail, but I feel like the compiler errors/warnings were more cryptic back them (most probably because my lack in english language though).

Sometime between college and uni I also dabbled making an engine in C. I think I got to the point having OpenGL renderer, mesh loader and basic cloth simulation. The sources might still lurk on one of the dozens of CDs I have, but I fear they are already in bit heaven

Anyway, engine coding is surely challenging, but rewarding task!