My Experience With Jai (Part 3)

Alright, with an actual file converted into Jai, I think we have a good idea of what these conversions are going to look like, so I’ll try to get through a few more in a single post this time.

gy_rectangles.h

gy_rectangles.h is one of the largest files I have in Gylib because it ended up doing more than just “rectangles.” In this context, rectangles are anything in 2D or 3D that has perpendicular flat faces. So that includes AABBs in both 2D (rec) and 3D (box) as well as OBB in both 2D (obb2) and 3D (obb3). I also have integer variants for rec and box called reci and boxi. So that gives us 6 distinct types that are somewhat similar but are actually, in reality, quite different than each other. What I mean by this, is a lot of functions that work just fine on axis-aligned rectangles in 2D do not make sense on an axis-aligned 3D box, or even a rotated 2D rectangle. So although some functions have 6 variants, many of them only have 2-3 variants. Because of this, the file will probably get split someday. But for now, we’ll move on.

As before, HERE is the (mostly) completed gy_rectangles.jai and I’ll point out some of the interesting things that happened with this file

Positive 1: #place and Operator overloads worked wonderfully

Rectangles and boxes are not normally pointed to as “standard” mathematical types for most CS or Linear Algebra discussions. However, I’ve found that they serve as fundamental a role in game development as vectors which means that proper member aliases and some simple operator overloads go a long way. For example, being able to say rectangle.topLeft to get a v2 that represents the x and y members of the rectangle is super useful. And I can still do rectangle.x instead of rectangle.topLeft.x. Also operator overloads being fully available in user space means that I can do things like rec + v2 to indicate I want to “offset” my rectangle by some amount.

Problem 1: No Way to Represent Derived Values (afaik)

This is not something I had in C/C++ but it is something I would find useful and is one of the things I miss from my C# days. Being able to treat simple derived values as if they are actual members of the struct. For example, being able to say rectangle.center as a shorthand for rectangle.topLeft + rectangle.size/2 would be mighty convenient. C# has the option to make “Properties” that have a getter and a setter that define what should happen if the derived member is read or written to. This has a host of downsides, since it can be easily abused to execute an arbitrary amount of code inside what seems to be a harmless read at the calling side. However, if there was some way that I could get a very restricted form of this functionality, I would be a lot happier for it.

gy_directions.h

This file serves as a standard way for me to talk about cardinal (or diagonal) directions in an enumeration like way. Because of the implicit assumptions about handedness and direction order, I feel like this is one of those things that would never make it into a standard library in a satisfactory way. I don’t want to have to specify if my coordinate system is right-handed or which axis is “up” all over the place. I want those sorts of assumptions to be baked into my library of tools and therefore uniform across the code-base. So that’s what this file does, it defines 4 enums: Dir2, Dir2Ex, Dir3 and Dir3Ex. The Dir2’s are for 2D directions (up, down, left, right) and Dir2’s are for 3D directions (up, down, left, right, forward, backward). The “Ex” variants are enumerations that allow for diagonal directions as well as cardinal ones. To be honest, the Dir3Ex enumeration rarely sees use but the Dir2Ex enumeration was widely used in Princess Castle Quest since we had some things that moved diagonally. All of these enumerations are designed to be bitwise or’d together so they can be used like flags.

You can download the completed file HERE.

Positive 1: enum_flags helps implicitly assign bitwise or-able enumeration values

In the old code we had to make sure each value of our enumeration got assigned a proper power of two since we intended each one to be a single bit in a larger field of flags. Jai has a feature called “enum_flags” which allows us to do the following:

Dir2 :: enum_flags u8
{
	Right; //+x ( 1,  0)
	Down;  //+y ( 0,  1)
	Left;  //-x (-1,  0)
	Up;    //-y ( 0, -1)
}

Positive 2: Enumerations can have a forced backing type

Given that enumerations like this can serve as the backbone for many different systems, having the option to specify the size (and signed-ness) of the integer that backs them is pretty useful. It makes serialization a bit less hairy because we get a guarantee that the enumeration won’t accidentally grow in size and push members of a struct following it down. However, the biggest win by far with this feature is the guarantee that the binary NOT operator will behave like we want. Doing something like ~Dir2.Right is guaranteed to give us 0xFE (not 0xFFFFFE or anything else). This is a big deal for bitwise or-able enumerations where we often want to perform the following operations

IsFlagSet(field, bit) -> ((field & bit) != 0)
FlagSet(fieldPntr, bit) -> *fieldPntr |= bit;
FlagUnset(fieldPntr, bit) -> *fieldPntr &= ~bit;
FlagSetTo(fieldPntr, bit, value) *fieldPntr = (*fieldPntr & ~bit) | (value ? bit : 0);

As a side note, I implemented these Flag macros in Jai using the #expand functions thinking they would be important for readability, but it turns out that the type safety and other guarantees the language gives us are enough to make doing the bitwise operations out in plain code feel pretty normal. So the following code feels pretty mundane:

if (directions & .Right)
{
	directions &= ~.Left;
	directions |= .Up;
}

Problem 1: non-pure enums are more obvious because of Type Metadata

This is more of my own problem but these enumerations in C would often include derived or “meta” values that were related to the enumeration but not actually a proper enumeration value. For example Dir2_Count is not supposed to be used as a “valid” value for the enumeration. In fact it directly “conflicts” with Dir2_Left since they both have a value of 4. And now that we have functions like enum_values_as_s64 and enum_names it seemed weird to include them as part of the actual enumeration. Serendipitously we can just pop them out of the enumeration and declare them as constants that have the enumeration type. Or for the _Count case we define them with a regular integer type

Dir2 :: enum_flags u8
{
	Right; //+x ( 1,  0)
	Down;  //+y ( 0,  1)
	Left;  //-x (-1,  0)
	Up;    //-y ( 0, -1)
}
Dir2_None:     Dir2 : 0x00;
Dir2_All:      Dir2 : (Dir2.Right|.Down|.Left|.Up);
Dir2_Count:    u8   : 4;

This makes it possible to do things like:

directionFlags := Dir2_None;
for 0..Dir2_Count-1
{
	side := cast(Dir2)(1 << it);
	if (somethingOnSide(side)) { directionFlags |= side; }
}

Problem 2: My GetDir#Str functions didn’t go away

I imagine this won’t be true for many of my enums but in this case, I opted to keep my hand-typed stringification functions rather than rely on the enum_names function that’s built into the language. This is largely due to the special case handling we do for stuff like .Left|.Right being “Horizontal” or handing Dir2_All like it’s a proper member of the enum. However, there’s also a problem of ergonomics of these enum functions. They are list providers but they don’t help me go from enum value to name in a direct way. If I wanted to use the enum functions to get a name from a enum value I’d have to do something like:

GetDir2Str :: (direction: Dir2) -> string
{
	names := enum_names(Dir2);
	for enum_values_as_s64(Dir2)
	{
		if (it == cast(s64)direction) { return names[it_index]; }
	}
	return UNKNOWN_ENUM_STR;
}

This is not terrible. It’s only a few lines. But a switch case statement feels a little more in-line with how I’d like this function to operate, since the compiler could decide to turn it into a jump table. The real solution here would be if I specifically could ask the compiler to generate a jump table for me. This isn’t important here, but for larger enumerations (like EntityType) this could be a very useful feature. I imagine this might already be possible, given some metaprogramming. I will have to revisit this when I get more familiar with the compile-time code generation features of the language.

Conclusion

Those are all the files I’ve fully converted so far. I will write some more blog posts as I get farther. As you may have noticed, gy_intrinsics.jai is somewhat finished (things like CosF32 are already being used in these files, but I have yet to alias all the “intrinsic” like functions that I use). That and gy_colors.h will probably be next

My Experience With Jai (Part 2)

So if we take a look at our dependency graph in the last post there’s a bunch of things we “should” implement next if we wanted to continue in a top-down order. Things like gy_assert.h, gy_debug.h, gy_intrinsics.h, etc. However, a lot of this stuff can be sorta stubbed out and side-stepped for the time being. And I think that’s for our benefit since it will allow us to get to the meat of the features we want to try converting to Jai really quickly. So next up on the chopping block:

gy_vectors.h

This file contains, as you might expect, vector structs like Vector2, Vector3, Vector4. Generally, since we were working with C in the past, this file tends to be a massive file of manually duplicated code to handle each vector type for each type of operation we’d like to perform with vectors. Moving into Jai, I had two main questions I wanted to answer:

1. Do Vectors already exist in the “standard library” of Jai and if so, do they do all the things we want them to do? If they exist, but don’t do everything we want, can we easily expand the functionality to do everything we want? Or do we need to wholesale replace those implementations with our own?

2. Can and should we try and do polymorphic programming for this kind of problem? At the end of the day, the number of vector variants we actually want is around 6. We want 2D, 3D, and 4D and then we want floating point and integer variants for all 3 of those. Is it worth doing all our programming in a polymorphic way to keep from having 6 of everything?

So we can tackle these questions one at a time. First, do Vector structs already exist. The answer is YES (sorta). In the Math module we have definitions for Vector2, Vector3 and Vector4 (as well as Quaternion and various Matrix sizes). However, on further inspection there is only a limited number of math operations defined for these structures so we at least need to add more functions for these vector implementations to be “complete” in our case. For example, there are dot_product and length functions for all the vector types but there are no functions to round the components of the vector easily (something I do often for UI code and text pixel alignment) or clamp the components between two sets of upper and lower values. Also missing are the integer based vector variants that we want (I call these Vector2i, Vector3i, and Vector4i). So at the very least we’ll need to completely implement those variants and add some functions that operate on the existing structs in common (to us) ways.

Second question: Should we write our structs and functions in a polymorphic way? I’ll admit that I didn’t fully investigate what this would look like before deciding against it. So it may be that my worries are not actually founded. However, the main things that dissuaded me are:

1. We have a very small set of cases that we actually want. And having properly named functions and types for each variant holds some value when we are debugging and looking at the call stack. Seeing that something is a “Vector2i” instead of a Vector{2, s32} might seem like a small distinction, but it makes a big difference when you are dealing with these types all day, every day.

2. We don’t actually need anything above 4 dimensions, nor below 2 dimensions. A lot of functions would degenerate in the 1D case and make implementation a bit noisy (i.e. checking for this edge case and erroring, or early returning). Also a lot of functions that could be generalized to an N-dimensional case are actually only ever used for 3D cases. For example, cross product can be generalized to more dimensions besides 3, but we literally would never use the other variants in our games.

3. Naming our functions specifically to 2D and 3D is often useful. For example Vec2PerpRight is a very common function I use that makes no sense in 3D or 4D. You could make somewhat similar functions that take a couple more parameters than the 2D case but we would basically never use these variants because they don’t make sense to be called “PerpRight.” They would be called something like “RotateAroundAxis” for 3D (and who knows what for 4D)

So we don’t want to do things entirely polymorphic but we do we want to try implementing some “generalizable” operations in a polymorphic way? When I first started writing the file I figured I would say yes to this question. After all, having one VecAdd function seems quite a bit better than 6 VecAdd variants. However, given the brevity of some of the syntax, I found that almost ALL of my functions could be collapsed to a single line and not lose much readability. So I opted to do everything in a bespoke way (mostly for the sake of debugging readability later).

So I went ahead and converted my gy_vector.h to gy_vector.jai. For the sake of brevity, I’m going to upload the result of that process HERE and then point to some of the interesting aspects instead of explaining the process of conversion from beginning to end.

Positive 1: Constant Value Definitions are awesome looking (Curly Bracket Initialization)

One of the things I like having for vectors are “Simple Value Definitions” (see lines 107-181). Stuff like Vec2_One or Vec2_Right. Jai has support for default initialization of a struct which solves half the problem (i.e. Vec2_Zero is probably going to be referenced very rarely). In C these value definitions took the form of #defines that are really just function calls to NewVec2.

#define Vec2_Zero      NewVec2( 0.0f,  0.0f)
#define Vec2_One       NewVec2( 1.0f,  1.0f)
#define Vec2_Half      NewVec2( 0.5f,  0.5f)
#define Vec2_Left      NewVec2(-1.0f,  0.0f)
#define Vec2_Right     NewVec2( 1.0f,  0.0f)
#define Vec2_Up        NewVec2( 0.0f, -1.0f)
#define Vec2_Down      NewVec2( 0.0f,  1.0f)

This meant that our value “definitions” were not actually compile time constants like we wanted them to be. However, the alternative was to do curly bracket initialization, but this came with a HOST of downsides because C’s bracket initialization is under-defined and not very strict.

In jai we have a few more promises from the implementation of this feature. Namely the curly bracket expression is strongly typed, with stuff like “v2.” before the open bracket forcing the expression to always be a v2. And the expression errors if the number of items passed does not match the number of items in the struct exactly. With those things in mind, we’re able to switch to curly bracket initialization for these definitions and reap the benefits of having truly constant values for our vector structures!

Vec2_Zero  :: v2.{ 0, 0 };
Vec2_One   :: v2.{ 1, 1 };
Vec2_Half  :: v2.{ 0.5, 0.5 };
Vec2_Right :: v2.{ 1, 0 };
Vec2_Left  :: v2.{-1, 0 };
Vec2_Down  :: v2.{ 0, 1 };
Vec2_Up    :: v2.{ 0,-1 };

Also, curly bracket initialization allows me to make many of my basic vector functions into single line implementations that are still relatively readable. For example:

Vec2Floor :: (vector: v2) -> v2 { return v2.{ floor(vector.x), floor(vector.y) }; }

Positive 2: I’m getting rid of _t suffix on my Types

For a long time now I’ve opted to add _t to the end of all my types in C. I forget exactly what the main impetus was for this but I suspect it has a lot to do with the fact that types are not treated the same as other constructs in C/C++. So normally my vectors would be defined like so:

union Vector2_t
{
	r32 values[2];
	struct { r32 x, y; };
	struct { r32 width, height; };
};
typedef Vector2_t v2;

So we’d put a _t on the end of the type name to make it clear it was a type. However, for really common mathematical types (like vectors) I would also make an super shorthand alias like “v2” (something I never did for higher level types like Color_t or Textbox_t for example).

With types being treated with more “first-classiness” in Jai, it seems like having this suffix is entirely redundant now, so I’ve opted to try getting rid of them and see where that takes me.

Problem 1: I can’t Augment Vector2, Vector3, and Vector4

If we look at one of our integer vector implementations, there is a feature that I like having that the Math module’s implementation does not have. And that is aliasing x, y, and z components as “width”, “height” and “depth.”

Vector2i :: struct
{
	x, y: s32;
	
	#place x;
	component: [2]s32 = ---;
	#place x;
	width: s32 = ---;
	#place y;
	height: s32 = ---;
}
v2i :: Vector2i;

This can be done by using the #place directive, which is great! However, as far as I can tell there is no way to do this after the structure has been defined. So if we want to actually use the Vector structures from the Math module (and it’s associated functions and operator overloads) we have to forego these aliases for floating point vectors. This is a common case of “If it doesn’t do everything I want out of the box, I can’t use it at all.” This isn’t a deal breaker for now, but I may change my mind later if there ends up being any other missing features that I haven’t noticed yet.

Problem 2: The Unary Negative Operator was missing

This is a small thing to fix, but worth mentioning. There is no definition for negating any of the Vector structs in the Math module, so we had to implement those ourselves.

Problem 3: Dead Code Elimination and Float32 is a “long” name

Since this is the first real file of substance that we’ve started bringing into Jai, there are 2 problems that aren’t Vector specific but are interesting to mention. The first one is that many of the mistakes I made would not get caught by the compiler until the function was actually being called by some external code. At first I solved this by writing a TestVectors function and #running it during compilation. I’m not a TDD proponent so this is very abnormal for me. It ended up wasting enough time that I had to take a brief break and go figure out how to make the compiler more strict. Turns out that the compiler does “Dead Code Elimination” by default, which was definitely my suspicion. There’s actually a standard command-line option -no_dce to turn this off that I had completely skipped over when reading the debug output of “jai -help”. Turning this elimination off definitely helped, but I feel like I still hit a few cases where my function would compile until I called it from somewhere. This might be anecdotal though, I’ll try and keep an example if I come across another instance of this happening. The other “problem” is that I distinctly feel the difference between a 3 letter type name of “s32” and a 7 letter type name for “float32.” I put problem in quotes because it’s entirely aesthetic and it’s not hard for me to fix as a user. But for now I simply noted my preference and continued on with my day.

My Experience With Jai (Part 1)

I recently got access to the beta of a new programming language called “Jai.” My goal is to try converting my existing set of C files that I collectively call “Pig Engine” over to Jai in order to get a sense for how the language works and also find places where my approach to the same problems may differ in Jai (hopefully in a positive way) compared to C/C++.

Along the way I figure it might be useful to keep a running dialogue of my thoughts on various features and how I approach learning and using the language. I don’t use this blog much, so I figure almost no one will read this, but I think it’ll still be a helpful practice for me.

part 1: Reading HOW_To’s

The language beta comes with a folder called “how_to” which has around 70 files worth of examples code and explanations for how the language works. So I figure before I start trying to write anything, I should at least read some of these files. Mostly starting from the beginning (jumping forward to particular sections out of order when I wanted to know something specific about the language) I read through about 15/66 files before switching to proper work in the language.

Although it’s a lot of reading, I found myself pleasently surprised at how many features of the language line up with my experience of the hard problems I encounter while making games.

As a simple example, the 013_enums.jai file explains the syntax for enums and one of the “new” features this language provides is a tag you can put on enums called "#specified.” The whole purpose of the tag is to force the programmer to specify every enum value explicitly (no implicit value assignment) and generally act as a callout that the exact values of this enum are being used for serialization and changing/shifting the values of existing entries is probably a bad idea unless you go change this serialization code. Serializing enums might sound like a random niche problem, but I actually have run into this many times. The choice of whether to serialize the integer value or the string name is a tricky one, particularly if your game goes through an early access period with user generated content that you need backwards compatibility for, while still actively developing the game and changing enums. Having this one small feature really goes a long way to making this particular type of problem recognized as a thing that should be considered. In a lot of ways it’s not much more than a comment with a small amount of compiler backing to enforce specified values, but the fact that it exists in the language means that it can become a standard idea with a commonly understood name, and I think that’s pretty valuable.

Part 2: Converting Gylib (Dependency Order)

Alright so now that we’ve got an understanding of the basic features that the language has, I started trying to convert my common library code into Jai. For dumb reasons, this library is called “Gylib” and it includes everything that I use in basically every project; from Vector structs and math to UTF-8 conversion functions. For a somewhat full list of features you can check out the block diagram for Gylib that’s on the Pig Engine page.

So if we want to “port” Pig Engine to Jai, I figured the best place to start would be this common library. The files are largely organized by “features” but they also have had a dependency order baked into them because C does not allow out of order compilation. And since they were all built in a single header style fashion, I was forced to make sure that they had a #include order that would work. So we’re going to start from the files that are near of the top of the dependency chain and work our way downwards. However, as we go along we may find that a file near the bottom of the chain can actually get split and merged with upper level files now since they used to harbor functions that would fit, logically, in another file. But because they were dependent on a item lower in the chain, they had to be separated and given a name. Here’s an incomplete diagram that roughly shows that dependency order of many of the files:

A good example of something that might get merged is gy_sorting.h. Because the concept of sorting is not uniquely owned by gy_variable_array.h, gy_bucket_array.h and gy_linked_list.h it generally has to get split into another file and then the specific applications of sorting on each data structure type has to get jammed into a very dependent file near the bottom (for example, gy_extras.h, which got missed in the diagram for time constraints but is literally just things that couldn’t fit in upper files because they depend on two or more items in an unsavory way)

So starting from the top we have gy_defines_check.h which mostly translates directly to our module.jai since that’s where we can define our module_parameters. Fortunately some of the defines we’d normally need to enforce ourselves (like WINDOWS_BUILD) are actually nicely handled by compiler constants. So the main things we have left are two defines

DEBUG_BUILD: Enables debug only macros like DebugAssert. This is often tied to whether we want the program to be compiled in an optimized way, but it’s really just “Am I compiling this for me to test right now or to distribute or transfer for someone else to use.” So it’s often nice to control this define ourselves, even if the compiler provides some way to detect if we are compiling for “release”

ASSERT_FAILURE_FUNC: This one is a bit interesting in jai. Essentially it’s nice to have a function run when an assertion fails BEFORE we hit the breakpoint/abort. This function needs to be implemented on a per-application basis but the Gylib library provides all the common Assert() #defines and it’s variants. So we need a way to tell it what function to call when an assertion fails. In C we used to just do extern AssertFailureFunc(…) in the gy_assert.h and that would force you to implement the function elsewhere. In Jai we can actually pass in the function as a properly typed variable

Here’s our module_parameters directive in module.jai

#module_parameters (DEBUG_BUILD := false, ASSERT_FAILURE_FUNC : (message: string) -> bool = null);

Besides that our module.jai #imports a couple common modules like “Basic” and “Math” and then #loads all of our gy_[whatever].jai files.

Next up on the chopping block is gy_std.h which (I hope) is mostly going to be unnecessary. This file mostly handles aliasing standard library functions with our own names (like memcpy becomes MyMemCopy) simply so we can reroute these standard library calls to our implementations or other stdlib implementations in certain contexts (web assembly for example). For the time being, we are going to trust that we can solve this problem later, given the tools that jai provides us when loading modules.

Next we have gy_basic_macros.h which has a bunch of #define function-like macros for common tasks we want function-like syntax for. For example ArrayCount(array) which is just an alias for sizeof(array)/sizeof(array[0]). Luckily arrays in jai come with length info so this is not needed. Also macros in general are actually just functions with the #expand directive attached, so many of the “macros” in this file are going to get bumped to gy_intrinsics.h because the distinction between the two is pointless now (we no longer have a preprocessor that acts like a second language. Macros ARE functions). So yay, another file down.

Next up, gy_types.h. This file mostly deals with #including stdint.h and defining my own naming convention for fixed size types like int32_t becomes i32, uint32_t becomes u32, float becomes r32, and so forth. Jai, luckily, has some built-in, fixed size, types with really similarly short names. Unfortunately i32 is actually s32 in Jai. And r32 is actually float32 (not very short). For now I think we’ll try using this naming convention and forego porting any of gy_types.h. If we want our own naming convention back, we can do a find and replace on our files later to convert from s32 to i32 and float32 to r32.

So that brings us to our first dependency level that has more than 1 file. I think that’s a good place to wrap up this blog post for now (running out of time to write this). In the next post we’ll take a look at vectors, rectangles, directions, and colors and see how these common type implementations map to Jai.