Project Update

We are so back

What happened? Where was I? How's it going

A couple months ago, I decided that I wanted a more personalized branding for the blog, which led me to hire my friend, Mikaël Neves, to design my website and blog. Knowing that I'd integrate the design , I decided to put the development and blog posts on hold while I took care of implementing everything

Fast-forward 8 months, I have about 5% of the website done, cause I forgot how much I hate doing frontend development.

So, here we are. We're back on development, and we're back to doing blog posts on this platform. I've hired a friend to do the website this time so, we'll actually have the website eventually.

I've renamed all existing weekly updates to 0.#, and we'll go from there.

What's going to happen with the project

From now on, no more weekly . Blog updates will occur when I've done substantial changes to the code base, improved performance, or added fun/challenging mechanics. Basically, when there's actual progress made, there will be an update, otherwise I'll probably have nothing to say.

Anyway, on to the actual updates.

Project Update ( no, like, actually )

Project Decisions

Back in update 0.6, I mentioned that I was switching to C++ for the game server, given the fact that I couldn't get the Steamworks.NET version on the client, to work with the server using the open-source version.

A couple weeks ago, I got back into the Go server, and spent the time investigating and looking for alternatives, as I really wanted to stick with Go. In the end, I managed to find a C# open-source version of Valve's GNS. Things worked, I could connect to the server without problems.

And then, I realized that I did not want to do the client side in C#. There's just something about C# that irks me, and I can't quite put my finger on it. So, I'll probably switch back to Unreal Engine for the client, and use the open source version for it.

On to the game server updates.

Game server updates

I spent most of the last days working on performance optimization on the networking.

There were a couple problems with the code to begin with. On every iteration of the ,

• I was creating a fairly large slice, one that was responsible for storing the incoming messages

• I was creating a new decoder, instead of resetting it and giving it new data

• I wasn't sleeping long enough in-between iterations

I fixed those problems the following way:

• I moved the large slice outside of the loop, and reset elements to nil at the end of the loop. I also created another slice of the exact size of the amount of messages read, allowing for more optimal looping. A more detailed explanation below

• Unbeknownst to me, there was a method on the decoder called "ResetBytes", which resets the decoder and prepares it for a new series of incoming bytes.

• I added a dynamic sleep. Every time we receive 0 messages, I increment a counter which lets the loop sleep for longer, up to a maximum amount. This reduces the CPU usage % when we're idling. Preferably, we'd be able to block until there's a message, but the underlying networking library doesn't work that way.

Explanation of the aforementioned fix for the large slice and performance

The way that the network thread works is that before we start looping, we create a slice of 13000 messages, as such

messagesPtr := make([]*gns.Message, 13000)

That slice has that specific size because we expect to receive about 10k packets per tick. We then calculate about a 30% burst of activity, which results in our 13000. We would then iterate over that messagesPtr, decode them to the GameMessage interface, and pass it along a channel for the game loop thread to process those messages.

But what happens if we didn't receive 10 000 messages? What if we only received 1500? Or 5000? We're gonna iterate the whole size ( 13000 ) of the slice every loop? That is, in my opinion, a waste of time and performance.

When we read the messages into the messagesPtr, like so

numMessages := s.PollGroup.ReceiveMessages(messagesPtr)

we get the amount of messages that were read. So every iteration, instead of looping over messagesPtr 13000 times, we create a new slice, the exact size of the messages received. We then iterate using that size, which is more optimal.

msgs := make([]gamemessages.GameMessage, numMessages)
for i := 0; i < numMessages; i++ {
	msgPtr := messagesPtr[i]
	
	if decoder == nil {
		decoder = codec.NewDecoderBytes(msgPtr.Payload(), &s.GameManager.MsgPackHandler)
	} else {
		decoder.ResetBytes(msgPtr.Payload())
	}
	
	err := decoder.Decode(&msgs[i])
	if err != nil {
		sentry.CaptureMessage("An error occured decoding a packet")
		return false, fmt.Errorf("failed to decode package. %w", err)
	}
	
	msgs[i].MsgNumber = msgPtr.MessageNumber()
	msgs[i].UserData = msgPtr.UserData()
	msgs[i].Connection = msgPtr.Conn()
	msgs[i].ReceivedAt = msgPtr.Timestamp()
	msgs[i].Flags = msgPtr.Flags()
	
	msgPtr.Release()
}

Conclusion of the game server updates

One thing that really bothered me is that when a lot ( I'm talking hundreds of thousands ) of messages came in and were being read by the network loop, the CPU usage would skyrocket to 115-195%, according to my docker container stats.

At that point, I had already implemented my optimization changes, which did improve performance by a small margin, and I was out of ideas on how to reduce CPU usage when doing heavy work.

Enter the Gopher Discord

I decided to join the Discord Gophers' Discord, in hopes of getting some help.

This is where I met _dl and YC, which helped me benchmark my application by showing me how to profile my application's performance

They taught me that my application's high CPU usage was due to my usage of the Go bindings library I use to interface with Valve's GNS.

Turns out that using Cgo adds a lot of overhead, and considering that I do a lot of network polling, I'm calling C functions many, many, many times.

They reassured me that while the CPU usage is high, it shouldn't be too much of a problem, considering I'm handling 1000 players, all of them sending a good amount of packets per tick, with approximately 2 CPU threads.

Conclusion

I love Go. Go works for me, it lets me iterate quickly, I do not need to work with CMake, concurrency is simple.

A big thanks to _dl and YC for their help, their time, their patience, and for reassuring me that I didn't waste hours of my life working on the Go server.

I feel like I've made good progress on the game server in the past couple days, and I'd say for now, I'm done working on the network thread.

Next step, improving the game loop thread, delaying non-important packets ( like chat messages or social ) to the next tick when we're experiencing big bursts of activity.

Should have an update in the coming days, although a reminder that weekly updates are gone, and I'll only release blog posts when I feel like I've made significant progress.