At this point I’m considering doing an exploratory fork of Iron in Rust. I’m doing some thinking and experimentation to see what it would take to get the engine written in Rust. I know that this is a big deal and a lot of work, but I think that it would be worth it if it could be done. I’ll be posting progress that I make on this front here.
So far I’ve done some experimentation with how we would load game code without having to re-compile the whole of Armory. I created a tutorial for doing plugins in Rust, which was a result of my experimentation. The setup would let you load shared libraries for game-specific code that gets dynamically linked to the Armory binary. The setup would allow full access to the engine’s API, so it would have essentially no less ability to interact with Armory than it would if it were compiled right into the engine.
Also the shared libraries would have no performance disadvantage; they would be running as native machine code just like engine. Mods could be implemented the same way.
Another interesting scripting option is dyon which actually doesn’t need a garbage collector, just like Rust. It looks like it infers variable lifetimes and allows you to specify them manually when necessary to eliminate the need for a GC. The language is also well integrated with Rust and written in Rust, so I think that it will be the first scripting language I look into integrating.
After doing a little bit of research, I found that Dyon, while a cool little language, isn’t really suited for a scripting language for Armory. It doesn’t have much structure or allow anything similar to OOP, not that OOP is necessary, but the language just doesn’t facilitate much abstraction so it’s not a good fit for binding the Armory API to it.
I also did some looking into PyO3 for doing the Python bindings. It looks like a very nice library that should make it easy to create Python bindings.
Initially I thought that these bindings might be able to be automatically generated, but now I’m pretty sure that that would not be possible; there is just too much stuff specific to each language that could be bound and no good way to automatically map Rust to Python or any another scripting language. Bindings would have to be manually kept up-to-date with the core, but there’s nothing that can be done about that, so it comes with the territory I guess.
Rust would create a world-of-hurt for you, because it’s quest for memory-safety without GC makes dynamic multi-user, multi-threaded, circular heap data-structures really hard… and 3d programs have tons of these between objects, physics, meshes, and spacial acceleration structures (like BVHs).
IMO, these less-known cottage languages (like Haxe, and Rust, and Dyon) are the problem, not the solution…
If you do really want to take on something as challenging as trying to port the so-far-experimental Armory3d to another language… IMO, you should make it more mainstream by moving to C++ or C#.
In both these langauges, you could pre-compile the engine to a library, and then just compile the game-code and run. Maybe it wouldn’t be as fast as the Haxe->JS->Krom target, but what you’d lose in speed here you’d make up for in source-debugging and language popularity.
A related possibility is to merge/port parts of Armory3d (like VXGI and the Blender scene loading) into Xenko C# and end up with a single project.
In the opposite direction, there is the option to go C++, similar to UE4, with with Armory3d logic blocks acting like blueprints. This would be a unified clean codebase of high-performance threaded C++. This would eliminate these Haxe-to-C memory sharing issues, and Haxe/JS/Krom single-threading issues. It would also eliminate GC pauses, but at the cost of a more challenging dynamic memory model. You could still use RAII and Smart Pointers for rust-like stack lifetime techniques where applicable. This would provide a higher performance WebGL target with multi-threaded WASM.
…or maybe just work on Armory3d so the Python exporter doesn’t barf on a moderate complexity scene, and worry about these bigger issues when they actually come up in an actual project.
The “fly in the ointment” of your proposal – and, frankly, also of the notion of re-doing the thing in Rust – is that you now have lost the cross-platform capability that Haxe (alone) brings.
Haxe is a transpiler. Given Haxe source-code, it produces source-code as its output which is appropriate for the specified target. This is the “secret sauce” that motivated the original decision to do Armory3D in Haxe and not in something else.
IMO Rust is a seriously difficult road to go down. Their community responds to requests for solutions to dynamic cyclic heap structures by telling people they shouldn’t need that. I don’t understand how one would built a performant multi-threaded 3d game engine in Rust without defeating Rust’s memory safety. Maybe I’m just not smart enough.
A data-structure which is simultaneously accessed and changed by independent threads running for independent purposes. Rust’s memory safety model really doesn’t like this, especially when the data-structures are mutable and cyclic.
Most programs with big messy dynamic cyclic heap data ultimately use phase based memory safety. For example, a big threaded job like physics simulation or rendering might need to traverse your data-structures, and it knows this is safe because during that phase you have a convention not to deallocate things. In C++ you mark them for deallocation in the proper phase (or you use threaded mutexes). In C# you could just deallocate them and the GC will take care of letting them live until everyone lets go of them.
As far as I can tell, Rust stops you from using this pattern entirely. If someone thinks a component based 3d game engine is possible in Rust, let’s talk about the details. I’d like to be educated.
However, I also think the notion that Haxe’s transpiling is buying you something critically important is a mis-prioritization. You can hit a multi-threaded WASM browser target with virtually any language, and certainly with C/C++, C#, and Rust.
Cost #1 - nobody knows Haxe. Creating an application in a language nobody knows is fine… but creating a piece of infrastructure designed for a language nobody knows is kind of a problem.
Cost #2 - The Haxe “any target” transpiling is a jello soup of different abstractions. How do you share data efficiently between Haxe and native modules when you have 2-3 different ways native modules integrate? How do you source debug your actual code through multiple transpiling steps?
Cost #3 - Leveraging existing code becomes a huge problem. In C, C++, or C#, there is lots of code out there to do great things (physics, mesh loading, mesh simplification, space partitioning, etc)… and if the engine was written in any of those languages it would be really easy to integrate that code.
Net Cost - IMO the “Haxe many targets” model is never going to yield efficient, performant, highly-leveraged solutions to heavy lifting 3d problems. You could solve some problems if you standardized on a target like HL/C or hxcpp, and then used WASM to hit the web, but then you’re giving up the Haxe->JS transpile anyway.
I’m not saying it’s terrible to use the “Haxe many targets” model, I’m just saying we shouldn’t hold it up as some holy grail without costs. It has major costs.
So when that’s a multi-user data structure, then what’s that multi-user multithreading data structure you talked about?
Anyway, Rust is actually excellent for multithreaded applications - after all that is what it was built for, it’s the reason Firefox components are ported to Rust and it’s also the reason AAA game developers are moving to Rust (like Ready at Dawn and Embark). It’s the first language since C++ that’s used for AAA game development and that’s quite a feat.
Multi-threading = when two different threads of the same code need to see the data at the same time. For example, multi-threaded physics.
Multi-user = multi-component… Meaning two different parts of the code base need access to data-structures at the same time. Such as having a mesh and BVH representation that is traversal by both rendering and physics.
Mult-user multi-threaded is when both of those things are happening.
I’ve heard of lots of pieces written in Rust, where someone else owns the data, but I have yet to see an engine written in Rust, where Rust owns the data. I’ll have to poke around and see what’s out there now.
Multithreading and your definition of multi-user still sound like exactly the same thing
You will find plenty about Rust game engine development. The problems you’re trying to make up don’t exist. Rust allows free pointer usage when needed.
I’m sure I’m under-educated about both Rust and game-engine design, so I’d like shift to a higher level point.
It’s much easier for an application to use less known language than a framework. An application only needs one team has to be on board with the language choice. However, a framework needs an entire community of developers to be on board.
Armory3d isn’t finished. It might be moving faster if it had more developers making commits. Very few developers know Haxe. Very few developers know Rust. Lots of developers know C++ and C#.
That’s really my main point. That Armory3d’s biggest problem is that it’s not finished enough to work, and I doubt Rust would help fix that.
This Rust gamedev article is excellent and interesting. Worth the long read.
You might not be able to use pointers safely since certainly if you used pointers your entire game state would have internal pointers to itself, so […] everywhere where there would have been a pointer, [we] instead store an index into some array.
Okay… that’s one way to look at the problem. I still worry about cyclic dependencies, like the Observer pattern in Rust. However, I’ve never coded in Rust. Maybe this is a non-problem. There are certainly things to like about it, and I would have an easier time with Armory3d in Rust than this Haxe-many-targets situation.
Though even better might be pushing the neat direct Blender-asset-handling into a generally useful C++ or C# library and integrating it (and VXGI) into a larger game engine community like Xenko, Unity, or UE4.
I don’t think that’s a valid point either, Rust is gaining users in game development so quickly, I’ve never seen that before with any other language. Or just have a look at the recent Stack Overflow survey. “Very few developers know Rust” isn’t true anymore. (I’d argue that it also isn’t a big issue with Haxe because while it is indeed known by only a few people, it’s very close to C#/Java).
But there’s also the less success metrics focused issue that C# (or Java) would drive me away because I really don’t like it and integration into another engine would probably drive Lubos away, which would effectively kill the project. Doing something that the main developers are interested in is very much the most important issue (which is also the reason that discussions like this actually have no point without Lubos chiming in).
So to sum the important things up: I’m totally in favor of Rust - if I’d create a game engine today myself, I’d use a mixture of C and Rust. zicklag is in favor of Rust and he’s an important contributor as well. No idea about Lubos and no idea how much work such a switch would be - which is another critical factor, probably the most critical.
“[Re-]doing it in Rust” is tantamount to “let’s start over from scratch.” And I simply do not believe that this avenue is justifiable.
“There’s always another language.” And, there’s always folks who think that the solution to any “problem” is to start over. But from an engineering point-of-view, I submit that this is simply not credible. Armory is built on top of several other components which all exist in the same (Haxe) ecosystem, and I am entirely sure that the original designer thought very-long and very-hard (and, no doubt, considered Rust …) in making this decision.
You could, of course, say … “fork it, I’m going.” But recognize that you’d be on your own – that what you’d be doing, from scratch, would not be “Armory.” It would be yours alone. And, I would simply counsel that I don’t see engineering viability here – return on investment – adequate coverage of business risks (which Haxe now covers) – etcetera.
“Engineer to engineer,” all I can say is this: that, if it were me (and, of course, it isn’t …) I would not proceed on this course. If you disagree, sail on with my blessings – “prove me wrong … it’s been done before.”
One point I do want to bring up with C++ is that it is an obstacle to many people, like with the UPBGE project, where the main developer is leaving and people are desperately calling “Does anybody know how to program in C++ properly without adding more bugs and performance issues ?”.
You can learn to start coding with Rust in a month, without half of the risks of introducing hard-to-track bugs and undefined behavior. Even if you are not an expert you can start learning and contributing to the project. In a community project, especially one filled with beginners, coming by the C++ experts necessary to contribute meaningfully to the engine is difficult. I’ve also been a part of a couple of other game communities where everyone groans and moans about C++, not that it is a bad language, but it does produce a lot of pain for the coders, especially beginners.
The difficulty of C++ isn’t just for beginners, either. In a blog post by PingCap they said:
The core team members are experienced C++ developer with rich experience in large C++ projects. But the seemingly inevitable problems in large projects like Dangling pointers, memory leak, or data race make them shudder at the thought [of writing a database in C++].
C/C++ used to be a necessary language that filled a use-case that no other language could. Just like we found that Haxe can’t do the manual shared memory that is necessary to support a CPU cloth solver and C++ would solve for that, but with Rust, now, more people can write programs that are just as efficient as C++, without half as much experience. Not that I’m downplaying experience; there is no replacement for experience, but dealing with something that is more difficult than necessary doesn’t help anybody if there is no reason for it.
That’s just my perspective on the C++ instead of Rust deal. I don’t have anything against C++ or using it, but that doesn’t mean that I would ever use it if I didn’t have to. Given a choice, I think Rust is a better route.
I completely agree. I have tried to involve @lubos in this, but he can be hard to get in contact with. I think this is an important subject though that is critical to the future of Armory, or at the very least critical to my future involvement in the project. Its not like I’m saying, “Guys if you don’t write this thing in Rust I’m forking it”, but what is of critical importance to me and my team is whether or not Armory is going to be able to scale to the point of competing with engines like Unreal or Unity. It has to do with the vision of Armory.
At this point, I actually don’t know what the vision of Armory is. Only @lubos can tell us that. Why is he making Armory. What does he want to accomplish. This is maybe themost important component of any project. What is the vision?
If Armory is not event meant to compete in performance with big name game engines, then this is irrelevant. Haxe is sufficient and going through the undeniable pain that a rewrite, even a partial one, would entail would be completely against the vision of Armory and would not support its direction.
On the other hand, if Armory does intend to compete with other major engines, like many enthusiasts on this forum believe that it could, then I don’t believe that it can while being written in Haxe. It will impose a limitation that will be impossible to overcome within the limits of that language. ( And I really like Haxe so this is not me bashing on that language, I heavily considered whether there was any other language target that we could add to Haxe that would make this a non-issue, but I could come up with nothing short of removing Haxe’s dependency on a garbage collector, which is impossible. )
When it comes to Armory, I have to consider what is the best decision for my team and the values that we have as an organization. Having a desire to make large scale games in the future, it would not be wise for me and my team to invest years of development in an engine that will not be able to scale with our organization.
I love this community and I have spent many, many hours investing in it over a period of almost a year, but I must make decisions that support my vision, just as Armory must make decisions that support its vision. Defining what Armory’s vision is is paramount for the project.
I understand that. If we had to make something new that wasn’t Armory, so be it. What would not be a return on investment would be investing years into an engine that fails us when we get the game made.
Summary: This needs @lubos’s input. Without it and an understanding of where Armory is going and what the vision is, all of this is pointless. If we cannot get @lubos’s input on this, then I can’t really bet on Armory for my games, above small demos and POCs. That doesn’t mean that it won’t be 100% suitable for other people’s use-cases, but me and my team’s vision goes beyond having the limits that Armory has today. That was the purpose of my experimentation and investigation for Rust in Armory, to remove the limits that are showstoppers for using Armory in my team.
Good point. It’s possible the vision is really close to BGE, which would be about users who are only going to author in Blender, with a few logic blocks and some Haxe script code. These users don’t need to integrate native code libraries.
Maybe down this path Armory3d is never planning to support larger terrain meshes or mesh collision shapes. It would be nice to have these goals and limitations explained somewhere so developers know when Armory3d can handle their project or not.
This is a very good point. Even if Rust has some challenges, it’s hard to argue that would be worse than C++'s endless failings, especially without the discipline of a coordinated team.
My reluctance comes because I don’t want to give up dynamic cyclic structures, and I don’t understand how they fit into Rust’s memory safety model. However, I’ll defer to others more knowledgeable. If it’s possible to do everything, even if sometimes unsafely, at least it wouldn’t be the morass that is C++. We can all cheers to that. I’m willing to learn new tricks.
I’ll also admit that even if Rust requires challenging new memory-ownership architectures - if those decisions can be made by core-architects of a project, and make it easier and safer for other developers, that sounds like a good tradeoff.
At this point, I doubt that @lubos is going to want to support any sort of rewrite of Armory, and if he doesn’t get on here then that is all I can assume.
If I were to pursue a game engine in Rust, instead of rewriting Armory I would more likely build an engine that is built on top of an ECS, like Amethyst is, or try to work the engine in with Amethyst and provide a Blender integration, logic node system, and maybe a renderer/shaders that are similar to or partially ported from Armory. Right now I would still very much prefer to build on top of Kinc as it is such a solid platform abstraction layer, and it goes to consoles, which is a big deal to me and my team.
That said, it probably isn’t useful to the Armory community to continue a discussion of how to build my own game engine that is not Armory. Still, thanks all for the productive discussion of the topic. I’m very glad to get everybody’s perspective.