Have you already made some experiences with IA ( like with Python for Artificial Intelligence ; ref here https://wiki.python.org/moin/PythonForArtificialIntelligence )
for example and an implementation in a scene in Armory 3d/ Blender ?
1 Like
On my side, what would interest me the most would be to look at a neural network and its learning into a 3D environment … before exporting it to a real environment.
1 Like
This video could give you an idea of the great possibilities that your games with Armory could gain through the use of some deep learning technics
MarI/O is a program made of neural networks and genetic algorithms that kicks butt at Super Mario World
2 Likes
This could give you another ideas where a real-time character control mechanism is using a novel neural network architecture called a Phase-Functioned Neural Network. In this network structure, the weights are computed via a cyclic function which uses the phase as an input.
http://theorangeduck.com/page/phase-functioned-neural-networks-character-control
2 Likes
You will find in the excellent and sympathic blog of Greg, expert in game development,
the info on the Dense Human Pose Estimation In The Wild project… or how to use a new model of Deep Learning to estimate the 3D pose of multiple people in a video, using a simple GPU.
We can think on how to reuse this with an integration into our 3D animations in Armory, with a specific editor for Estimation de Poses 3D.
For example, you train your neural network with the different videos you created through Armory with your characters, into different situations that may occure then in you game.
Then you obtain an incredible coordinate prediction, by mapping each pixel of a detected set for your 3D model model in Armory.
(the source code and Database are open-source and available on Github; If you want to test, I advise you to install/use Anaconda and thus create a specific/isolated environment with all the necessary packages and Python version)
It could be a nice way to start with the IA inside Armory, with specialized nodes to make it easily … interested ?
2 Likes
C’est parti !
This is the preliminary design for a Neural Network to use for Armory games or other simulations…
3 Likes
How you create it ? This nobe is made from you ? https://youtu.be/vppFvq2quQ0 something like this will be very usefull
@Chris.k with Armory, you can create your own nodes … if not already done, first look at the doc here http://armory3d.org/manual/#/dev/logicnodes
So I am testing those new nodes that you can see in this picture, and I create them with the Haxe langage used for Nodes in Armory. As you can see, the NNFactory (Network Nodes Factory) is a central node, that I use to define different parameters that will allow then to simplify the test of different kinds of NN architectures that fit better according to what we want to do.
The link you give is very demonstrative and gives a good idea of fun things that can be done with the NN in 3D … like throwing cubes to your trained model 
2 Likes
I dont know how to code .so can you give me download link for nobe and tutorial will be helpfull.Btw awsome work.Ps if you see armory github paje you will see that krom support tensorflow.
You can start here with coding in Haxe https://code.haxe.org/
For info, NNFactory is made all in Haxe with no dependencies with others libraries in the NN domain and with the idea to have something very specific to Armory capabilities for example to develop new systems equiped with IA and have a tool for tests and explorations of new concepts.
2 Likes
Amazing progress. Yes there will be tensorflow support, but having all-in haxe library with nodes would be super cool.
1 Like
First training & test for a Neural Network in Armory : training output at 3100th step with 100% success rate of predictions on the learning set. Then 100% of the samples with defects are detected. Now the distribution of the NN haxe code into the Armory nodes begins.
The speed of learning if very good too with env 1 sec/1000 iterations.
5 Likes
That sounds absolutely awesome! I hope you will share your work with the community .
@donalffons Sure, this is already the case. And new players to prepare 3D experimentations, inspired by those showned before in this thread, are welcome !
1 Like
This nodes architecture called NNA (for Neural Network Armory), is now operationnal :
- on the left we train the neural network with a training set coming from a Json file
- on the right we ask a question to the neural network (again the question is inside a json file)
Next step will be to use data coming from objects in the Amory scene … see preceeding examples.
4 Likes
This looks highly promising, good work!
Which network types besides Perceptron are currently supported?
1 Like
@NothanUmber The Perceptron is the current one with a sigmoid function. Feed-Forward and Backpropagation are available inside the Node Factory for training and Feed-Forward inside the Node Activation to maintain high performance for real-time applications.
Armory Nodes could then offer the possibility of testing different networks and architectures, such as coupling several different networks.
3 Likes
What we could be inspired for NNA tests … how to propose to game developers in Armory a method (1) that will learns a 3D human pose without any 3D datasets or (2) to be able of reconstructing a full 3D mesh of a human body from a single RGB image.
see here an example for (1): “Our method can predict a 3D pose from 2D joint locations in a single image. Our system is based on the generative adversarial networks, and the networks are trained in an unsupervised manner. Our primary idea is that, if the network can predict a 3D human pose correctly, the 3D pose that is projected onto a 2D plane should not collapse even if it is rotated perpendicularly. We evaluated the performance of our method using Human3.6M and the MPII dataset and showed that our network can predict a 3D pose well even if the 3D dataset is not available during training.”
https://nico-opendata.jp/en/casestudy/3dpose_gan/index.html
3 Likes
Interesting! Extracting 3d meshes from rgb images would be a more difficult task than the one they solve in this paper though. (they are “just” guessing the z coordinate from an already available 2d mesh). My guess is that you will need some kind of CNN at least for the generator network instead of just MLPs. (Not sure why they use MLPs in the paper instead of CNNs - doesn’t this make the task more difficult as the networks first have to learn how to decode the spatial structure of the 2d mesh from the linearized input vector?)
In which scenario would you want to use the 3d-mesh from rgb image network? Wouldn’t the 3d mesh infos of all objects in an Armory3d game already be available to the game AI? Or do you want to build something that plays 3rd party games just from screenshots? (Like the Atari games playing RL agents from Deepmind)
Edit: Ah, you want to build some motion capturing software that just works with 2d images from a single camera?
Edit2: Ah, the approach in the second paper looks more like it
@NothanUmber Thanks for your comments.
In short, I’m trying to discover if there can exist and develop a kind of ecosystem of enthusiasts around the use of NNA and what could be the most expected uses (cf. the examples we find of what is done elsewhere with NN and 3d-mesh from rgb image is one of them that could interest some of us to develop)?
First, on my side, I test a tracking that could then be portable on a machine equipped with a camera. Taking care of RealTime constraints is determinant. One of the advantages here is to be able to quickly generate object views via 3D for NN training without having to go through labeled or unlabeled web image libraries.
Another is to be able to do the training on a 3D modeled environment, which takes into account as many difficulties as possible before going on the field.
In a second time, it will be the opportunity to test various NN architectures thanks to the ease offered by the use of nodes. See an example of a concept on this diagram.
(source https://einstein.ai/research/domain-specific-language-for-automated-rnn-architecture-search)
2 Likes