Wrangling with external data

Hi there!

There’s lots of possibilities, depending on your preferred target - Would you prefer csv or json? Which kind of data from which kind of solution? There’s libraries for any kind of possible scenario, whether it be CSV (https://lib.haxe.org/p/csv/) or JSON (https://api.haxe.org/haxe/Json.html), or if it’s browser-based, there’s also javascript solutions where you can make fancy graphs on top of your Armory layer in html - I’d be happy to try and cook something up if you need anything specific?

1 Like

I would go for Json, it’s organized in structures, easy to read.
We should be able to create structures in logic nodes, and call a “save to JSON” logic function to write the data structure on a file.

thank you both ! Ok I ll start playing with json first since I agree that it is more reliable and all the hard stuff will be done in pandas or wherever else so to be quick and easy.

@Naxela 1. I am looking to put in my scene data driven geometry (just scaling and generally transformation stuff - I wouldnt say no to a lower level but this would be really ambiguous for now).
2. Is it possible to mix JS with Haxe (or any other language for scripting) ?

@MagicLord For now it will be sufficient just to import a ready made JSON file instead of saving - creating one in armory. although some times for such tasks textual programming is better for me than visual (node).

thanks again !

Why do you want js for scripting? Haxe itself isn’t that hard. But well, it is possible.

@cnisidis , as Naxela write it depends of your target and as I’m not sure you read the post Read external data in detail… Why this post is among other things because Haxe targets in JS cannot directly access the file system.

Thus if it’s can help, a warning to consider for your future choices.

(The solution explained is what I’m using to store and read trained Neural Networks with Html target).

@Didier

I see what you mean, I read it indeed many times, two things to mention here:

  1. in your example and the very nice node implementation you have an http request and you are addressing to a specific port (8040) instead of 80 or 8080 - why is that ? is it reserved , is it a standard or did you chose this specific port to communicate with your server where you host your file ?
  2. I am looking rather for a textual implementation since I am planning obviously to load a bit complicated and big json file in my init and then retrieve data from it on demand in my update (?).

I ll try of course to just tweak what you have done so far and fit it in my needs, if I encounter anything or if I have a decent result I ll come back with my findings.

PS shall I include external libs or packages in order to use http and other url (server side) related stuff ?

I thought that it would be easier if not obligatory to deal with external files in Haxe with the ease of other tools and libs.

@cnisidis

  1. up to you to make a choice … see your firewall config
  2. what do you mean ?

no other library for reading/writing files

ah its ok I got it, so you created a blender read file node and then you got its input to do the rest and yield the data back to your scope. There is nothing similar to your json (read file) in A3D right ?

As Didier mentioned, there’s no direct way to access the filesystem from JS targets, although for HTML5 browser targets there’s the Filesystem API, but not all browsers support it. If you’re using Krom there’s backend abstractions (through the various Kha LoaderImpl classes, if I understood the code correctly).

In any case, here’s a small example I cooked up for the Krom target (which is what I suppose what most will use anyway). In my example, in the project folder, I’ve a folder called “Bundled”. In that folder, put whatever data file you will need - in my case, I’ve a file called “dataFile.json” - It looks like this:

/Bundled/dataFile.json

{
	"mars":"Hello from Mars in the JSON file!"
}

In your blender project, setup a haxe file (just one trait attached to whatever, scene, object, etc.). In my example, I just made one called loadJSON:

/Sources/arm/loadJSON.hx

package arm;

class LoadJSON extends iron.Trait {

	public function new() {
		super();

		 notifyOnInit(function() {

			iron.data.Data.getBlob('dataFile.json', function(blob:kha.Blob) {

				var martianData:{mars:String} = haxe.Json.parse(blob.toString());
				trace(martianData.mars);

			});


		 });

	}
}

This basically first loads the file (backend through Krom.getBlob), and when it’s loaded the json file is parsed as per normal through the haxe.Json.parse method - Notice though, that I’ve declared which type of data I’m loading, as otherwise it will default to a dynamic type (which will give you can “Object object” return value - you can use Reflect in that case, but I find it messy). I can probably upload the zipped example folder somewhere if anyone wants it, but there’s really not much to it.

Hope it helps :slight_smile:

1 Like

It worked like a charm, thank you so much, so “Bundled” folder is like “Libraries” (a reserved name to store certain things ? ) where can find more info about the structure of the folders and how to use them properly in my code ?

Great to hear :slight_smile: - Yeah, “Bundled”, “Libraries”, “Sources” and possibly(?) "Shaders, “Subprojects” are some of the stuff that is included, I think most of them is listed here: https://github.com/armory3d/armory/blob/21427d519e547bf3d4cad3a56faa269beda59712/blender/arm/write_data.py - The code there sort of deals with the whole asset structure and how some folders are loaded, etc.

but as you mentioned, even if I use Krom to build my app I get this error in FFox

Error loading this URI: Could not load the source for blob:http://localhost/6a122ab1-bf65-4329-985c-cece7136f7f6

meaning probably that it is restricted even if I use xampp as local server.

I ll dig more on that and I ll get back soon

Yeah, for that you need to use different measures for getting the data - I think the best solution for browsers is something like what Didier made, namely where you use an HTTP request to acquire a resource.

Another example is provided by Lubos here - to retrieve an OBJ file, but it can be used to get JSON data from a file too: https://github.com/armory3d/armory_examples/blob/master/server_stream/Sources/arm/LoadTrait.hx#L27

1 Like

yes intuitively I imagined that, so now I am trying to fit his solution to yours. In @Didier 's case the http request is a part of a custom class with an override function where he does the http request and he doesnt also need to use Blob (because of http I guess). Anyhow, I have to understand better how the functions are being called in the main class and why he needs an override (probably because this is an independent “node” class) .

Thanks again for your time and your patience !

I am back needing a clarification, I took some decisions and one of them is to go with http request and json, so I use one local server on port 80 (xampp’s default) and compiling my Krom (html hosted on 8040) so it asks from Appache to pass a json (generated with php). This is not going well since this CORS thing restricting the access, so I moved the php file on my web server, but even there I noticed that I still get the same error (http #0) meaning that it can’t access the file. Do you have any idea why this is happening since I am using distant server ? Is there any way to change it before I tweak my .htaccess headers file ?

Ok I solved it, it was just stupid from my behalf to not load it from xampp directly (putted my php file into bundled folder) , I try to outreach other servers too.

hm ok, I still cant reach other servers, but it doesnt matter for now, if this can be solved in future I would be interested.

The reason you can’t reach other servers, is it because of the CORS issue?
If so, have you tried adding the following to the top of your PHP file:

header("Access-Control-Allow-Origin: *"); //Try this first, should mostly work by itself
header("Access-Control-Allow-Headers: *"); //Might make it work, if the first line doesn't solve it

Personally, I’ve no issue with it, but I suppose most server setups are different - and I take it could also have something to say what server software your server is setup with (apache, nginx, etc.)

1 Like

Nice ! yes it worked with this one too (without include it in my bundle directly from distant server) thx !