Godot Dev: Custom Type Overhaul

willnationsdev
Posts: 7
Joined: Sat May 14, 2016 6:08 pm
Location: Waco, TX
Area of Focus: Programming, Design
Occupation: Web Developer at Baylor University
Contact:

Godot Dev: Custom Type Overhaul

Postby willnationsdev » Fri Mar 30, 2018 10:50 pm

I guess I'll be using this as an area to post updates on my work with Godot Engine. Progress: ~35%

As some might know, Godot provides users with "nodes" that can form "trees". Users can then save assets for game content by either extending a specific node through a script or by saving a user-defined node tree as a scene. Scenes can be both nested into each other and inherited by each other.

Problem 1: In order to load and reference any of these scripts/scenes, you have to provide a filepath (relative or absolute from project root). This makes it extremely cumbersome reference these resources.

Problem 2: In order to create new "types" of nodes in the engine, people can define "custom types" in a plugin, but all these do is initialize the creation of a node with a script and assign a custom icon to it in the editor (it's extremely barebones). You can even remove the script entirely and thereby undo its "custom"-ness. This is hardly ideal.

Problem 3: In order for users to solve Problem 1, plugins need to be able to assign explicit names to custom types, names that WON'T conflict with names defined by other plugins. Therefore a namespace system is required to ensure that equivalent names from different sources can be resolved safely by scripts.

Problem 4: Engine nodes written in C++ are able to efficiently create an entire node tree to represent a single node in the editor. To do the same in scripts would be comparatively slow (since adding sub-nodes and initializing their properties would require several calls to the engine's scripting API). The fastest way to do this is to create scenes and instance them, effectively offloading the node tree creation/initialization to the C++ side (so, batching all of those scripting API calls into a single call). However, people can only define "custom types" that are scripts; telling the engine to think of an entire scene of nodes as a single node type is not possible.

Problem 5: Even after all if you find a way to change all of these infrastructure problems, you still have to make sure the new backend that enables these features / solves these problems is backwards compatible / is integrated into the existing Editor UI. This involves making changes to the GDScript compiler so that it can access the global map of typenames. The Editor's dialog for creating nodes and resources both each need to be updated to no longer use a single, initially-loaded, static list of C++ types, but to instead use multiple, constantly-reloaded, dynamic lists of both C++ types and a variety of scripts and scenes of different types. To enforce that custom types' scripted inheritance hierarchies are preserved, even if you attempt to remove the script, you have to perform type-checking in advance at all of the locations where users COULD remove or alter the script (there are several areas), and you must update the UI accordingly for every action.

As such, I am currently at work creating a dynamically updated type database (TypeDB) in C++ for Godot Engine's editor. With it, various changes can be made...

Solution 1: Automatically pick up compatible scripts and scenes in the file system when they are created/moved/modified and map their filepath to a name. Derive the name from the name of the file and the file type for optimum backwards compatibility ("my_script.gd" -> "MyScript", "my_scene.tscn" -> "MySceneScn"). Make these resources accessible from a dynamically-updated globally-accessible script. It's source code will be generated from the TypeDB whenever the user edits the file system (more on that in the next point). This way, users can get "MyScript" in a script file, and it will access the global map of type names, fetch the path, and then load the script.

Solution 2: We can track the inheritance hierarchies of scripts. Then have the editor perform type checking on nodes as they are edited so that it can enforce the scripted inheritance defined by the custom type. In order to do this, we have to cache the inheritance hierarchies of scripts and scenes so that we can query these inheritance hierarchies without loading the resources and while updating the inheritance hierarchies and their dependencies as they change (since scripts' base type can be changed on the fly). This also has to dynamically store the records with string-filepath-passing setters and asset-returning getters; if we just directly stored the assets themselves, then the editor would have every asset loaded at all times (massive strain on memory).

Solution 3: The TypeDB and the global map from point 1 will need to each support separate logic for interpreting typenames under namespaces of some sort (so if I request "MyPlugin/MyNode", it will give me the path for "MyNode" under the "MyPlugin" namespace). This will provide at least 1 layer of separation to prevent obvious conflicts. This comes with its own problems though, because we then also have to support the possibility for a plugin's nodes to inherit from plugins in a different namespace, meaning users need to be able to easily access the namespace for a different plugin. So, the way the namespace is generated needs to be simple and consistent, and the engine needs to expose a means of fetching this value in the scripting API.

Solution 4: Problem 4's functionality can easily be wrapped into the TypeDB system by allowing us to store not only scripts, but also scenes. This does include an additional level of logic that must be calculated in real-time though, since we now have different control flow to handle for each TypeDB record depending on whether it's a script or scene. We also still have to figure out how we are going to modify the UX for the Node creation dialogue since it will now be able to show scenes in addition to nodes and the user will need to be able to distinguish between the types, even if it's a minimal difference. Considering adding a scene icon of some sort.

Solution 5: I already have solutions outlined for all of Problem 5's portions (you basically just fix all the issues directly), and I previously had already implemented them for a system that didn't rely on dynamic data and multiple sources of type definitions, but now I have to re-apply them with the new back-end. So, yay~ the design for all of these is already figured out. But booo, I still have to figure out how to replicate the behavior with all of the new code.

Current Status:

Even though I've been working on this for a while, I've mostly just gotten a lot of the new back-end infrastructure designed and written. There's still a bunch of testing and UI integration that needs to be done. A lot of the content has been "made" so to speak, but until I get a chance to test it all, I won't be able to tell how drastic the bug fixes will be (since there will assuredly be several bugs to deal with). I'm currently trying to get the create dialog figured out for the addition of custom types (so that I can add custom type nodes to my scenes). After I get that working, I'll do the same for scenes. And once I get both scripts and scenes capable of being created, I'll start testing the inheritance type-checking which will necessitate getting the dynamically updating inheritance and type information working (one of the most difficult parts of this whole system).

Now, the ORIGINAL goal I had at the start of the small group process was to improve my standing in the Godot community, get enough plugins made that I can start a Patreon campaign for my plugin development, and then get to a point where I'm making $20 per month from that work.

Well, to be blunt, there's no way that is going to happen, this year, let alone by CGDC (the deadline for the goal). There are many plugins I've been designing privately, but I haven't been actively developing any of them because you cannot currently use them in the ways in which I believe they SHOULD be used. And if there usability isn't high, then people will feel like it's a chore to use them in the first place. So the best way to make my plugins in general much more useful and appealing to people is if I improve the underlying system that makes them available in the first place.

In regards to my recognition in the Godot community, I would say I've made significant progress. I've developed a bit of a reputation already as being familiar with some of Godot's more advanced features. I've been made an official co-author of the engine, due to the volume of commits I've made to the repository. I've also become more familiar with particular people in the engine (have even had certain people contact me directly asking questions about various topics). So, on that front, I am satisfied, but also eager to improve my standing in the community even more by integrating these engine changes and making life easier on everyone in the entire Godot ecosystem.

I've been delayed in working on this stuff the last few weeks because I'd been dealing with sickness at home and technical issues (installed Linux on my desktop, loving it, after I finally got it working).

report.close();



User avatar
Exevalon
Posts: 25
Joined: Thu May 12, 2016 1:21 pm
Location: Richmond, Indiana
Area of Focus: Concept Art, Illustration, Graphic Design & Game Design
Occupation: Freelance Artist
Contact:

Re: Godot Dev: Custom Type Overhaul

Postby Exevalon » Sat Mar 31, 2018 9:57 pm

Great post Will, with a indepth breakdown of what you're doing.

Have you considered having others in the Godot community assist you with some of your plugins, in order to get them completed?


One thought opens a thousand eyes, one sun brings a thousand dawns.

willnationsdev
Posts: 7
Joined: Sat May 14, 2016 6:08 pm
Location: Waco, TX
Area of Focus: Programming, Design
Occupation: Web Developer at Baylor University
Contact:

Re: Godot Dev: Custom Type Overhaul

Postby willnationsdev » Wed Apr 04, 2018 5:56 am

I have considered it, but it isn't really realistic. Not unless I first publish the underlying foundation on its own and then have others build on top of that (for which I can't necessarily rely on others because no one else has expressed interest in working on this). To even get the foundation integrated, I'd have to prove it works properly and THEN I'd have to prove how it would be useful to even have in the engine (which, it wouldn't be useful, until the other features are added). So I'm just going to bundle them all together in a single PR with multiple commits.



willnationsdev
Posts: 7
Joined: Sat May 14, 2016 6:08 pm
Location: Waco, TX
Area of Focus: Programming, Design
Occupation: Web Developer at Baylor University
Contact:

Re: Godot Dev: Custom Type Overhaul

Postby willnationsdev » Fri Apr 06, 2018 10:19 am

Oh wait, I just realized you were referring to the plugins, not the engine features.

I mean, it would be entirely possible for someone else to build the plugins, but it suffers from the same problems. Most people don't have the skill or interest to build the same content that I want to build. I have more than a half-dozen plugins I wanna make.




Return to “Raul Rivera Arroyo's Small Group”

Who is online

Users browsing this forum: No registered users and 0 guests