New models format

[tomangelo]

As you know, current model format is very old and insufficient for our goals (easy modding, skeleton animations). Many things are hardcoded in game engine, so currently modding is almost impossible.

Of course we could use some of existing formats, but:

• we need to move as much as possible from code to model files (so it must be something better than just mesh file)
  
• it should be easy to update (when someone wants to change the wheel model in bots, then it would be better to modify one file instead of modifying every model that contain wheels)
  

There might be an idea - making one in-game file from parts files by assembling them with an converter. Parts files would be in anyone format that support animations and anything else. But how converter would like to know how it should be placed? It would use a text file, with describing all properties, like lights, health points, animations (simple ones, rotating, moving, like it looks like in our code), etc. It could allow us to move as much as possible from code to model files, leaving only loading code and some things that wouldn't be possible to implement in this way.

I'm working on how would this file look like, for now I've got something like this (still WIP, but I want to know if this idea makes any sense) :

Spoiler :

It all base on our current model specification, so it doesn't contain information about crashspheres, or textures.

Code:

// Lines started with these characters are comments

Header
//There are all properties of final model

frames [x] //amount of total frames for animations
is_destructible Y/N
models n //optional parameter, if implementing this format would require to know how many source models will be included in final model before converting.
lights n //like above

Lights
//There you can set where lights will be placed. Also you can change their color, power, direction etc. By default, they're set to light forward
Light [x] [y] [z] [ID] [color] [power] [type] [additional parameters] //color may be defined in hexadecimal number; types described below; ID is related to parent/child points, so it allow us to make light source that will move with model while animation.

//Available types:
//Point - small, shining point
//Sun [sx] [sy] [sz] - shining ball, you can set scale for ball
//Cylinder [rx] [ry] [rz] [s1x] [s1y] [s2x] [s2y] – Light pole, you can set rotation of cylinder and set scale for both ends. 2 lasts parameters can be optional, so we'll get cone-sized light, as one end will be size=0.

Models
//Importing models from files

//Main model
//Note: first model is always master model, it's placed in local coordinates 0,0,0
file filename.txt
child [ID] [x] [y] [z] [rx] [ry] [rz] [sx] [sy] [sz] //create new child point and set it initial position/rotation/scale. There will be locked child model to this parent model. ID should be unique
animation [sf] [ef] [mx] [my] [mz] [rx] [ry] [rz] [sx] [sy] [sz] [sound] //sf - start frame, ef - end frame, m - move, r - rotate, s – scale, sound – sound that will be played over animation.
//if we'll import model with embeded animations, then we can pass move/rotate/scale parameters as 0.

//Second model
file second.txt
parent [ID] //linking child model to parent model
//Non-master models can have child models too.


//Parent/child points should be able to move, if we're rotating arm, then points on arm should be moved too.

How would look like all this process?

In this example we'll build a WheeledGrabber bot.
At first we'll need to know what models we'll be using. We'll need body frame as main model, 4x wheels, 2x arms, also there are 3 tool models. Also we'll use some lights.

So our file would look like this:

Spoiler :

Code:

Header

frames x
is_destructible Y
models 10 //like I said, it's optional to implement
lights 4

Lights

Light 10 1 -2 #FFFFFF 1 cylinder 0 100 0 0 0 4 4 //left front bulb, cone-sized ray
Light 10 1 2 #FFFFFF 1 cylinder 0 100 0 0 0 4 4 //right front bulb, cone-sized ray
Light -10 1 -2 #FF0000 1 sun 0.2 0.2 0.2 //left rear bulb, it may be even replaced with point-type light
Light -10 1 2 #FF0000 1 sun 0.2 0.2 0.2 //right rear bulb

Models

//main body
file body.ext //whatever extension we'll use
child 1 8 0.2 -5 0 0 0 1 1 1 //left front wheel
child 2 8 0.2 5 0 0 0 1 1 1 //right front wheel
child 3 -8 0.2 -5 0 0 0 1 1 1 //left rear wheel
child 4 -8 0.2 5 0 0 0 1 1 1 //right rear wheel
child 5 0 3 0 0 -15 0 1 1 1 //arm #1
//second arm will be connected to first arm

//L F wheel

file wheel.ext
parent 1
animation 30 45 0 0 0 0 90 0 1 1 1 "drive.ogg" //driving forward; frames numbers are only for example
animation 45 60 0 0 0 0 -90 0 1 1 1 "drive.ogg" //driving backward
animation 60 75 0 0 0 0 0 40 1 1 1 null //turning right, null means that there is no sound in this animation

//L R wheel

file wheel.ext
parent 3
animation 30 45 0 0 0 0 90 0 1 1 1 "drive.ogg" //driving forward; frames numbers are only for example
animation 45 60 0 0 0 0 -90 0 1 1 1 "drive.ogg" //driving backward
animation 60 75 0 0 0 0 0 -40 1 1 1 null //turning right, but rear wheel need to turn left

//and so on

file arm.ext
parent [5]
child [6] 0 2 0 0 90 0 1 1 1 //keep in mind, that there are relative coordinates to this mesh,
//at first, "youngest" childs are placed/rotated/scaled by this child lines, next there are placed/rotated/scaled their parents along with childrens, and so on
animation 90 105 0 0 0 0 -45 0 1 1 1 //grabbing, in this case rotating arm, all child models need to be rotated along with this arm
//second arm and tools will be placed in similar way

This will be both with model files converted to in-game file.

This still need to be detailed, also there are still features that must be included. I just want to know if this makes any sense, if this is even possible to implement and if it's worth to work with that or I should just reject this idea and try to figure out something better.

[RaptorParkowsky]

How it will be look as BLEND source file? There will be also every part of the object in different file? If so, that will be difficult for designing and modyfying hi-poly models.

If I understand correct, the converter of CGI models to game models must do:
- triangularize all meshes (if this is needed)
- rewrite materials propelly (if shaders in the game will support some metalic effects, etc.)
- rewrite the UV maps
- extract duplicated parts with their animation parametres, crashboxes (wheels, tools, etc.)
- make files for duplicated parts, non-duplicated parts and files with informations about every object in the game (meshes, animations, sounds, etc.)
- make lot of other stuff that I don't understand or just forgot.

Well, I'm not sure if this is gonna work in every case.

[tomangelo]

Every part in different file, something like it looks now. You prefer to modify one model in several another models, or modify one and just "recompile" all models that require this one model?
I wasn't thinking about exporting it from Blender. It will be assembled from many models like we're using now, that can be exported from Blender. They can include hitboxes, animations or whatever else is needed to describe a single part. Then we'll import 4 files, place them, add some another informations, add some simple animations (like rotating arm) and voila - we'll get full model.
Materials doesn't have to be rewriten, it's described in parts files. It can be even still unchanged. Same UV maps.
Converter will just add all meshes (like importing in Blender), crashspheres, lights and another informations, then make it animate and save that in final file.

[RaptorParkowsky]

Every part in different file, something like it looks now. You prefer to modify one model in several another models, or modify one and just "recompile" all models that require this one model?

If in the future there will be still a lot of duplicated meshes in CGI hi-poly objects, so I would just name them in the same way in every Blender source file and then the converter should find those duplicates and split them into another model file that will be used by every object that is related with that mesh.

I did a quick and easy example for what I'm looking for in case of containing the models, textures, crashboxes and animations of the objects in one Blender file. Of course this is not 100% accurate and optimal (I didn't know how to delete texture duplicates without doing a mess with UV mapping - well, I really hate textures in Blender ) and the Radar Station will have probably hardcoded animations as there's some random movements when the radar detects an Alien or AlienWorm just infcted this building. But I would probably stick with similar standardized process of making/modyfying and containing the CGI models that must be converted into game models later.

Just open the blend file and hit Alt+A to play the animation. To see more and modify anything else just change the screen lay-out of the scene.

[tomangelo]

Ok, such animations (like pointing an alien) would be impossible (or very hard) to implement in this case. Such spinning would be possible (just like in Blender, start keyframe, end keyframe for every part mesh), but I've another (maybe better) idea, but it will require some changes.
Crashboxes, textures, meshes are already in newest model format (https://github.com/colobot/colobot-data/...9f2c8d5161), my format doesn't apply to these models, they're will be imported to final in-game model with some additional informations like lights.

[piotrdz]

From my point of view, I can say that I created my new model format with the idea that model formats should express everything that so far has been hardcoded in functions creating objects in code. I also assumed that handling one model file per object is better than keeping the code that tracks down all object parts and assembles them together.

So if only we can achieve that input model carries all data like that, from programmer's point of view, I'm happy and it doesn't really matter how the models are actually assembled together outside program.

However, having said that, I would like to keep things as simple as possible, and not create new tools like special assemblers without reason. The current version of my model format can be easily imported/exported directly from Blender*. If we want to add new features, I would first try to include them in import/export script rather than creating special tool.

* - I know, the script in repository is not updated, but I'll be working on it presently.

Besides, as I've said before, Blender models can reference other models and we don't need to duplicate anything if we don't want to. So "assemly" of models in this way would be actually creating a Blender model that only imports meshes and other things from other models and only places them in appropriate configuration. I'm not sure, but I think even animations can be referenced this way.

[tomangelo]

There is a problem, especially with animating RadarStation and DefenceTower - their animation depend of where it should end. I cannot imagine making animations to DefenceTower or Shooter bots that would aim in any possible direction. On the other hand, I haven't seen everything and maybe there are some solutions.
Now I was thinking about some sort of library files, that would contain meshes, animations, physics, lights and some source code to manage all actions, that won't need to be in game engine any longer. But this would require making some sort of simple API to models, like run(); turn(); special_action(value);. Special actions would be just for such models like Shooter, that could get parameters X and Y, then some code in this file would calculate how much gun should be rotated in X and Y axes.
Now when I'm thinking about that, I'm afraid that it might be impossible to completely unhardcode engine code without messing around in another parts of the game.

[krzys_h]

Object logic is still hardcoded, so I don't think it's a big problem if DefenseTower and RadarStation will have some special handling. At least until we'll have some external scripts for object logic, then I think we could also make special animation scripts.

[tomangelo]

That's what I wanted to achieve - move object logic to model file. Then everything the model would need would be in one file.

[RaptorParkowsky]

How to store object logic in BLEND source file of the model? We'll be loading text file into Blender's text editor?

[tomangelo]

Well, it won't. Object logic will be outside parts models.
   
Parts models will be editable in Blender, object logic will be stored in separate text file editable even in Notepad, it might be possibly merged with additional parameters file.

[krzys_h]

I don't see a need for a converter. As @piotrdz said, Blender export script should be able to do everything once it's updated for the new model format.
The only problem I can see is that after editing e.g. wheel model you have to reexport all objects with wheels.

[tomangelo]

If there is such possibility, then converter isn't needed.
Now there is only question about scripting object logic and animations.

[piotrdz]

The only problem I can see is that after editing e.g. wheel model you have to reexport all objects with wheels.

Blender allows for running user scripts on startup, for example:
  blender -b file.blend -P script.py

So we could just write a set of scripts and Makefile rules to export everything that's changed.

[RaptorParkowsky]

Well, I don't like this solution. This will be hard for implementation all-new redesigned, hi-poly models. They will be created in one files and their animations will be probably more advanced. If this solution is going to work with present models, so I'm ok with that, but new CGI models will probably require something else.

[tomangelo]

Well, I don't like this solution. This will be hard for implementation all-new redesigned, hi-poly models. They will be created in one files and their animations will be probably more advanced. If this solution is going to work with present models, so I'm ok with that, but new CGI models will probably require something else.

So how will you animate DefenceTower that might aim to any point from any direction? Without some coded logic it might be impossible.

[RaptorParkowsky]

So how will you animate DefenceTower that might aim to any point from any direction? Without some coded logic it might be impossible.

Well, I don't talk about scripted animations. It would be helpful to show you what I'm talking about:



This kind of animation I'm talking about. It would be hard to keep all those new meshes in different files.

[tomangelo]

No, it wasn't. Models with such animations would be stored in parts models, that would be in any format that support such animations. Animations like these in text file were related to such animations like we got now in code. It was supposed to move some logic from code.
All this concept applies to interacting between models, that don't know about each other, because arm mesh might be connected to winged body, but it can be connected to tracked body too. At the end of this arm might be claws, but also it might be sniffer module. Relations between these models was supposed to be described in this file, but it doesn't interfere in these models themselves, so they can be animated like what you point at.
If there is possibility to import another files while exporting, then we only need to think out how to import lights, crashspheres/crashboxes and other object logic.

[Krux]

The only open 3D model format that I know with animations that is used by more than one engine is the inter quake model format. Here is a link: http://sauerbraten.org/iqm/
Sadly it's documentation could be better. There is an example that actually loads a 3D model, but it has a coding style that has space for improvements. But I managed to write a working importer for that format in the nim programming language. You can look up the code here: https://github.com/krux02/opengl-sandbox...g_tiny.nim even if it's another programming language, it tries to be as minimal as possible for loading the mesh and rendering the bones, and might be a nice alternative to the original example.