Page 1 of 8

New Celestia unified runtime database

Posted: 03.04.2019, 17:09
by pirogronian
--- Edit 06.06.2019 ---
I changed topic name (and disscussion subject) to more general.
--- End edit ---
Dear Celestians
We, CDT (Celestia Developers Team) :wink: , are going to implement new Celestia runtime database, which could easly handle various astronomical catalogs with various types of objects (starsm, DSOs or planet bodies). As many new catalogs have enormous amount of data, we want to enable modularization of Celestia database and so we are fased with problem, how to do it properly and convenient for data/addons contributors.

The main question is now cross-indexes structure. Up to now, main Celestia index was Hipparcos catalog number. We want to make HIP catalog functionally equal with others and have ability to cross-index various catalogs, including Gaia and Kepler. However we are unsure, how to choose universal object identifier, needed for effective and clear cross-indexing. Problem could be marginal, if avaliable objects count would relatively small. But if data should be modular and dynamic, we are unsure, how to organize its structure. So, we call for opinion, especially from peaple involved in content creation.

My draft proposition is support for dynamic load of multiple files with stars.dat format (or similar). Content developer could, for example, provide stars.dat equivalent along with proper cross-index files for a specified object id range. Or for specified numbers range from particular catalog.

Posted: 03.04.2019, 18:09
by Janus
Edited to attempt greater clarity.

When I was looking at doing that, what I came up with is first having an internal index number that has nothing to do with any external DB at all.
What I mean is to have a separate DB for each catalog {HIP, HD, Gaia, whatever else}.

Then have a core DB which is primarily an index into the others.
Since the same star can occur in one or more catalogs, or perhaps not, structure the primary database for that.

Celestia core DB star Entry:
Celestia Index:
Celestia Type:
Celestia Name:
Active Data:
Load Data:
Load Name:
Hip Number:
Hip Data:
HD Number:
HD Data:
Tycho Number:
Tycho Data:
Gaia Number:
Gaia Data:

The core database would do nothing except provide a uniform interface to stellar data.
This would allow for independent catalogs to be loaded, whether they overlapped or not.
A default load would have to be designed of course, but that would still leave options for adjusting data use later.
Which could be something as simple as an index like CEL_DFLT and a series of catalog entries based on an enum, at least to start with.

Code: Select all

   HIP = 1;
   Tycho = 2,
   Draper = 3,
   Gaia = 4

The display/render routine would get coordinates from here.
For index = 1 to CoreDbSize do Render(DB[index].ActiveData.pos);
Allowing star catalogs to be switched on command by simply changing ActiveData & ActiveName from Hip to Tycho to HD to Gaia.
Starting over would be as simple as copying LoadData & LoadName to Active from Load.

The same basic idea could be applied to other things, such as Asteroids, Comets, Asterisms, and whatever else.
The ability to switch out zodiac symbols could be very useful, It would be great for me because I have asterisms files showing stellar routes I Use for visualization of journeys already, and switching them out would be useful.
It would also open the ability to create them on the fly from scripts, which I would like a lot.
Imagine being able to record movements during a lecture that the students can then play back and explore for themselves.

All numbers, except universalcoord, stored on disc, and held in memory to be uint64_t or double, both of which use 8-bytes.
Even if working in float{single}, allocate double storage.
Strings to be kept in bytes alignments if feasible.
This simplifies reading data from disc, and stabilizes memory layout as well.
It also aids in the long term conversion to 64-bit by allowing memory alignment on 16 byte boundaries.
That should help porting to Android or other portable OS where predictability is required.
With predictable in memory structures, it open the possibility for the display subsystem to scan for coordinate data directly.

Text searches can then be simplified.
Turn greeking on if an individual wants it, while leaving in memory strings as ascii.
On a search, convert greeking back to plain text to make search simpler.


EDIT: Attempted to add clarity I lacked before.
All I can do is hope I succeeded.

Posted: 03.04.2019, 19:12
by pirogronian

There is plenty of places I don't understand in Your description. But I will try.
1. At first, what exactly is DataSource? String path to data file name or URL to astro catalog?
2. What is Dex?

Now let me try to explain, how current implementation looks like (it's quite similar to legacy implementation):

m_mainIndex map:
UI -> Object

^ for search object by UI

m_catxindex map:
CI -> { CN -> UI }

m_celxindex map:
CI -> { UI -> CN }

^ for search UI by CN and vice versa

m_names map:
UI -> names
name -> UI

^ for searching UI by name and vice versa

m_catalogs map:
CI -> Catalog class

Catalog class:
`-> nameToCatalogNumber function
`-> catalogNumberToName function

UI - Universal Index
CI - Catalog Index (registered Catalog id)
CN - catalog number (object id per catalog)

Database is though to handle dynamic objects loading, even one-at-once, and the same for unloading (planned in the future).

UI have to be either deterministically generated from dynamic data or stored statically in adequate data files. I proposed direct Catalog1-number to Catalog2-number cross-index without any UI at all, but @onetwothree refused it and I think he was right.

Posted: 03.04.2019, 21:44
by Lafuente_Astronomy
I would also call for modularization of stars in every galaxy. We should be able to enable or disable stars in other galaxies, specially if we are going to use what Astronomy has given us in this day. Same with the catalogs.

And if you can, you can also add modularization for the grids, so that when I go to another planet, the grids there are as seen from that planet and not from Earth.

Added after 4 hours 59 minutes:
Quiet unrelated but since this is a development post, I'll say it here:

Can you add some changes to the "Star Browsers" function? I think what should also be included as a category for Star Browsers is an Alphabetical Star list of all named Stars from A-Z, both proper names and designations, and if possible, 88 separate categories for all stars in a single constellation. Like a category for all stars in Andromeda, all stars in Antlia, and so on.

Posted: 04.04.2019, 12:36
by Alexell
By the way, maybe it is time to make the database dynamically updated (from the web)?

Posted: 04.04.2019, 13:11
by Lafuente_Astronomy
Alexell wrote:By the way, maybe it is time to make the database dynamically updated (from the web)?

Well, I think that's a good idea. By all means, do what you think is best :smile:

Posted: 04.04.2019, 13:44
by Janus
I realize I may be in the minority, but the lack of that sort of "Feature" is one of things I like about Celestia.
Automatic online or cloud anything is also an evil I avoid.

If others want to play with web updates, enjoy.
Any and ALL!!! forks, tweaks, experiments, or anything else I put up here will NEVER!!! have an auto update or live link feature active.

Not trying to pontificate, but no thank you for my part.


Posted: 04.04.2019, 15:06
by pirogronian
Data autoupdate, if any, will be rather via lua script or other external tool, and ever in core, it never will be main option, I think.

So, calm down, @Janus. We care about our users (God is a witness, we currently have not too many of them) :wink:

Posted: 04.04.2019, 15:28
by Janus
My apologies if I gave insult.
It was not intentional if I did.

In my work, stand alone feature updates cause far more problems than they fix.
I also work quite near the hardware level normally, so it is very touchy.
The law of unintended consequences is a harsh one, and it bites, so I am always wary.
Compilers like to optimize, and sometimes things are done in a particular order, or with a certain timing, on purpose.
Funny what happens when semaphores are checked on a timer interrupt, and some idiot gives his "Little" routine a high priority by turning off interrupts for a few micro or milliseconds to save time on the context reload.
Does interesting things to IO in the real world.

A drop down menu that checks and does file updates from within the program is logical.
Generate some checksums or hashes, then compare from an online list, and grab the individual files or entries as needed.
Supcom did a thing kind of like that, their patches were a set of diffs.
That could help Celestia be more portable and self contained, which is always a good thing in my opinion.


Posted: 04.04.2019, 23:01
by Lafuente_Astronomy
Online Data-upload would be good I think. The Gaia Archive website has lots of data that can be used of Celestia. It's not necessary that we have to download everything, since the graphics of Celestia cannot handle a lot. But we can at least get enough with the most accurate information so far.

Posted: 06.04.2019, 14:14
by Lafuente_Astronomy
Alright everyone, I motion to put this thread under "Announcements", since this is a development topic. If you agree, then I'll bring it there

Posted: 06.04.2019, 17:00
by pirogronian
I think it is indeed a very good idea.

Posted: 06.04.2019, 22:13
by Lafuente_Astronomy
pirogronian wrote:I think it is indeed a very good idea.

Alright. Putting this in Announcements now

Posted: 16.04.2019, 14:57
by Lafuente_Astronomy
I think I have an idea for a universal object identifier but it may only apply to stars.

As we all know, not all stars are cataloged in all the same catalogs. A star may have a membership in one catalog and a membership in another catalog. But another star can have a membership in the first catalog while not having a membership in the second catalog.

So, for Celestia only, there should be a set catalog which contains all stars from all catalogs, and includes stars from the Milky Way and other galaxies with proper star catalogs. It would be called th "Celestia Catalog". It is through the use of that universal catalog that it may fulfill the universal object identifier problem, at least for stars. The difficulty would be in starting where identity Number 1 begins. In the HIP, it starts at the Vernal Equinox, and it may also be different for other catalogs. As such, I'll leave it to you whether factors would be determined in giving numbers to the stars in the Catalog.

If that is feasible, then that's good. But if it cannot be done, perhaps there are better alternatives that can be used for the Universal identifier

More power to you.

Posted: 16.04.2019, 18:33
by pirogronian
In general we (with @onetwothree) have the same idea. But here is a problem: catalogued objects (mostly stars) are in enormous number. Even if done, Celestia Catalog fould be also enormous in size. This is why I'm looking for distributed way of building such a catalog. I suppose this work could be performed with various people, even as part of their addons. So we need some way to avoid duplication.

Here is my recent idea. Given the fact that particular maintainer would start with some particular catalog data, ID could be built with symbolid catalog id and catalog number. Supposing ID to be 64-bit integer type and HIP to have id = 1, we can propose following procedure for this catalog objects:
ID of HIP X = (1 << 56) + X. I mean, original catalog id would be stored in forst ID byte, and then would be appropriate catalog number.
Props: Easy ID evaluation procedure, much less probability of ID conflict.
Cons: Very prodigal, many values would be never used. It also allow to mistakely bind the same object to different IDs without being noticed.

Posted: 16.04.2019, 18:54
by Janus
Edited my earlier post @ #2 in an attempt at greater clarity, I hope it helps.


Posted: 16.04.2019, 22:18
by Lafuente_Astronomy
Perhaps the Celestia Catalog can be divided into sections either per galaxy or per section of the Galaxy(That is entirely up to you, for me, I'd chose the former). There is still a Universal Object Identifier but there could be put them into different sections without changing its identifier. Like if we add a suffix to differentiate it from a similar number of a different section. So there still exists an overall set catalog for all stars in Celestia (The Celestia Catalog), but for modularity purposes, sections from it can be added in which it has be added a suffix to indicate its position(At least known to programmers) and thus be properly identified as to what and where it is.

If for example, the Celestia Catalog would comprise all the stars in Celestia, and that includes the Milky Way Stars(Interstellar Medium and Halo, if there are any stars in the Halo) and Stars beyond the Milky Way. For the Milky Way, it would have the specific Celestia Catalog Suffix of 1, because that's our home galaxy. So, the first star identified as Number 1 in the Milky Way should be called C-1 1. Similarly, if suppose stars in the Large Magellanic Cloud are classified as Celestia Catalog Suffix of 2, then its first star is identified as C-2 1. So in The Celestia Catalog, it has sections for easier access, C-1, representing the Milky Way Stars(I think adding at least 1.4 billion stars in the Milky is the main priority here), and C-2, representing the Large Magellanic Cloud Stars, and obviously, there are more sections per galaxy to come. This way, due to the immensely numerous numbers of galaxies and their respective stars, it could in some way, manage the overall Celestia Catalog. While there remains the entire Celestia Catalog with its Universal Object Identifier, it can be categorized and organized by creation sections of it which are indicated by a single suffix alone.

The Celestia Catalog and its divisions for non-stellar objects will be up to you, as they are not of my interest. For Galaxies however, I think there would be need of a suffix based on its group, bubble, cluster, super-cluster or any other massive universal structure it belongs to.

As again, let me know if that idea is feasible or not. Thanks in advance

Added after 12 minutes 52 seconds:
For me, if there are any star identifiers for stars in any kind of Cluster, especially Open and Globular Clusters, their Celestia Catalog identifies must be within the Galaxy that they are in or being captured by. But for clarity and modularization's sake, I think there should be a catalog for all stars in Open Clusters, and all stars in Globular Clusters. We can start with catalogs of all stars belonging to specific kinds of clusters in the Milky Way.

In the long run, doing this for immensely huge elliptical galaxies like M 87(Whose Black Hole photograph was recently revealed to the public last week) would be a very long and tedious chore, due to M 87 possessing around 12,000-13,000 globular clusters

Posted: 16.04.2019, 23:07
by Janus

Why not just add a parent to the standard star type, the same way bodies have parents?
That could be galaxy, or cluster, or whatever.
Extend that idea to have a catalog per galaxy or cluster.
Then have the cluster be parented to a galaxy.
With of course, clusters, star catalogs & bodies listed as children of galaxies.
Which will of course mean a different catalog of bodies for each galaxy or cluster.

MW:HIP {These point to stars in the Milky Way}
MW:Bodies {Planets and stuff here}

AN:?? {This points to stars in Andromeda}
AN:Bodies {Planets and stuff in Andromeda}

Just dummy names to convey an idea.


P.S. Is there a reason your posts are showing as a wall of text, or is just the way I have my system setup.

Posted: 17.04.2019, 01:13
by Lafuente_Astronomy
Ahhh, that could be a better idea. But since the devs are calling for a Universal Object Identifier, thus I explained that there could be categories and sets for differing objects while all the while belonging to a mother set, that is the Celestia Catalog.

And since many of the objects, especially stars in Celestia have memberships in more than just one catalog, it would be wiser to just put them all in a single Celestia Catalog set for all stars in the Milky Way, and then divide them according to their respective memberships in many catalogs there. Any stars that do not belong to any of those catalogs can be put in Celestia Catalog MW:Other. In short, I agree with you but for the sake of a Universal Identifier, we must create a Celestia Catalog.

As for your P.S. I usually type a lot, so it gives the illusion of being a wall of text

Posted: 17.04.2019, 03:06
by Janus

I operate under a simple rule I learned from a database and UI designer.
"If you can see it, you can sort it."
So to me, creating indexes to act as pseudo catalogs makes perfect sense.
One DB/Class for planets just barely to small to fuse hydrogen and down.
One DB/Class for that line and up.
Generally speaking, a planet or anything smaller is going to be orbiting a star.
Also generally speaking, a star orbits a galactic center.
Sure you have *inary stars orbiting each other around a barycenter, but what does the that orbit.
Though exceptions are rare, relatively speaking.
They can be handled by linking to the proper parent structure directly.
Thus while a human might find a planet with a galactic orbit odd, a program will just plot it.
The same goes for Oumuamua, all it needs is very custom xyzv files to go with its galactic orbit.

I refer to putting each catalog in its own copy of a star DB/Class for ease of loading.
How you index that/those in memory Db(s) is up to you.
If you join them via index, you can still work with them individually, while displaying them collectively.

The big thing about that approach is the ability to use a DB/Class appropriate for whatever.
You have a base, then add whatever that kind of thing needs.
Imagine something like Celestia Origin where you can select objects according to the year they were discovered.
All you would have to is add 'Discovered' as a property.
You could have multiple DBs of asteroids and comets, with each from a different source, then turn sources on and off.

Another way to look at it is using indexes to translate the raw data into a display list for us to look at.
I know it is a layered approach, but I am looking at more than just using Celestia to display pretty pictures.
My hope is to be able to use it to show things as they really are, somewhere else.