1 billion star ESA Gaia catalogue in Celestia?

Discussion forum for Celestia developers; topics may only be started by members of the developers group, but anyone can post replies.
Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #21by john71 » 25.04.2018, 16:35

97 million stars database is available on this link:

https://zah.uni-heidelberg.de/institutes/ari/gaia ... gaiasky/downloads/#dr2catalogs

It is 9.1 GB.

It would be fantastic if someone could convert it to a Celestia conform format.

Avatar
selden
Posts: 10243
Joined: 04.09.2002
With us: 16 years 1 month
Location: NY, USA

Post #22by selden » 25.04.2018, 18:11

Celetia is still fundamentally a 32 bit application. It has not yet been modified to work with databases larger than about 1 GB.

Sorry.
Selden

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #23by john71 » 25.04.2018, 18:17

:cry:

Added after 1 minute 24 seconds:
Wait a minute. Celestia 1.7 has a 64 bit version...

Avatar
selden
Posts: 10243
Joined: 04.09.2002
With us: 16 years 1 month
Location: NY, USA

Post #24by selden » 25.04.2018, 18:22

But does it actually work?

Just because a program has been built with 64 bit compiler and linker options does not mean that it can handle large arrays. Someone has to carefully go through all of the code to make sure that the array definitions and subscript calculations actually do the right things.
Selden

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #25by john71 » 25.04.2018, 18:25

OK, so Alexell (or Fridger?), please help us... :help:

Janus
Posts: 208
Joined: 13.08.2016
With us: 2 years 2 months

Post #26by Janus » 25.04.2018, 18:30

Actually, I believe the issue here is the catalognumber, not memory size.

Catalog number is a uint32, and the stars are checked via sequence of pointers, which are size agnostic.

There is nothing memory wise to prevent the x64 version from loading and running with it.
Besides of course, physical ram limits.

What concerns me at the moment is the catalog number itself.
It is possible that celestia uses the catalog number implicitly assuming it is also the HIP#, which would break addons.
This is not something I have delved into yet.

I am currently downloading the DR2 formatted catalog of 97M stars.
My intent is to use the buildstar.pl script as a basis for converting DR2 to celestiaDB.
Two rounds, the first winnows the DR2 format down to the hip_main datafile format.
The second, assuming the first actually works, go ahead and use the existing perl script.

If that fails for some reason.
I won't be using perl since I am neither good with it, nor do I like perl.
Same goes for python.
More than likely I will use Lua or php, perhaps pascal.

One I get it converted, I will try it and see what happens.
If the catalog number is implicitly the HIP#, then the celestia code will need to be adapted.

Once I have it converted, I will upload it so others can play with it as well.


Janus.

Avatar
selden
Posts: 10243
Joined: 04.09.2002
With us: 16 years 1 month
Location: NY, USA

Post #27by selden » 25.04.2018, 18:42

Janus,

If you can, I suggest writing the software in Lua 5.1, primarily because it's included in Celestia. That'd ensure that anyone with Celestia could, in principle, run your catalog translation software on any platform without having to install some other OS-specific compiler and whatever dependencies it might have.
Selden

Janus
Posts: 208
Joined: 13.08.2016
With us: 2 years 2 months

Post #28by Janus » 25.04.2018, 19:00

@selden

A lua interpreter is not included with celestia, it is built in to the program itself.
There is no way to run the lua script.
Besides, file access in lua is horrid.

My plan is to convert the DR2 data into stars_97m.dat, then load it instead.
Doing the same thing as the 2M star catalog, only with 97M instead.

If it ever finishes downloading that is.

I will post when I am able to start making some progress.


Janus.

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #29by john71 » 25.04.2018, 19:02

Thanks Janus!

A few pictures from Gaia Sky:

20180425_screenshot_00000.jpg


20180425_screenshot_00002.jpg


20180425_screenshot_00007.jpg

Avatar
selden
Posts: 10243
Joined: 04.09.2002
With us: 16 years 1 month
Location: NY, USA

Post #30by selden » 25.04.2018, 19:08

Janus,

Lua 5.1 programs (scripts) just need to be given the extension .CELX in order to run them from Celestia.

As you wrote, the Lua 5.1 interpreter is built into Celestia, which is why I suggested using it.

Personally, I write most of my one-offs in Fortran 77 ;)

My understanding is that the full DR2 dataset is over 550 GB, so it would take quite a while to download. I haven't taken the time to investigate either it or the abbreviated 97 GB version. So far I've just been extracting and downloading very small subsets using the GAIA version of SQL.
Selden

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #31by john71 » 25.04.2018, 19:18

I'm using the 13.8 GB (compressed 9.1 GB) version. It will take years to sort this out...

Janus
Posts: 208
Joined: 13.08.2016
With us: 2 years 2 months

Post #32by Janus » 26.04.2018, 02:29

Okay, I have finally gotten the 9.1G set, and extracted it.
Oh man, what a mess.

I am at the pseudo code stage, but here is what I have so far.
The log file in 48-adp-040000-0.5-0.125 is where you start, this gives you an idea of what you have.
Then look inside the file metadata.bin next.
It starts with a uint32 big endian number 00 00 24 E0 which is 0x24E0, $34E0, or 9440 base 10.
That is followed by 9440, 76(0x4C, $4C) byte headers for the individual particle files.

Still working on the header.
The particle files, minus the opening header, are relatively easy.
If you can call a variable length structure easy.

Here is a sample of the parsing code used the gaia project.
Warning 1: Sample is in java instead of a real language.
Warning 2: I HATE java, and I freely admit to being biased against it.
Not saying it doesn't work, just that I hate it.

Code: Select all

                    // name_length, name, appmag, absmag, colorbv, r, g, b, a,
                    // ra[deg], dec[deg], dist[u], (double) x[u], (double) y[u],
                    // (double) z[u], mualpha[mas/yr], mudelta[mas/yr],
                    // radvel[km/s], pmx[u/yr], pmy[u/yr], pmz[u/yr], id, hip,
                    // tychoLength, tycho, sourceCatalog, pageid, type
                    int nameLength = data_in.readInt();
                    StringBuilder sb = new StringBuilder();
                    for (int i = 0; i < nameLength; i++) {
                        sb.append(data_in.readChar());
                    }
                    String name = sb.toString();
                    float appmag = data_in.readFloat();
                    float absmag = data_in.readFloat();
                    float colorbv = data_in.readFloat();
                    float r = data_in.readFloat();
                    float g = data_in.readFloat();
                    float b = data_in.readFloat();
                    float a = data_in.readFloat();
                    float ra = data_in.readFloat();
                    float dec = data_in.readFloat();
                    float dist = data_in.readFloat();
                    double x = data_in.readDouble();
                    double y = data_in.readDouble();
                    double z = data_in.readDouble();
                    float mualpha = data_in.readFloat();
                    float mudelta = data_in.readFloat();
                    float radvel = data_in.readFloat();
                    float pmx = data_in.readFloat();
                    float pmy = data_in.readFloat();
                    float pmz = data_in.readFloat();
                    long id = data_in.readLong();
                    int hip = data_in.readInt();
                    int tychoLength = data_in.readInt();
                    sb = new StringBuilder();
                    for (int i = 0; i < tychoLength; i++) {
                        sb.append(data_in.readChar());
                    }
                    String tycho = sb.toString();

                    byte source = data_in.readByte();
                    long pageId = data_in.readInt();
                    int type = data_in.readInt();
                    if (appmag < GlobalConf.data.LIMIT_MAG_LOAD) {
                        Vector3d pos = new Vector3d(x, y, z);
                        Vector3 pmSph = new Vector3(mualpha, mudelta, radvel);
                        Vector3 pm = new Vector3(pmx, pmy, pmz);
                        float[] cc;

                        if (Float.isNaN(colorbv)) {
                            colorbv = 0.62f;
                            cc = new float[] { 1.0f, 0.95f, 0.91f, 1.0f };
                        } else {
                            cc = new float[] { r, g, b, a };
                        }

                        Star s = new Star(pos, pm, pmSph, appmag, absmag, colorbv, name, ra, dec, id, hip, tycho, source);



Which then needs to be translated to a hip file compatible form.

Sadly, it looks like either perl or C/C++ is going to be the language of choice for this.
The former because I have a template to go by.
The latter for memory/performance reasons.

It may be necessary to modify the star class in celestia to add a hip, tycho & gaia index values.
I have not traced everything all the way through, but it appears so far that catalog index doubles as the hip#.
Not a huge change by itself, but one with a lot of ripples.

I will try to keep everyone up to date on my progress.


Janus.

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #33by john71 » 05.05.2018, 07:06

Whoa. This is the crown jewel: 601 million stars, 56 GB data.

It is finally online.

How the hell can Gaia Sky handle this much data?

https://zah.uni-heidelberg.de/institutes/ari/gaia/outreach/gaiasky/downloads/#dr2catalogs

Gurren Lagann
Posts: 92
Joined: 31.01.2018
With us: 8 months 15 days

Post #34by Gurren Lagann » 05.05.2018, 12:10

john71 wrote:Whoa. This is the crown jewel: 601 million stars, 56 GB data.

It is finally online.

How the hell can Gaia Sky handle this much data?

https://zah.uni-heidelberg.de/institutes/ari/gaia/outreach/gaiasky/downloads/#dr2catalogs

Gaia Sky's servers are Supercomputers. Done.

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #35by john71 » 05.05.2018, 21:03

601 million stars! Amazing! There are star "corridors" on the other side of the Milky Way!

20180505_screenshot_00002.jpg


20180505_screenshot_00000.jpg


Added after 9 hours 24 minutes:
Gurren Lagann wrote:Gaia Sky's servers are Supercomputers. Done.

I meant the Gaia Sky software!

It handles flawlessly more than 90GB of star data!!!!

No RAM issues, no performance problems.

Celestia should use a similar method.

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #36by john71 » 14.05.2018, 19:46

By the way, Gaia Sky is open source software, so it is possible to integrate parts of it's code into Celestia...in that way Celestia would be able to use Gaia Sky data directly.

Added after 4 minutes 41 seconds:
Also in this Space Engine forum thread (http://forum.spaceengine.org/viewtopic.php?f=10&t=297&sid=39d63c7b7b8d0ab0db1efa8e8571e765&start=15) it is mentioned that 2.7 GB data is sufficient to create an usable "stripped down" version of Gaia stars.

Avatar
Alexell M
Site Admin
Posts: 283
Joined: 07.10.2010
Age: 24
With us: 8 years
Location: Moscow, Russia
Contact:

Post #37by Alexell » 29.05.2018, 15:06

john71, Yes, this is an interesting idea. Perhaps this will really be done only by hiring programmers for money. I postponed the idea of collecting donations for Celestia, but I plan to return to this idea.
Admin of celestia.space
PC: Intel Core i7-8700 @ 3.20GHz, SSD, 16 Gb RAM, NVIDIA GeForce GTX 1080, Creative Sound Blaster ZxR. Windows 10 x64.
Phone: iPhone 7 Plus 128 Gb. iOS 11.
Image


Return to “Ideas & News”

Who is online

Users browsing this forum: 3 guests