1 billion star ESA Gaia catalogue in Celestia?

Discussion forum for Celestia developers; topics may only be started by members of the developers group, but anyone can post replies.
Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

1 billion star ESA Gaia catalogue in Celestia?

Post #1by john71 » 10.08.2016, 17:01

As I know the ESA Gaia project's first release date is 14 September 2016.

When will this one billion star catalogue be available in Celestia?

Can Celestia use this catalogue considering the current version of the software?

***
by FarGetaNik » 03.08.2016, 23:02
It is technically possible for Celestia to handle such a catalogue, but I noticed that Celestia gets pretty slow when handling a huge amount of xyzv data (some megabytes), wich is quite understandable. I think there is an addon on the Motherlode including a million stars, I don't know anymore if this slowed the program, I don't have it installed anymore. But one billion stars? I think Celestia will have to stuggle alot to load all this data, and probably the disc space necessary would also be a problem.

I'm not into the details Celestia's deep sky features, but I think there is a distance limit for rendering stars?

***
by selden » 04.08.2016, 01:37

A quick check of Celestia's ChangeLog.txt reveals that the 16K light year limit was removed in v1.6.0
Selden

***

by john71 » 04.08.2016, 09:29

That's OK, it is now approximately 10 million light years (I tried to populate IC1101 with stars without success...I had to "clone" the galaxy at a much closer distance).

My question is that 1 billion stars will make a huge dat file, maybe around 20 GB. Can Celestia load stars data somehow partially?

And by the way thank you for your online guides and add-ons, you really helped me VERY much with your work on Celestia. :) (Selden)

Oh, one more thing: do you have any information about a 64 bit version for Windows? The PC hardware can handle for example a few 8k dds textures (without tiles or ctx) but the memory limit is annoying...Celestia crashes after loading a lot of textures...

***

by selden » 04.08.2016, 12:24

When built for about 2 million stars, stars.dat is about 40MB, so I have to agree with your estimate :(

I'm not sure how Celestia would decide which stars to load. Redesigning Celestia to use a database file for stars might be necessary.

Thanks for the kind words!

Unfortunately, extending Celestia to be a 64 bit program wouldn't be just a recompilation. It would require somewhat of an internal redesign, too. It would have to use larger array pointers, for example. It also would need to do some kind of memory management for the graphics card.

Selden

Avatar
Alexell M
Site Admin
Posts: 283
Joined: 07.10.2010
Age: 24
With us: 8 years
Location: Moscow, Russia
Contact:

Post #2by Alexell » 14.08.2016, 17:33

Maybe it will be interesting. This a ajtribick's post on Celestial Matters forum:

In order to prepare for the upcoming GAIA release of data for the TYC2 catalogue, I've created an updated stars.dat based on the XHIP catalogue. Source code (Python 2) and a zip file are on my GitHub page.

It will also be possible to use XHIP to generate scripts to highlight the various stellar associations, hopefully will be able to add this enhancement soon.

Source code on GitHub
Release page including download of stars.dat
Admin of celestia.space
PC: Intel Core i7-8700 @ 3.20GHz, SSD, 16 Gb RAM, NVIDIA GeForce GTX 1080, Creative Sound Blaster ZxR. Windows 10 x64.
Phone: iPhone 7 Plus 128 Gb. iOS 11.
Image

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #3by john71 » 18.08.2016, 16:21

I don't really get it. This is NOT the new Gaia TYC2 (very accurate) catalogue... Isn't it already available in Celestia (XHIP catalogue)?

Avatar
selden
Posts: 10245
Joined: 04.09.2002
With us: 16 years 1 month
Location: NY, USA

Post #4by selden » 18.08.2016, 22:52

While Celestia does use information provided in the second reduction of Hipparcos data, XHIP is a compilation of additional, more accurate, information about most of those stars.

For details, see https://arxiv.org/abs/1108.4971
Selden

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #5by john71 » 19.08.2016, 06:12

Oh, thanks.So these are the relevant star catalogues: 1.) Hipparcos Catalogue (118,200 stars), 1997 2.) lower-precision Tycho Catalogue (more than a million stars) 1997 3.) enhanced Tycho-2 Catalogue (2.5 million stars) 2000 4.) Extended Hipparcos Compilation (XHIP) (116,096 spectral classifications) 2011 5.) Tycho-Gaia Astrometric Solution (TGAS) (about 2 million stars in common between the Tycho-2 Catalogue and Gaia) 2016 6.) GAIA (positions and G magnitude for about one billion stars) 14 September 2016.

nur
Posts: 1
Joined: 29.09.2016
With us: 2 years

Post #6by nur » 29.09.2016, 06:08

I am working on CAELI which is Celestia and I am currently working on getting 1 billion objects into CAELI. A script might have to be written to import ESA GAIA .csv files. I am working on it. CAELI is at http://illimitae.com

CAELI is now running 64 bit and a beta for MacOs will be ready most likely oct. 2016.

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #7by john71 » 30.09.2016, 13:51

Very interesting news! Will there be a Windows version too?

Avatar
John Van Vliet
Posts: 2710
Joined: 28.08.2002
With us: 16 years 1 month

Post #8by John Van Vliet » 01.10.2016, 03:27

a "stars.dat" is also a windows file, and a apple mac and a linux and a bsd file

now if you are referring to the OSX "CAELI" that is Apple
now seeing as apple is really a fork of BSD and bsd and linux share a code base ( sort of if you squint hard)

that is easy

a MS windows should be rather easy if the changes have been ALSO included into the "win32" sections
and for a long time it will be 32 bit on windows
not sure about 10 but win7 still had 16 bit coding ( for backwards compatibility)

Janus
Posts: 208
Joined: 13.08.2016
With us: 2 years 2 months

Post #9by Janus » 03.12.2016, 18:53

My personal solution to the star catalog problem may seem weird.

I am working on making a different star catalog system.
My intent is to have a catalog for Hip, Tycho & Gaia.
With each having the index number as the first field.
However, I would have the catalog sorted by Parallax/Dist so partial loads result in uniform coverage.
With this have a text based data file for additional info.
What I have so far is an html like structure.
I dislike xml, so please, understand I will not be using it.
Instead it would be more like

<star>index#
<starname>Name given to star.</starname>
<staraka>HIP?????</staraka>
<staraka>Tycho?????</staraka>
<starattribute>Something here<starattribute>
<startrivia>Something trivia like</startrivia>
<note>who knows what</note>
<note>Could have more than one</note>
</star>

With a parser that would skip or ignore unrecognized sections.
I know it is very xml like, however with no unpaired sections.

The purpose being that the binary data could be there for speed.
While the text data could be parsed via lua quite easily for presentation.

I remain undecided on combining the DBs in memory, or offering (a) choice(s) between them.
Might be fun to make an animation switching between the sets.

Janus.

focin albert
Posts: 1
Joined: 09.12.2016
With us: 1 year 10 months

Post #10by focin albert » 10.12.2016, 00:02

@Janus

That looks too weird. Would you please explain it via visuals....!

Janus
Posts: 208
Joined: 13.08.2016
With us: 2 years 2 months

Post #11by Janus » 12.12.2016, 07:43

I do not have any visuals to use.
It would require something like an animated gif, which I have no idea how to make, and I do not have any of this ready to run anyway.

I am also still working it out.
It is just something I am poking at on a separate copy of the sourcecode.

Right now Celestia uses a Hipparcos catalog, in the singular.
What I am looking at is using catalogs, in the plural.

Then allowing a script or menu setting to determine which catalog is used.
This could allow for instance, rapidly switching catalogs, for visualizing the difference between catalog sets.
If two of them have a star position cataloged differently, it would appear to move back and forth.

The extra data files would be so that catalog numbers in one, could be indexed to another.
Allowing things like keeping marks on the same stars when switching catalogs.

It is just a visualization aid.
I see it for comparing things like the stars today, with the stars when Cheops was built, or stonehenge erected, that sort of stuff.

It is just an extension of another project of mine which is applying the motion of stars to record them.
Focus on a star, then set the date for 5,000 years ago, and run it forward to 5,000 years from now.
Record this at one decade per second focused on big or little dipper, or something else.
Basically, just see what things looked like, and will look like.
Watch the constellations progress from when something was built, up to now.

Random things like that, looked at just to see what they look like.


Janus.

FSword7
Posts: 12
Joined: 08.09.2017
With us: 1 year 1 month

Post #12by FSword7 » 09.10.2017, 00:25

I am working on CAELI which is Celestia and I am currently working on getting 1 billion objects into CAELI. A script might have to be written to import ESA GAIA .csv files. I am working on it. CAELI is at http://illimitae.com

CAELI is now running 64 bit and a beta for MacOs will be ready most likely oct. 2016.

I tried to download it but it does not exist anymore. Does anyone know where CAELI is?

I am working on my OFS program that is using XHIP database and HYG 3.0 database. They have proper motion data.

I watched Timeline time-travel sci-fi movie and they mentioned constellation difference between 1357 and today. Stars are slowly moving through many years.

Thanks,
Tim

Avatar
selden
Posts: 10245
Joined: 04.09.2002
With us: 16 years 1 month
Location: NY, USA

Post #13by selden » 09.10.2017, 09:45

Some records of the site can be found by using the Internet Archive's "Wayback machine" at

http://web.archive.org/web/*/http://illimitae.com/

Unfortunately, they don't look very helpful.
Selden

FSword7
Posts: 12
Joined: 08.09.2017
With us: 1 year 1 month

Post #14by FSword7 » 09.10.2017, 18:43

Ok, thanks for that reply. I tried to google it but I can't find any source so far.

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #15by john71 » 25.03.2018, 18:48

Following the release of the first catalogue in 2016, ESA's Gaia mission will publish its long-awaited map of more than 1 billion stars in our Galaxy on 25 April 2018.

Links:

http://sci.esa.int/gaia/59944-save-the-date-gaia-s-second-data-release-set-for-25-april/
https://www.cosmos.esa.int/documents/29201/1645651/GDR2_DataModel_draft.pdf/938f48a2-a08d-b63c-67e7-eae778c9a657

Avatar
vpontin
Posts: 13
Joined: 20.04.2017
Age: 25
With us: 1 year 5 months

Post #16by vpontin » 06.04.2018, 23:25

Any news about this? Just saw the news about the release and thought if is possible to import the data entirely on celestia (at least the 64 bit version).

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #17by john71 » 07.04.2018, 08:31

We have to wait until 25 April 2018.

If I'm correct, one billion stars should make a 20 GB dat file in Celestia.

I'm not sure who can convert the original Gaia data to a Celestia dat file, and how can Celestia 1.7 64 bit handle a 20 GB dat file.

But it would be spectacular, no doubt about it.
Last edited by john71 on 07.04.2018, 18:02, edited 1 time in total.

Janus
Posts: 208
Joined: 13.08.2016
With us: 2 years 2 months

Post #18by Janus » 07.04.2018, 16:26

It would probably be wise to avoid trying it if you have less than 24Gb of ram, but 32Gb would be much better.

For my part, I am considering making several subsets of it.
Spheres based in distance starting at 50LY, every 25LY, and see how many that makes.

Or adding a MaxStarDist setting to the cfg file parser to control that on the fly.

I know this will mean some changes to the stars.dat file fomat, and maybe the in program stardb, but I am not making any decisions until I have the data to play with.

Just some random thoughts.


Janus.

Topic author
john71
Posts: 323
Joined: 10.08.2016
With us: 2 years 2 months

Post #19by john71 » 07.04.2018, 18:06

I'm really curious how Gaia Sky will handle the challenge...

https://zah.uni-heidelberg.de/institutes/ari/gaia/outreach/gaiasky/

Avatar
selden
Posts: 10245
Joined: 04.09.2002
With us: 16 years 1 month
Location: NY, USA

Post #20by selden » 07.04.2018, 18:40

Personally, I'd like to be able to load subsets of DR2 which are centered around arbitrary locations, around specific cluster locations, for example, to see how some clusters merge in with "field stars" in that vicinity.

My experience has been that Gaia Sky currently takes quite a while to start up. I dunno why that is. I suggested to them that they should consider having an open forum for discussing its use. It isn't clear if that'll ever happen, though.
Selden


Return to “Ideas & News”

Who is online

Users browsing this forum: 1 guest