Updated stars.dat including Gaia DR2 data (update 2020-02-15)

The home for finished/released addons.
Topic author
ajtribick
Posts: 1788
Joined: 11.08.2003
With us: 16 years 6 months
Location: Switzerland

Updated stars.dat including Gaia DR2 data (update 2020-02-15)

Post #1by ajtribick » 12.01.2020, 13:45

Since I wasn't able to post this on the Celestial Matters forum (possibly a glitch in their website), I thought I'd also post it here.

Here are the Python scripts for generating a stars.dat from the Gaia DR2 data, using the subset of stars cross-matched to HIP2 and TYC2. XHIP is used to fill in the data for missing Hipparcos stars (this includes ones that are missing because the cross-match is incomplete). More details are in the README.

https://github.com/ajtribick/celestia-gaia-stardb

Scripts may take an hour or two to run and require downloading quite a lot of data, so I'd recommend picking up the zip file in the 0.1.2-alpha release instead, unless you want to edit the scripts yourself. I haven't actually tested these in the current release version of Celestia so depending on what's been done, this may or may not work. I also haven't tested it in a 32-bit build either, so no guarantees this won't cause everything to crash with out-of-memory.

Note that I characterise this as "alpha" because of the previously-mentioned cross-match issue. Furthermore, this may well conflict with star definitions in other files, e.g. binaries, extrasolar planet hosts or CHARM2 stellar radii, I have made no attempt to address this.
Last edited by ajtribick on 15.02.2020, 18:17, edited 4 times in total.

Janus
Posts: 469
Joined: 13.08.2016
With us: 3 years 6 months

Post #2by Janus » 12.01.2020, 17:14

Very nice work.
Readable for python, which is great.
Not my favorite language, or one I like, but an improvement over the perl script used to make the original DB.

I would recommend you check your alpha release download name, output.zip is indistinct.
Makes it easy to misplace.
Just like the places that offer setup.exe, with no clue what it sets up in the name, so you have to extract it to know.

I will test the output.zip data once my current debug run for a customer is done.
I think I have finally found the memory corruption they are having, but I have been wrong before.


Janus.

Topic author
ajtribick
Posts: 1788
Joined: 11.08.2003
With us: 16 years 6 months
Location: Switzerland

Post #3by ajtribick » 12.01.2020, 17:34

Good point, I renamed the zip file.

an improvement over the perl script used to make the original DB.
Well, I do like to believe I've gotten a little better at coding over the last decade or so...

Janus
Posts: 469
Joined: 13.08.2016
With us: 3 years 6 months

Post #4by Janus » 12.01.2020, 18:20

Just to be clear, I meant no disrespect.
Most python reads like poly macro expanded templated C/C++ code to me, this doesn't.
I dislike & distrust layered macros, and am wary of templates, though I admit the latter have their place.

The over use of shorthand renders so many scripts unreadable to anyone who doesn't code just like the author.
Which you avoided, and is greatly appreciated.
I am not very good with python, but I could follow it.

I am of the firm opinion that sourcecode is for humans, and the rest is the compilers problem.
If you can't scan the sourcecode like a book looking for things, then something is wrong.

You might laugh, but I tend to use php for stuff like this, and if I need a quick readout, I then send it on to a browser where js is used for display.
I like that php allows you to set a hard timeout in the config file so a hung script is not an issue.
I also find php more readable, I just get along with it.

A quick question.
Does the Gaia data include stellar motion?
If so, how hard is it to calculate XYZ movement from it?
I know this limits the time the movement is valid for, but I can live with a few million years, it will do.

I ask because I am playing with modifications to star::getposition & star::getorbitbarycenter to use the time to compute a position.
The result is multiplying the time in years from 2000, then adding the result to give a time adjusted position.
The purpose is set a time in celestia, then use a script to change the time ten years per second, and watch the stars move.
I will include a time ratio settable from script so you can things like make a year equal to a thousand, to allow looking at stars from a million years ago, or for a million years from now.

Tweaking the database and its reader is simple, I have done so before.
I dislike perl, a LOT!, but I can use it.
I made a DB with double star positions, another where I added Ra/Dec/Dist, and some others.
Using the python scripts will be easier.

Though I can eventually work out the math, it is much simpler to ask if anyone else has already done the work.
I know about the astronexus hyg-databse, but the gaia data would be more accurate.


Janus.

Avatar
Sirius_Alpha
Posts: 69
Joined: 21.03.2019
With us: 11 months

Post #5by Sirius_Alpha » 12.01.2020, 18:34

This is absolutely astounding. The structure of the local Galaxy is very visible.

You can see clusters very easily now from afar. The older poor-accuracy parallaxes and photometric distances were so bad that you really couldn't see clusters beyond the Pleaides. But now all kinds of detail and clusters pop out. There's still some smearing due to the parallax uncertainties, but it's nowhere near as bad as before.

Clusters.png
Several galactic clusters circled in red.


Below is a view of the Praesepe Cluster from the Pr0201 system. We always kinda knew these sorts of views were possible from known extrasolar planetary systems but before this it's not like we could approximate them very well.

Pr0201.png
Praesepe Cluster from Pr0201.

Avatar
SevenSpheres
Posts: 211
Joined: 08.10.2019
With us: 4 months 13 days

Post #6by SevenSpheres » 12.01.2020, 18:53

I've added these files in this branch of Celestia's GitHub repository (it previously added LukeCEL's star database files). Should I make a pull request from that branch?
My Addons: viewtopic.php?f=23&t=19978 • Contributing to development as much as I can...
PC specs: HP desktop, Windows 10 Home 64-bit, Intel Core i7-7700 (3.60 GHz), 16 GB RAM, 1 TB HDD + 128 GB SSD, Intel HD Graphics 630
Celestia versions: 1.5.1, 1.6.1, 1.6.2, and multiple 1.7.0 builds

onetwothree
Developer
Posts: 384
Joined: 22.09.2018
With us: 1 year 4 months

Post #7by onetwothree » 12.01.2020, 19:18

SevenSpheres wrote:I've added these files in this branch of Celestia's GitHub repository (it previously added LukeCEL's star database files). Should I make a pull request from that branch?

Maybe not yet until cross-index issue is resolved.

Topic author
ajtribick
Posts: 1788
Joined: 11.08.2003
With us: 16 years 6 months
Location: Switzerland

Post #8by ajtribick » 12.01.2020, 19:30

Janus wrote:A quick question.
Does the Gaia data include stellar motion?
If so, how hard is it to calculate XYZ movement from it?
I know this limits the time the movement is valid for, but I can live with a few million years, it will do.
Gaia DR2 does contain proper motions and radial velocities. This information is not present in the output files (the generation scripts do use the HIP2 proper motions to update the positions of the non-Gaia stars to the Gaia DR2 epoch though).

SevenSpheres wrote:Should I make a pull request from that branch?
You should keep at least the reference to the source data (see my repository's README.md) around if you're going to merge this add-on into the code base, and provide credit in the appropriate Celestia README files.

Avatar
LukeCEL
Posts: 219
Joined: 26.09.2017
With us: 2 years 4 months

Post #9by LukeCEL » 12.01.2020, 19:39

Awesome, thank you!

Janus
Posts: 469
Joined: 13.08.2016
With us: 3 years 6 months

Post #10by Janus » 16.01.2020, 06:17

@ajtribick

Nice set of scripts.
However, small issue.

They require a gaia account to do the queries.
Something you should warn people about.


Janus.

Topic author
ajtribick
Posts: 1788
Joined: 11.08.2003
With us: 16 years 6 months
Location: Switzerland

Post #11by ajtribick » 16.01.2020, 07:02

Janus wrote:However, small issue.

They require a gaia account to do the queries.
Something you should warn people about.
Indeed, that's why it's the second bullet point in the Prerequisites section of the README.md that's displayed on the main page of the repository on GitHub.

Topic author
ajtribick
Posts: 1788
Joined: 11.08.2003
With us: 16 years 6 months
Location: Switzerland

Post #12by ajtribick » 15.02.2020, 12:19

Fixed an issue with duplicate stars which resulted from not accounting for duplicate entries in the SAO cross match. The v0.1.2-alpha release is linked in the first post of the thread.

Avatar
Art Blos M
Moderator
Posts: 688
Joined: 31.08.2017
Age: 27
With us: 2 years 5 months
Location: Volgodonsk, Rostov Oblast, Russia

Post #13by Art Blos » 15.02.2020, 17:58

ajtribick wrote:I also haven't tested it in a 32-bit build either, so no guarantees this won't cause everything to crash with out-of-memory.
You underestimate 32-bit systems. Your database not big enough to crash with out-of-memory. :smile:

Can you tell me the exact number of stars?
Founder and head of the project "Celestia Origin"

Avatar
SevenSpheres
Posts: 211
Joined: 08.10.2019
With us: 4 months 13 days

Post #14by SevenSpheres » 15.02.2020, 18:09

Art Blos wrote:Can you tell me the exact number of stars?

I can tell you: there are 2,463,589 stars according to Celestia's console.
My Addons: viewtopic.php?f=23&t=19978 • Contributing to development as much as I can...
PC specs: HP desktop, Windows 10 Home 64-bit, Intel Core i7-7700 (3.60 GHz), 16 GB RAM, 1 TB HDD + 128 GB SSD, Intel HD Graphics 630
Celestia versions: 1.5.1, 1.6.1, 1.6.2, and multiple 1.7.0 builds

Avatar
Art Blos M
Moderator
Posts: 688
Joined: 31.08.2017
Age: 27
With us: 2 years 5 months
Location: Volgodonsk, Rostov Oblast, Russia

Post #15by Art Blos » 15.02.2020, 18:15

SevenSpheres wrote:I can tell you: there are 2,463,589 stars according to Celestia's console.
Cool! In LukeCEL's version was 2 448 220. I.e +15 369 stars. :clap:

Added after 3 minutes 30 seconds:
I hope the accuracy of the coordinates is also at a height compared to its version. At least a little bit. :smile:
Founder and head of the project "Celestia Origin"

Topic author
ajtribick
Posts: 1788
Joined: 11.08.2003
With us: 16 years 6 months
Location: Switzerland

Post #16by ajtribick » 15.02.2020, 18:19

Good to know it works on 32-bit! One way of finding out how many stars there are in a stars.dat file is to take the file size in bytes, subtract 14 (to account for the file header, which incidentally also contains the number of stars) then divide by 20.

Avatar
Art Blos M
Moderator
Posts: 688
Joined: 31.08.2017
Age: 27
With us: 2 years 5 months
Location: Volgodonsk, Rostov Oblast, Russia

Post #17by Art Blos » 15.02.2020, 18:23

ajtribick wrote:Good to know it works on 32-bit! One way of finding out how many stars there are in a stars.dat file is to take the file size in bytes, subtract 14 (to account for the file header, which incidentally also contains the number of stars) then divide by 20.
This formula works, thanks.

Are you planning to increase the number of stars?
Founder and head of the project "Celestia Origin"

Avatar
jujuapapa
Posts: 104
Joined: 24.06.2018
With us: 1 year 7 months
Location: Western E.U.

Post #18by jujuapapa » 16.02.2020, 09:43

Great work ajtribick ! :clap:

=> same question than ArtBlos...

#
### UPDATE
#


I saw differences between GrantHutchinson's catalog (found on celestiamotherlode :
http://celestiamotherlode.net/catalog/show_addon_details.php?addon_id=1114) and gaia catalog !

First example : HIP 33562
extended : D=74.973 LY & ST= K0V
gaia : D=0.58163 Kpc & ST=G5IV


2nd example : TYC 7630-1045-1
Ext : D=110.53 LY & ST=K8V
gaia : D=1.3803 Kpc & ST=K5

Which are true ? :eek:

Anyone has seen problems ?
PC : Intel Core i9-9900K (3.6 GHz) , Chipset Z390 Exp, RAM 32 Go DDR4 3000 Mhz, SSD M.2 512 Go + HDD 3 To, MSI GeForce RTX 2080 8Go

=> It is by doubting everything that one approaches the truth !

Avatar
Art Blos M
Moderator
Posts: 688
Joined: 31.08.2017
Age: 27
With us: 2 years 5 months
Location: Volgodonsk, Rostov Oblast, Russia

Post #19by Art Blos » 16.02.2020, 14:32

jujuapapa wrote:I saw differences between GrantHutchinson's catalog
This catalog was created back in 2005, and also had certain problems with duplicates. Leave it in the past.
Founder and head of the project "Celestia Origin"

Topic author
ajtribick
Posts: 1788
Joined: 11.08.2003
With us: 16 years 6 months
Location: Switzerland

Post #20by ajtribick » 16.02.2020, 20:48

jujuapapa wrote:I saw differences between GrantHutchinson's catalog (found on celestiamotherlode :
http://celestiamotherlode.net/catalog/show_addon_details.php?addon_id=1114) and gaia catalog !
This is expected. That catalogue (originally by Pascal Hartmann, converted to Celestia 1.4.0 format by Grant Hutchison) is based on the All Sky Compiled Catalogue (ASCC) and uses distances estimated from the B-V colour index and the apparent magnitude. This was required because most of the stars did not have measured parallaxes.

I also use the ASCC to provide magnitudes for Tycho-2 stars, but I'm not using photometric distance estimates: instead I'm taking distances from either the Gaia distance catalogue (preferred), distances based on cluster membership in XHIP (second choice), or computed distances based on parallaxes in XHIP (third choice). Note that in this third case I'm not using 1/parallax as the distance, I'm using the technique in Astraatmadja & Bailer-Jones (2016), which is consistent with the Gaia distance catalogue.

Spectral types are different because I'm again using different sources: in the case of HIP 33562, ASCC and HIP1 give a spectral type of K0. I'm using the XHIP spectral type which is G5IV. For Tycho stars, I use the Tycho-2 spectral type catalogue. For stars which don't have spectral type entries I'm using a spectral type from the estimated temperature: Gaia (first choice), the Tycho effective temperature catalogue (second choice) or estimating it using the colour indices (B-V, V-I, V-K, J-K and H-K, reddening is not taken into account). I don't attempt to estimate luminosity classes.

Art Blos wrote:Are you planning to increase the number of stars?
No. Celestia's current approach of loading the entire database and building the octree at startup doesn't scale to handling the Gaia catalogue. It would also require changing Celestia's identifier handling: currently the HIP/encoded TYC identifier (which incidentally originated with the Pascal Hartmann catalogue) is treated as fundamental but that's not going to work with the larger catalogue as the additional stars will not be HIP/TYC stars. Gaia identifiers are 64-bit integers so you can't stuff them into the 32-bit field, in any case you don't want to use those as the fundamental identifier because you need to include non-Gaia stars, notably the stars that are too bright for Gaia to measure).


Return to “Add-on releases”

Who is online