The format of the cross index files

Post requests, images, descriptions and reports about work in progress here.
Topic author
LukeCEL
Posts: 71
Joined: 26.09.2017
With us: 1 year 1 month

The format of the cross index files

Post #1by LukeCEL » 07.11.2018, 23:45

Hi everyone. Unrelated to the other stuff I was posting about, but does anyone know exactly what format the cross index files (hdindex.dat and saoindex.dat) are in? The WikiBooks only has a page about Binary Star Files (stars.dat) I don't really know how to use the tools on Github (such as makexindex.cpp), so that won't really be of any help. Thanks in advance.

Avatar
Art Blos M
Posts: 388
Joined: 31.08.2017
Age: 26
With us: 1 year 2 months
Location: Volgodonsk, Rostov Oblast, Russia

Post #2by Art Blos » 08.11.2018, 05:57

I join the question (although a little on another occasion). I asked Greg to compile utilities to edit a binary file, but it didn't work out. :sad:
If I could translate a binary file stars.dat into a text version and vice versa, "Celestia Origin" would be better. Can anyone make a successful compilation and write a detailed manual, how to use ready-made utilities?
Founder and head of the project "Celestia Origin"

Avatar
selden
Posts: 9997
Joined: 04.09.2002
With us: 16 years 2 months
Location: NY, USA

Post #3by selden » 08.11.2018, 13:02

The Perl script buildstardb.pl can be used to create stars.dat

The documentation is in the buildstardb.pl Perl script itself. It is used to create stars.dat from the Hipparcos catalog. See https://sourceforge.net/p/celestia/code/HEAD/tree/trunk/celestia/src/tools/stardb/
Selden

Avatar
gironde M
Posts: 301
Joined: 16.12.2016
Age: 66
With us: 1 year 11 months
Location: Montigny-Les-Metz, Grand Est, Lorraine, Moselle, France

Post #4by gironde » 08.11.2018, 13:28

usually, .dat files are database files. There is not a single program to open them; you need to know the database software for which they were designed.
For hdindex.dat and saoindex.dat we can see with a text editor that the header of the file is CELINDEX. The .dat file reader is inside the Celestia code.

:hi:

Janus
Posts: 216
Joined: 13.08.2016
With us: 2 years 3 months

Post #5by Janus » 08.11.2018, 18:05

Has anyone looked in tools?

src\tools\stardb to be specific.

celdat2txt.cpp might do what you want.

You can also get format information from

src\tools\xindex

buildxindices.pl to be exact.

Then there is stars.txt, which is used to build stars.dat from.

Somewhere in my stuff I have listings of all the database formats from when I made my own.
I can find or reproduce them if needed.
The formats are all simple.

The stars.dat is just a relative position file, a conversion from RaDecDist to XYZ, then saved in binary compact format to speed loading.
In one of my forks I preserved RaDecDist to work with some custom commands I added for looking stars up.
That preconversion is one of the ways early versions of Celestia brought down loading time.

One of the things I have been working toward is conversion directly in the GPU perframe with controlled framerate.

For those who actually care, here is how stars.dat is created.
Exercpt from buildstardb.pl

Spoiler

Code: Select all

sub WriteDat
{
   my $numStars = keys %stars;
   print "Writing databases...\n";

   print "  Writing binary database to $DAT_PATH\n";
   local(*DATFILE);
   open(DATFILE, '>', $DAT_PATH) or die "ERROR: Could not write to $DAT_PATH\n";
   binmode(DATFILE);
   
   print "  Writing text database to $TXT_PATH\n";
   local(*TXTFILE);
   open(TXTFILE, '>', $TXT_PATH) or die "ERROR: Could not write to $TXT_PATH\n";
   
   # write file header
   print DATFILE pack('a8ccL', 'CELSTARS', 0, 1, $numStars);
   print TXTFILE sprintf("%u\n", $numStars);
   
   # write each star
   foreach my $HIP (sort { $a <=> $b } keys %stars)
   {
      my $dist = PlxToDistance($stars{$HIP}{'Plx'});
      my $theta = $stars{$HIP}{'RArad'} + pi;
      my $phi = $stars{$HIP}{'DErad'} - pi / 2;
      my @xyz = (
          $dist * cos($theta) * sin($phi),
          $dist * cos($phi),
         -$dist * sin($theta) * sin($phi)
      );
      my $xc = $eqToCel[0][0] * $xyz[0] + $eqToCel[1][0] * $xyz[1] + $eqToCel[2][0] * $xyz[2];
      my $yc = $eqToCel[0][1] * $xyz[0] + $eqToCel[1][1] * $xyz[1] + $eqToCel[2][1] * $xyz[2];
      my $zc = $eqToCel[0][2] * $xyz[0] + $eqToCel[1][2] * $xyz[1] + $eqToCel[2][2] * $xyz[2];
      my $absMag = AppMagToAbsMag($stars{$HIP}{'Vmag'}, $stars{$HIP}{'Plx'});
      my $spType = ParseSpType($stars{$HIP}{'SpType'});
      print DATFILE pack('LfffsS', $HIP, $xc, $yc, $zc, $absMag * 256, $spType);
      print TXTFILE sprintf("%u  %.9f %+.9f %.6f %.2f %s\n", $HIP,
                            rad2deg($stars{$HIP}{'RArad'}), rad2deg($stars{$HIP}{'DErad'}),
                       $dist, $stars{$HIP}{'Vmag'}, $stars{$HIP}{'SpType'});
   }
   
   close(DATFILE);
   close(TXTFILE);
   
   print "  Wrote a total of $numStars stars.\n";
}


And here is how the CELINDEX files are created.
buildindices.pl

Spoiler

Code: Select all

#!/usr/bin/perl

# buildcrossidx.pl by Andrew Tribick
# version 1.0 - 2008-08-26

open XIDS, '<', 'crossids.txt';

@HDtoHIP  = ();
@HDlevels = ();
@SAOtoHIP = ();

while($curLine = <XIDS>) {
   chomp $curLine;

   # get Hipparcos designation
   $HIP = '';
   $HIP = $2 if($curLine =~ m/(^|:)HIP ([0-9]+)(:|$)/);
   next if($HIP eq ''); # ignore entries which are not Hipparcos stars

   @dList = split(/:/, $curLine);
   for($i = 0; $i <= $#dList; $i++) {
      # no component identifiers on SAO designations - makes things easy...
      $SAOtoHIP{$1} = $HIP if($dList[$i] =~ m/^SAO ([0-9]+)$/);

      # only use HD designations with component identifiers A,J or none
      if($dList[$i] =~ m/^HD ([0-9]+)([AJ]?)$/) {
         $HDnum = $1;
         $Level = HDlevel($2);
         if(!exists($HDtoHIP{$HDnum})) {
            # if this HD number is not already assigned, add it to list
            $HDtoHIP{$HDnum}  = $HIP;
            $HDlevels{$HDnum} = $Level;
         } elsif($Level > $HDlevels{$HDnum}) {
            # otherwise we prefer A over none over J.
            $HDtoHIP{$HDnum}  = $HIP;
            $HDlevels{$HDnum} = $Level;
         }
      }
   }
}

close XIDS;

# write out HD index file
open HDX,  '>', 'hdxindex.dat';
binmode HDX;
print HDX pack('a8S', 'CELINDEX', 0x0100);
foreach $HD (sort { $a <=> $b } keys %HDtoHIP) {
   print HDX pack('LL', $HD, $HDtoHIP{$HD});
}
close HDX;

# write out SAO index file
open SAOX, '>', 'saoxindex.dat';
binmode SAOX;
print SAOX pack('a8S', 'CELINDEX', 0x0100);
foreach $SAO (sort { $a <=> $b } keys %SAOtoHIP) {
   print SAOX pack('LL', $SAO, $SAOtoHIP{$SAO});
}
close SAOX;

# ---END OF MAIN PROGRAM---

sub HDlevel {
   # return a score based on component identifier
   my $d = shift;
   return 0 if($d eq 'J');
   return 1 if($d eq '');
   return 2 if($d eq 'A');
   return -999;
}

I hope that helps.

Keeping the RaDecDist directly in the db rather than computing it is one of those things I was going to recommend.
However, things stalled out, so I went my own way.

Now that onetwothree is getting things moving, I will bring it up at a later time.


Janus.

Avatar
selden
Posts: 9997
Joined: 04.09.2002
With us: 16 years 2 months
Location: NY, USA

Post #6by selden » 08.11.2018, 18:14

The star positions in stars.dat are actually in Celestia's internal xyz coordinates, not ra,dec,distance. This reduces the runtime computation needed to determine their positions relative to the current viewpoint.
Selden

Topic author
LukeCEL
Posts: 71
Joined: 26.09.2017
With us: 1 year 1 month

Post #7by LukeCEL » 08.11.2018, 20:47

Janus wrote:Somewhere in my stuff I have listings of all the database formats from when I made my own.
I can find or reproduce them if needed.
The formats are all simple.

Sorry, I think there was a misunderstanding. Instead of the tools used to make these files (which have pretty scant documentation anyway), I'm requesting this instead.

onetwothree
Posts: 23
Joined: 22.09.2018
With us: 1 month 26 days

Post #8by onetwothree » 11.11.2018, 12:03

Hi LukeCeL, xindex.dat file has very simple format:

HEADER, DATA_CHUNKS[]

The header is:
MAGIC, 8 bytes, CELINDEX
VERSION, 2 bytes, 0x0100

A data chunk is:
STARS_DAT_INDEX, 4 bytes
CATALOG_INDEX, 4 bytes

So for entry #1 in stars.dat we have 0x000000D9 (217) in saoxindex.dat and 0x000001A6 (422) in hdxindex.dat.

All numbers are store in Low Endian format.

But instead of editing existing stars.dat and indexes I'd suggest to use a source file stars.txt and build that files using buildstardb.pl (see src/tools/stardb/readme.txt) and buildxindices.pl (see src/tools/xindex/readme.txt).

But be warned that current version of stars.dat doesn't support spectral class Y.


Return to “Add-on development”

Who is online

Users browsing this forum: 11 guests