How to use the generator utility

The generator tool allows to generate synthetic photometry in a set of available systems from the input internally-calibrated continuously-represented mean spectra.

Information about the units of the variables used in this tutorial can be found here.

To import the tool and the required PhotometricSystem class:

from gaiaxpy import generate, PhotometricSystem

To see what systems are available:

PhotometricSystem.get_available_systems()

'DECam, Els_Custom_W09_S2, Euclid_VIS, Gaia_2, Gaia_DR3_Vega, Halpha_Custom_AB, H_Custom, Hipparcos_Tycho, HST_ACSWFC, HST_HUGS_Std, HST_WFC3UVIS, HST_WFPC2, IPHAS, JKC, JKC_Std, JPAS, JPLUS, JWST_NIRCAM, LSST, PanSTARRS1, PanSTARRS1_Std, Pristine, SDSS, SDSS_Std, Sky_Mapper, Stromgren, Stromgren_Std, WFIRST'

For a number of systems a standardisation procedure has been carried out to minimise zero point differences and/or trends as a function of colour (see Montegriffo et al. 2022). These are available in GaiaXPy by using the _Std version of the desired photometric system.

Basic usage

The generator returns a single output:

1. A DataFrame of all synthetic photometry results. The DataFrame contains the magnitudes, fluxes and flux errors for each band in the system.

# Path to file with XP CONTINUOUS RAW data (csv, ecsv, fits, or xml)
f = '/path/to/XP_CONTINUOUS_RAW.xml'
# Select a photometric system
phot_system = PhotometricSystem.Gaia_DR3_Vega
synthetic_photometry = generate(f, photometric_system=phot_system)
synthetic_photometry

Done! Output saved to path: ./output_synthetic_photometry.xml                                                         0/2 [00:00<?, ?spec/s]

	source_id	GaiaDr3Vega_mag_G	GaiaDr3Vega_mag_BP	GaiaDr3Vega_mag_RP	GaiaDr3Vega_flux_G	GaiaDr3Vega_flux_BP	GaiaDr3Vega_flux_RP	GaiaDr3Vega_flux_error_G	GaiaDr3Vega_flux_error_BP	GaiaDr3Vega_flux_error_RP
0	5853498713190525696	8.953604	11.353152	7.562385	6.646585e-15	1.181787e-15	1.225988e-14	2.687982e-18	1.330194e-18	4.299604e-18
1	5762406957886626816	13.172242	13.056574	13.400146	1.365033e-16	2.461334e-16	5.667372e-17	5.450830e-20	1.521506e-19	2.870977e-20

A list of PhotometricSystem objects is also accepted.

# Create a list of photometric systems
phot_system_list = [PhotometricSystem.Gaia_DR3_Vega, PhotometricSystem.JKC, PhotometricSystem.Pristine]
synthetic_photometry = generate(f, photometric_system=phot_system_list)
synthetic_photometry

Done! Output saved to path: ./output_synthetic_photometry.xml                                                         0/2 [00:00<?, ?spec/s]

	source_id	GaiaDr3Vega_mag_G	GaiaDr3Vega_mag_BP	GaiaDr3Vega_mag_RP	GaiaDr3Vega_flux_G	GaiaDr3Vega_flux_BP	GaiaDr3Vega_flux_RP	GaiaDr3Vega_flux_error_G	GaiaDr3Vega_flux_error_BP	GaiaDr3Vega_flux_error_RP	...	Jkc_flux_R	Jkc_flux_I	Jkc_flux_error_U	Jkc_flux_error_B	Jkc_flux_error_V	Jkc_flux_error_R	Jkc_flux_error_I	Pristine_mag_CaHK	Pristine_flux_CaHK	Pristine_flux_error_CaHK
0	5853498713190525696	8.953604	11.353152	7.562385	6.646585e-15	1.181787e-15	1.225988e-14	2.687982e-18	1.330194e-18	4.299604e-18	...	3.849211e-15	1.305855e-14	5.966184e-18	7.583555e-19	1.468439e-18	2.945754e-18	7.269074e-18	13.818329	2.162462e-16	9.483388e-18
1	5762406957886626816	13.172242	13.056574	13.400146	1.365033e-16	2.461334e-16	5.667372e-17	5.450830e-20	1.521506e-19	2.870977e-20	...	1.062562e-16	5.024040e-17	2.702577e-18	3.499559e-19	1.437515e-19	8.295570e-20	3.187286e-20	13.016468	4.525774e-16	4.957204e-18

2 rows × 28 columns

Input types

The input does not have to be a path to a file. There are other options: a pandas DataFrame, an ADQL query or a list of source IDs.

Passing a DataFrame

import pandas as pd

f = '/path/to/XP_CONTINUOUS_RAW.csv'
df = pd.read_csv(f) # At this point, the values in the DataFrame can be edited if the user wishes to do so.

synthetic_photometry = generate(df, photometric_system=PhotometricSystem.Els_Custom_W09_S2)
synthetic_photometry

Done! Output saved to path: ./output_synthetic_photometry.csv                                                         0/2 [00:00<?, ?spec/s]

	source_id	ElsCustomW09S2_mag_Halpha	ElsCustomW09S2_mag_Hbeta	ElsCustomW09S2_mag_O3	ElsCustomW09S2_mag_CHalpha	ElsCustomW09S2_mag_CHbeta	ElsCustomW09S2_mag_CO3	ElsCustomW09S2_mag_r	ElsCustomW09S2_mag_i	ElsCustomW09S2_flux_Halpha	...	ElsCustomW09S2_flux_r	ElsCustomW09S2_flux_i	ElsCustomW09S2_flux_error_Halpha	ElsCustomW09S2_flux_error_Hbeta	ElsCustomW09S2_flux_error_O3	ElsCustomW09S2_flux_error_CHalpha	ElsCustomW09S2_flux_error_CHbeta	ElsCustomW09S2_flux_error_CO3	ElsCustomW09S2_flux_error_r	ElsCustomW09S2_flux_error_i
0	5853498713190525696	9.322986	12.011328	11.772323	9.996499	12.418606	11.171822	10.234842	7.648619	3.623145e-15	...	2.011792e-15	1.121586e-14	1.406098e-17	3.498787e-18	3.658347e-18	1.145484e-17	4.089501e-18	3.986050e-18	2.963246e-18	7.769428e-18
1	5762406957886626816	13.351317	13.148943	13.131455	13.267573	13.018521	13.134297	13.261008	13.393297	8.866517e-17	...	1.239130e-16	5.648866e-17	2.775161e-19	6.395538e-19	5.618435e-19	4.977890e-19	1.078378e-18	4.220455e-19	1.054331e-19	3.647875e-20

2 rows × 25 columns

Running a query

query_input = "select TOP 2 source_id from gaiadr3.gaia_source where has_xp_continuous = 'True'"
phot_system_list = [PhotometricSystem.Stromgren, PhotometricSystem.Stromgren_Std]
synthetic_photometry = generate(query_input, photometric_system=phot_system_list)
synthetic_photometry

INFO: Query finished. [astroquery.utils.tap.core]
Done! Output saved to path: ./output_synthetic_photometry.csv                                                         0/2 [00:00<?, ?spec/s]

	source_id	Stromgren_mag_u	Stromgren_mag_v	Stromgren_mag_b	Stromgren_mag_y	Stromgren_flux_u	Stromgren_flux_v	Stromgren_flux_b	Stromgren_flux_y	Stromgren_flux_error_u	...	Stromgren_flux_error_y	StromgrenStd_mag_v	StromgrenStd_mag_b	StromgrenStd_mag_y	StromgrenStd_flux_v	StromgrenStd_flux_b	StromgrenStd_flux_y	StromgrenStd_flux_error_v	StromgrenStd_flux_error_b	StromgrenStd_flux_error_y
0	2851858288640	12.829329	13.254145	12.874594	12.483260	2.368255e-16	3.683510e-16	4.067827e-16	3.671945e-16	4.030506e-18	...	2.028857e-18	13.414131	12.87440	12.502047	3.697156e-16	4.064809e-16	3.673004e-16	1.765563e-18	1.592358e-18	2.175534e-18
1	7834020517760	18.146721	18.239382	17.783585	17.395128	1.767956e-18	3.733937e-18	4.423500e-18	3.982436e-18	6.105945e-19	...	1.162250e-19	18.379352	17.77506	17.412889	3.817502e-18	4.454266e-18	3.987349e-18	1.598756e-19	1.089533e-19	1.252491e-19

2 rows × 22 columns

Passing a list

A list of source IDs can be passed to the generator as the first argument. The generator will then query the Archive for these objects.

sources_list = ['6030020833890693248', 6030064028385961344] # The source IDs can be string or long.
synthetic_photometry = generate(sources_list, photometric_system=PhotometricSystem.JKC_Std)
synthetic_photometry

Done! Output saved to path: ./output_synthetic_photometry.csv                                                         0/2 [00:00<?, ?spec/s]

	source_id	JkcStd_mag_U	JkcStd_mag_B	JkcStd_mag_V	JkcStd_mag_R	JkcStd_mag_I	JkcStd_flux_U	JkcStd_flux_B	JkcStd_flux_V	JkcStd_flux_R	JkcStd_flux_I	JkcStd_flux_error_U	JkcStd_flux_error_B	JkcStd_flux_error_V	JkcStd_flux_error_R	JkcStd_flux_error_I
0	6030020833890693248	18.403455	17.476459	15.704760	14.528387	13.199215	1.990939e-18	6.622414e-18	1.921569e-17	3.450842e-17	6.192043e-17	1.659116e-19	4.778960e-20	3.725142e-20	3.929187e-20	3.980868e-20
1	6030064028385961344	17.628197	19.025791	17.914239	17.133965	16.512081	4.065943e-18	1.589584e-18	2.511103e-18	3.131078e-18	2.928785e-18	5.736788e-19	2.767727e-20	1.521533e-20	1.644628e-20	1.049679e-20

Advanced usage

The generator offers further functionality via the optional parameters error_correction, and truncation.

Error correction

An analysis based on a dataset of sources for which the available observations per source were randomly split in two sets thus leading to the generation of two mean spectra for each object, has shown that errors on synthetic photometry are somewhat underestimated.

This same dataset has been used to compute empirical corrections to the uncertainties (see Montegriffo et al. 2022). These corrections are available in GaiaXPy and can be activated when generating synthethic photometry.

# To apply the error correction, first create a list of systems.
phot_system_list = [PhotometricSystem.Euclid_VIS, PhotometricSystem.Els_Custom_W09_S2]
synthetic_phot_corrected = generate(df, photometric_system=phot_system_list, error_correction=True)
synthetic_phot_corrected
# A warning is raised for each of the systems that cannot be corrected. No changes are made to these systems.

Done! Output saved to path: ./output_synthetic_photometry.csv                                                         0/2 [00:00<?, ?syst/s]

	source_id	EuclidVis_mag_VIS	EuclidVis_flux_VIS	EuclidVis_flux_error_VIS	ElsCustomW09S2_mag_Halpha	ElsCustomW09S2_mag_Hbeta	ElsCustomW09S2_mag_O3	ElsCustomW09S2_mag_CHalpha	ElsCustomW09S2_mag_CHbeta	ElsCustomW09S2_mag_CO3	...	ElsCustomW09S2_flux_r	ElsCustomW09S2_flux_i	ElsCustomW09S2_flux_error_Halpha	ElsCustomW09S2_flux_error_Hbeta	ElsCustomW09S2_flux_error_O3	ElsCustomW09S2_flux_error_CHalpha	ElsCustomW09S2_flux_error_CHbeta	ElsCustomW09S2_flux_error_CO3	ElsCustomW09S2_flux_error_r	ElsCustomW09S2_flux_error_i
0	5853498713190525696	8.229913	8.775556e-15	1.483339e-17	9.322986	12.011328	11.772323	9.996499	12.418606	11.171822	...	2.011792e-15	1.121586e-14	1.752444e-17	5.387092e-18	5.212650e-18	1.555349e-17	4.705509e-18	5.991718e-18	6.915351e-18	3.364387e-17
1	5762406957886626816	13.318073	8.091163e-17	1.059743e-19	13.351317	13.148943	13.131455	13.267573	13.018521	13.134297	...	1.239130e-16	5.648866e-17	2.682293e-19	7.579193e-19	6.293417e-19	5.549550e-19	1.154106e-18	4.808390e-19	1.619426e-19	8.789338e-20

2 rows × 28 columns

Truncation

The source mean BP/RP spectrum is described as a combination of basis functions. Particularly for faint sources or sources with a low number of observations, it is useful to represent the spectrum using a smaller set of basis functions to avoid higher-order bases fitting the noise in the observed data.

The truncation parameter is a boolean that toggles truncation of the set of bases and is False by default. In the case of the generator, this parameter is available from GaiaXPy 2.1.4 onwards.

To apply truncation:

synthetic_photometry = generate(f, photometric_system=phot_system, truncation=True)

Done! Output saved to path: ./output_synthetic_photometry.csv                                                         0/2 [00:00<?, ?spec/s]

Additional parameters

Additional arguments can be passed to the generator.

These are:

1. output_path
2. output_file
3. output_format
4. save_file

Three parameters: output_path, output_file, and output_format define the entire path of the resulting file.

The default output path is the current path. If the given output path does not exist, it will be created.

The default output file name is output_spectra.

The default output format is the format of the input file (i.e. if the extension of the input file is fits, then the output file will be a FITS file by default), or CSV in any other case (DataFrame, ADQL query or list).

NOTE: If a file with the same path and name already exists, it will be AUTOMATICALLY OVERWRITTEN.

synthetic_photometry = generate(f, photometric_system=phot_system, output_path='/path/to', output_file='my_file', output_format='xml')

Done! Output saved to path: /path/to/my_file.xml                                                             0/2 [00:00<?, ?spec/s]

The additional parameter save_file is a boolean that tells the program whether to save the results or not. If output_file is given but save_file is set to False, a warning will be raised.

synthetic_photometry = generate(f, photometric_system=phot_system, output_file='my_file', save_file=False)

Reading input file... Done!

UserWarning: The argument 'output_file' was provided, but 'save_file' is set to False. Set 'save_file' to True to store the output of the function.

                                                                                                                      0/2 [00:00<?, ?spec/s]