Pipeline¶
This is a tutorial showcasing VASCA’s pipeline flow on a simple example. We will go
through all the steps equivalent to what is done in vasca_pipe.run_from_file.
This is the same function that is called when starting the pipeline from the CLI using vasca-pipe.
The goal is to create a VASCA Region from multiple GALEXField for which we
download the raw data online from MAST.
We apply quality cuts and do source clustering followed by variability analysis.
For this tutorial we are interested in the near-ultraviolet (NUV) observations by GALEX. We are going to look at neighboring/overlapping fields all of which contain the location of famous Tidal Disruption Event PS1-10jh discovered by Pan-STARRS and observed by GALEX in 2010.
GALEX sky map with field footprints of observations around the location of PS1-10jh ( purple crosshair). Screenshot from MAST Portal¶
General Configuration¶
The standard pipeline processing starts by reading a yaml file. To keep this tutorial simple, we are going to introduce parts of the configuration step by step at the point where they are required in the pipeline flow.
Note
An important premise of the configuration is that each parameter needs to be configured explicitly. This means even default values are specified all the time. This is a design decision purposefully made in order to ensure transparent and complete configurations. As a result, all possible parameters are always included when looking at configuration file.
Let’s begin with the general section. Here, basic information and functionality is
configured. The name of the pipeline run specifies also the name of directory in
which all results will be stored. The location of output directory is at out_dir_base
relative to the root directory of the package.
VASCA uses the powerful logging system provided by loguru.
The configuration specifies the log_level,
which we are going to set to debugging mode here. By default VASCA is going to save
all logging messages in a file stored in the output directory. log_file specifies
the name of that file, while default tells the pipeline to use a default name.
Parallel processing of the field-level analysis can be enabled when setting the number
of CPU threads nr_cpus > 1.
VASCA can include field-averaged reference data, if such data is available additional
to the visit-level data from the instruments mission pipeline. To save memory/storage
and computation time it is configurable wether to include reference sources in the
final Region-file (save_ref_srcs) and to repeat already processed fields that
are included in the region (run_fields).
# Dictionary holding the configuration
config = {}
# General section of the configuration
config["general"] = {
"name": "simple_pipe",
"out_dir_base": "docs/tutorial_resources/vasca_pipeline",
"log_level": "DEBUG",
"log_file": "default",
"nr_cpus": 3,
"save_ref_srcs": True,
"run_fields": True,
}
Tip
In case the output location is on a remote server and multiple users should be
able to edit the data, i.e., reprocess data using an updated configruation, then
one needs to manage user access priviliges. For convenience this can be done
using umask:
import os
os.umask("0o003", 0)
This will grand user and group full permissions. The setting is only persistant throughout the Python session.
The pipeline begins with some prerequisites including enabling logging and creating the output directory
import sys
from loguru import logger
from pathlib import Path
from importlib.resources import files
# Setup output directory
out_dir_base = Path(files("vasca").parent / config["general"]["out_dir_base"])
out_dir_base.mkdir(parents=True, exist_ok=True)
pipe_dir = out_dir_base / config["general"]["name"]
pipe_dir.mkdir(parents=True, exist_ok=True)
# Path to log file
log_file = (
pipe_dir / f'{config["general"]["name"]}.log'
) # Pipeline name is used by default
# Logger configuration, both to stdout and .log file
log_cfg = {
"handlers": [
{
"sink": sys.stdout,
"format": "<green>{time:YYYY-MM-DD HH:mm:ss.SSSS}</green> "
"<cyan>{name}</cyan>:<cyan>{line}</cyan> |"
"<level>{level}:</level> {message}",
"level": config["general"]["log_level"],
"colorize": True,
"backtrace": True,
"diagnose": True,
},
],
}
logger.configure(**log_cfg) # Set config
logger.add(log_file) # Add file logger
logger.enable("vasca") # Enable logging
# Some initial logging messages
logger.info(f"Runing '{config['general']['name']}'")
logger.debug(f"Config. file: {log_file}")
2024-11-08 14:07:53.6459 __main__:38 |INFO: Runing 'simple_pipe'
2024-11-08 14:07:53.6470 __main__:39 |DEBUG: Config. file: /opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/simple_pipe.log
Resources¶
Next is the section about resource handling. This specifies the method used to load
(load_method) field data and which data products should be included (tables,
tables plus visit-level images, or only metadata load_products). Additionally
the coadd_exists flag tells the pipeline wether it can expect co-added (field-
averaged) data. Finally, field_kwargs allows to pass parameters directly to
the init function of a field class.
Here we are going to initialize fields from local raw data, if present. Else the
required data is downloaded from MAST. All data products will be included including
co-add data. Using the ResourceManager we can tell the field class where to
locate the data.
from vasca.resource_manager import ResourceManager
# Get the data locations using ResourceManager
rm = ResourceManager()
data_path = rm.get_path("docs_resources", "vasca")
visits_data_path = rm.get_path("gal_visits_list", "vasca")
# Resource section of the configuration
config["resources"] = {
"load_method": "MAST_LOCAL",
"load_products": "ALL",
"coadd_exists": True,
"field_kwargs": {
"data_path": data_path,
"visits_data_path": visits_data_path,
},
}
Observations¶
The observations section of the configuration is responsible for configuring
which combination of instrument (observatory) and filter (obs_filter)to
load data. Also it specifies the exact list of fields to load (obs_field_ids).
In a later step we will also add here the selection parameters (selection) used
for the quality cuts on the data and the field-level clustering settings
(cluster_det).
config["observations"] = [
{
"observatory": "GALEX",
"obs_filter": "NUV",
"obs_field_ids": [
3880803393752530944, # MISGCSN2_10493_0117
2529969795944677376, # ELAISN1_09
2597664528044916736, # PS_ELAISN1_MOS15
],
# "cluster_det": {},
# "selection": {},
},
# More instruments/filters...
]
Find below the visit metadata about the fields under investigation.
Show code cell source
from astropy.table import Table
df_gal_visits = (
Table.read(visits_data_path)
.to_pandas()
.apply(lambda x: x.str.decode("utf-8") if x.dtype == "O" else x)
)
query = f"ParentImgRunID in {list(config['observations'][0]['obs_field_ids'])}"
df_select = df_gal_visits.query(query)
show(
df_select,
classes="display nowrap compact",
scrollY="300px",
scrollCollapse=True,
paging=False,
columnDefs=[{"className": "dt-body-left", "targets": "_all"}],
)
| RATileCenter | DECTileCenter | survey | nexptime | fexptime | imgRunID | ParentImgRunID | joinID | tileNum | specTileNum | source | nPhotoObjects | nPhotoVisits | PhotoObsDate | spectra | nSpectra | nSpectraVisits | SpecObsDate | visitNum | subvis | minPhotoObsDate | maxPhotoObsDate | minSpecObsDate | maxSpecObsDate | PhotoObsDate_MJD | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Loading ITables v2.2.3 from the init_notebook_mode cell...
(need help?) |
In the next step we will initialize a VASCA Region with all fields sequentially.
load_from_config is a convenience function that acts as an interface between the
region object and field-specific loading functions. This will downloads the data from
MAST, it will detect if the data is already present on disc and loads the cashed
files. To safe compute time later, a VASCA-field file is written to the download
location so that one can use this file instead of creating a new field from raw data.
This will be used during the field-level processing.
from vasca.region import Region
rg = Region.load_from_config(config)
Show code cell output
2024-11-08 14:07:55.3413 vasca.tables:156 |DEBUG: Adding table 'region:tt_fields'
2024-11-08 14:07:55.3441 vasca.region:71 |DEBUG: Loading fields from config file
2024-11-08 14:07:55.3448 vasca.field:747 |INFO: Loading field '3880803393752530944' with method 'MAST_LOCAL' for filter 'NUV' and load_products 'ALL'.
2024-11-08 14:07:55.3460 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944'
2024-11-08 14:07:55.3466 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca/vasca/vasca/test/resources/GALEX_visits_list.fits'
2024-11-08 14:07:55.3472 vasca.field:888 |DEBUG: Downloading archive field info and saving to /home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/MAST_3880803393752530944_NUV_coadd.fits.
2024-11-08 14:07:55.8831 vasca.field:908 |DEBUG: Constructing 'tt_fields'.
2024-11-08 14:07:55.8863 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_fields'
2024-11-08 14:07:55.8881 vasca.field:957 |DEBUG: Loading GALEX visit info :3880803393752530944 NUV None
2024-11-08 14:07:55.8888 vasca.field:988 |DEBUG: Reading archive visit info from cashed file '/home/runner/work/vasca/vasca/vasca/test/resources/GALEX_visits_list.fits'
2024-11-08 14:07:55.9205 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_visits'
2024-11-08 14:07:55.9224 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU3880803393752530944', , S
2024-11-08 14:07:55.9229 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'MISGCSN2_10493_0117', None, S
2024-11-08 14:07:55.9236 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-08 14:07:55.9243 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-08 14:07:55.9249 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-08 14:07:55.9259 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-08 14:07:55.9264 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-08 14:07:55.9270 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-08 14:07:55.9277 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-08 14:07:55.9298 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-08 14:07:55.9302 vasca.field:1076 |DEBUG: Reading archive field info from cashed file '/home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/MAST_3880803393752530944_NUV_coadd.fits'
2024-11-08 14:07:55.9459 vasca.field:1081 |DEBUG: Downloading archive data products list and saving to /home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/MAST_3880803393752530944_NUV_data.fits.
2024-11-08 14:07:56.4569 vasca.field:1164 |DEBUG: Downloading archive data products. Manifest saved to /home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/MAST_3880803393752530944_NUV_down.ecsv.
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR7/pipe/01-vsn/44763-MISGCSN2_10493_0117/d/00-visits/0001-img/07-try/MISGCSN2_10493_0117_0001-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803256112250880/MISGCSN2_10493_0117_0001-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR7/pipe/01-vsn/44763-MISGCSN2_10493_0117/d/00-visits/0001-img/07-try/MISGCSN2_10493_0117_0001-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803256112250880/MISGCSN2_10493_0117_0001-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR7/pipe/01-vsn/44763-MISGCSN2_10493_0117/d/01-main/0007-img/07-try/MISGCSN2_10493_0117-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803393752530944/MISGCSN2_10493_0117-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR7/pipe/01-vsn/44763-MISGCSN2_10493_0117/d/01-main/0007-img/07-try/MISGCSN2_10493_0117-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803393752530944/MISGCSN2_10493_0117-xd-mcat.fits.gz ...
[Done]
2024-11-08 14:08:01.9655 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_detections'
2024-11-08 14:08:01.9707 vasca.field:1391 |DEBUG: Constructed 'tt_detections'.
2024-11-08 14:08:01.9876 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_coadd_detections'
2024-11-08 14:08:01.9923 vasca.field:1396 |DEBUG: Constructed 'tt_coadd_detections'.
2024-11-08 14:08:01.9931 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803393752530944/MISGCSN2_10493_0117-nd-int.fits.gz'
2024-11-08 14:08:02.5141 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/mastDownload/GALEX/3880803256112250880/MISGCSN2_10493_0117_0001-nd-int.fits.gz'
2024-11-08 14:08:03.1035 vasca.tables:211 |INFO: Writing file with name '/home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/VASCA_GALEX_3880803393752530944_NUV_field_data.fits'
2024-11-08 14:08:03.1047 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-08 14:08:03.1190 vasca.tables:236 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-08 14:08:04.9589 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-08 14:08:04.9738 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-08 14:08:04.9862 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-08 14:08:05.0214 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-08 14:08:05.0872 vasca.field:683 |INFO: Loaded new GALEX field '3880803393752530944' with obs_filter 'NUV'from MAST data .
2024-11-08 14:08:05.0896 vasca.field:747 |INFO: Loading field '2529969795944677376' with method 'MAST_LOCAL' for filter 'NUV' and load_products 'ALL'.
2024-11-08 14:08:05.0906 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376'
2024-11-08 14:08:05.0913 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca/vasca/vasca/test/resources/GALEX_visits_list.fits'
2024-11-08 14:08:05.0921 vasca.field:888 |DEBUG: Downloading archive field info and saving to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/MAST_2529969795944677376_NUV_coadd.fits.
2024-11-08 14:08:05.2798 vasca.field:908 |DEBUG: Constructing 'tt_fields'.
2024-11-08 14:08:05.2830 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_fields'
2024-11-08 14:08:05.2850 vasca.field:957 |DEBUG: Loading GALEX visit info :2529969795944677376 NUV None
2024-11-08 14:08:05.2859 vasca.field:988 |DEBUG: Reading archive visit info from cashed file '/home/runner/work/vasca/vasca/vasca/test/resources/GALEX_visits_list.fits'
2024-11-08 14:08:05.3144 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_visits'
2024-11-08 14:08:05.3161 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2529969795944677376', , S
2024-11-08 14:08:05.3166 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'ELAISN1_09', None, S
2024-11-08 14:08:05.3172 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-08 14:08:05.3179 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-08 14:08:05.3186 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-08 14:08:05.3193 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-08 14:08:05.3199 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-08 14:08:05.3208 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-08 14:08:05.3214 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-08 14:08:05.3230 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-08 14:08:05.3235 vasca.field:1076 |DEBUG: Reading archive field info from cashed file '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/MAST_2529969795944677376_NUV_coadd.fits'
2024-11-08 14:08:05.3388 vasca.field:1081 |DEBUG: Downloading archive data products list and saving to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/MAST_2529969795944677376_NUV_data.fits.
2024-11-08 14:08:06.2543 vasca.field:1164 |DEBUG: Downloading archive data products. Manifest saved to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/MAST_2529969795944677376_NUV_down.ecsv.
INFO: 173 of 1824 products were duplicates. Only downloading 1651 unique product(s). [astroquery.mast.observations]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0001-img/07-try/ELAISN1_09_0001-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658505723904/ELAISN1_09_0001-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0001-img/07-try/ELAISN1_09_0001-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658505723904/ELAISN1_09_0001-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0003-img/07-try/ELAISN1_09_0003-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658572832768/ELAISN1_09_0003-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0003-img/07-try/ELAISN1_09_0003-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658572832768/ELAISN1_09_0003-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0004-img/07-try/ELAISN1_09_0004-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658606387200/ELAISN1_09_0004-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0004-img/07-try/ELAISN1_09_0004-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658606387200/ELAISN1_09_0004-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0005-img/07-try/ELAISN1_09_0005-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658639941632/ELAISN1_09_0005-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0005-img/07-try/ELAISN1_09_0005-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658639941632/ELAISN1_09_0005-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0006-img/07-try/ELAISN1_09_0006-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658673496064/ELAISN1_09_0006-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0006-img/07-try/ELAISN1_09_0006-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658673496064/ELAISN1_09_0006-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0007-img/07-try/ELAISN1_09_0007-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658707050496/ELAISN1_09_0007-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0007-img/07-try/ELAISN1_09_0007-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658707050496/ELAISN1_09_0007-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0008-img/07-try/ELAISN1_09_0008-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658740604928/ELAISN1_09_0008-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0008-img/07-try/ELAISN1_09_0008-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658740604928/ELAISN1_09_0008-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0009-img/07-try/ELAISN1_09_0009-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658774159360/ELAISN1_09_0009-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0009-img/07-try/ELAISN1_09_0009-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658774159360/ELAISN1_09_0009-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0010-img/07-try/ELAISN1_09_0010-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658807713792/ELAISN1_09_0010-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0010-img/07-try/ELAISN1_09_0010-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658807713792/ELAISN1_09_0010-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0011-img/07-try/ELAISN1_09_0011-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658841268224/ELAISN1_09_0011-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0011-img/07-try/ELAISN1_09_0011-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658841268224/ELAISN1_09_0011-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0012-img/07-try/ELAISN1_09_0012-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658874822656/ELAISN1_09_0012-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0012-img/07-try/ELAISN1_09_0012-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658874822656/ELAISN1_09_0012-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0013-img/07-try/ELAISN1_09_0013-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658908377088/ELAISN1_09_0013-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0013-img/07-try/ELAISN1_09_0013-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658908377088/ELAISN1_09_0013-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0014-img/07-try/ELAISN1_09_0014-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658941931520/ELAISN1_09_0014-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0014-img/07-try/ELAISN1_09_0014-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658941931520/ELAISN1_09_0014-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0015-img/07-try/ELAISN1_09_0015-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658975485952/ELAISN1_09_0015-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0015-img/07-try/ELAISN1_09_0015-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658975485952/ELAISN1_09_0015-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0017-img/07-try/ELAISN1_09_0017-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659042594816/ELAISN1_09_0017-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0017-img/07-try/ELAISN1_09_0017-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659042594816/ELAISN1_09_0017-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0018-img/07-try/ELAISN1_09_0018-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659076149248/ELAISN1_09_0018-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/00-visits/0018-img/07-try/ELAISN1_09_0018-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659076149248/ELAISN1_09_0018-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/01-main/0001-img/07-try/ELAISN1_09-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969795944677376/ELAISN1_09-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/06370-ELAISN1_09/d/01-main/0001-img/07-try/ELAISN1_09-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969795944677376/ELAISN1_09-xd-mcat.fits.gz ...
[Done]
2024-11-08 14:09:27.2215 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_detections'
2024-11-08 14:09:27.2561 vasca.field:1391 |DEBUG: Constructed 'tt_detections'.
2024-11-08 14:09:27.2981 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_coadd_detections'
2024-11-08 14:09:27.3066 vasca.field:1396 |DEBUG: Constructed 'tt_coadd_detections'.
2024-11-08 14:09:27.3079 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969795944677376/ELAISN1_09-nd-int.fits.gz'
2024-11-08 14:09:27.9073 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658505723904/ELAISN1_09_0001-nd-int.fits.gz'
2024-11-08 14:09:28.5408 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658572832768/ELAISN1_09_0003-nd-int.fits.gz'
2024-11-08 14:09:29.1758 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658606387200/ELAISN1_09_0004-nd-int.fits.gz'
2024-11-08 14:09:29.8238 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658639941632/ELAISN1_09_0005-nd-int.fits.gz'
2024-11-08 14:09:30.4750 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658673496064/ELAISN1_09_0006-nd-int.fits.gz'
2024-11-08 14:09:31.1237 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658707050496/ELAISN1_09_0007-nd-int.fits.gz'
2024-11-08 14:09:31.7845 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658740604928/ELAISN1_09_0008-nd-int.fits.gz'
2024-11-08 14:09:32.4588 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658774159360/ELAISN1_09_0009-nd-int.fits.gz'
2024-11-08 14:09:33.1250 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658807713792/ELAISN1_09_0010-nd-int.fits.gz'
2024-11-08 14:09:33.7981 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658841268224/ELAISN1_09_0011-nd-int.fits.gz'
2024-11-08 14:09:34.4783 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658874822656/ELAISN1_09_0012-nd-int.fits.gz'
2024-11-08 14:09:35.1502 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658908377088/ELAISN1_09_0013-nd-int.fits.gz'
2024-11-08 14:09:35.8322 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658941931520/ELAISN1_09_0014-nd-int.fits.gz'
2024-11-08 14:09:36.5242 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969658975485952/ELAISN1_09_0015-nd-int.fits.gz'
2024-11-08 14:09:37.2460 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659042594816/ELAISN1_09_0017-nd-int.fits.gz'
2024-11-08 14:09:37.9422 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/mastDownload/GALEX/2529969659076149248/ELAISN1_09_0018-nd-int.fits.gz'
2024-11-08 14:09:38.6576 vasca.tables:211 |INFO: Writing file with name '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/VASCA_GALEX_2529969795944677376_NUV_field_data.fits'
2024-11-08 14:09:38.6590 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-08 14:09:38.6688 vasca.tables:236 |DEBUG: Storing image data of shape (16, 3840, 3840)
2024-11-08 14:09:54.4991 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-08 14:09:54.5138 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-08 14:09:54.5261 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-08 14:09:54.6290 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-08 14:09:54.7108 vasca.field:683 |INFO: Loaded new GALEX field '2529969795944677376' with obs_filter 'NUV'from MAST data .
2024-11-08 14:09:54.7145 vasca.field:747 |INFO: Loading field '2597664528044916736' with method 'MAST_LOCAL' for filter 'NUV' and load_products 'ALL'.
2024-11-08 14:09:54.7154 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736'
2024-11-08 14:09:54.7159 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca/vasca/vasca/test/resources/GALEX_visits_list.fits'
2024-11-08 14:09:54.7167 vasca.field:888 |DEBUG: Downloading archive field info and saving to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/MAST_2597664528044916736_NUV_coadd.fits.
2024-11-08 14:09:54.9240 vasca.field:908 |DEBUG: Constructing 'tt_fields'.
2024-11-08 14:09:54.9268 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_fields'
2024-11-08 14:09:54.9289 vasca.field:957 |DEBUG: Loading GALEX visit info :2597664528044916736 NUV None
2024-11-08 14:09:54.9294 vasca.field:988 |DEBUG: Reading archive visit info from cashed file '/home/runner/work/vasca/vasca/vasca/test/resources/GALEX_visits_list.fits'
2024-11-08 14:09:54.9590 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_visits'
2024-11-08 14:09:54.9612 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2597664528044916736', , S
2024-11-08 14:09:54.9617 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'PS_ELAISN1_MOS15', None, S
2024-11-08 14:09:54.9626 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-08 14:09:54.9632 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-08 14:09:54.9638 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-08 14:09:54.9644 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-08 14:09:54.9650 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-08 14:09:54.9656 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-08 14:09:54.9662 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-08 14:09:54.9681 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-08 14:09:54.9688 vasca.field:1076 |DEBUG: Reading archive field info from cashed file '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/MAST_2597664528044916736_NUV_coadd.fits'
2024-11-08 14:09:54.9838 vasca.field:1081 |DEBUG: Downloading archive data products list and saving to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/MAST_2597664528044916736_NUV_data.fits.
2024-11-08 14:09:56.0691 vasca.field:1164 |DEBUG: Downloading archive data products. Manifest saved to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/MAST_2597664528044916736_NUV_down.ecsv.
INFO: 173 of 2328 products were duplicates. Only downloading 2155 unique product(s). [astroquery.mast.observations]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0001-img/07-try/PS_ELAISN1_MOS15_0001-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390404636672/PS_ELAISN1_MOS15_0001-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0001-img/07-try/PS_ELAISN1_MOS15_0001-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390404636672/PS_ELAISN1_MOS15_0001-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0002-img/07-try/PS_ELAISN1_MOS15_0002-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390438191104/PS_ELAISN1_MOS15_0002-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0002-img/07-try/PS_ELAISN1_MOS15_0002-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390438191104/PS_ELAISN1_MOS15_0002-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0003-img/07-try/PS_ELAISN1_MOS15_0003-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390471745536/PS_ELAISN1_MOS15_0003-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0003-img/07-try/PS_ELAISN1_MOS15_0003-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390471745536/PS_ELAISN1_MOS15_0003-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0004-img/07-try/PS_ELAISN1_MOS15_0004-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390505299968/PS_ELAISN1_MOS15_0004-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0004-img/07-try/PS_ELAISN1_MOS15_0004-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390505299968/PS_ELAISN1_MOS15_0004-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0005-img/07-try/PS_ELAISN1_MOS15_0005-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390538854400/PS_ELAISN1_MOS15_0005-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0005-img/07-try/PS_ELAISN1_MOS15_0005-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390538854400/PS_ELAISN1_MOS15_0005-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0006-img/07-try/PS_ELAISN1_MOS15_0006-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390572408832/PS_ELAISN1_MOS15_0006-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0006-img/07-try/PS_ELAISN1_MOS15_0006-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390572408832/PS_ELAISN1_MOS15_0006-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0007-img/07-try/PS_ELAISN1_MOS15_0007-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390605963264/PS_ELAISN1_MOS15_0007-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0007-img/07-try/PS_ELAISN1_MOS15_0007-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390605963264/PS_ELAISN1_MOS15_0007-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0008-img/07-try/PS_ELAISN1_MOS15_0008-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390639517696/PS_ELAISN1_MOS15_0008-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0008-img/07-try/PS_ELAISN1_MOS15_0008-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390639517696/PS_ELAISN1_MOS15_0008-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0009-img/07-try/PS_ELAISN1_MOS15_0009-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390673072128/PS_ELAISN1_MOS15_0009-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0009-img/07-try/PS_ELAISN1_MOS15_0009-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390673072128/PS_ELAISN1_MOS15_0009-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0010-img/07-try/PS_ELAISN1_MOS15_0010-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390706626560/PS_ELAISN1_MOS15_0010-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0010-img/07-try/PS_ELAISN1_MOS15_0010-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390706626560/PS_ELAISN1_MOS15_0010-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0011-img/07-try/PS_ELAISN1_MOS15_0011-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390740180992/PS_ELAISN1_MOS15_0011-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0011-img/07-try/PS_ELAISN1_MOS15_0011-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390740180992/PS_ELAISN1_MOS15_0011-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0012-img/07-try/PS_ELAISN1_MOS15_0012-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390773735424/PS_ELAISN1_MOS15_0012-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0012-img/07-try/PS_ELAISN1_MOS15_0012-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390773735424/PS_ELAISN1_MOS15_0012-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0018-img/07-try/PS_ELAISN1_MOS15_0018-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390975062016/PS_ELAISN1_MOS15_0018-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0018-img/07-try/PS_ELAISN1_MOS15_0018-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390975062016/PS_ELAISN1_MOS15_0018-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0021-img/07-try/PS_ELAISN1_MOS15_0021-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391075725312/PS_ELAISN1_MOS15_0021-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0021-img/07-try/PS_ELAISN1_MOS15_0021-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391075725312/PS_ELAISN1_MOS15_0021-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0022-img/07-try/PS_ELAISN1_MOS15_0022-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391109279744/PS_ELAISN1_MOS15_0022-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0022-img/07-try/PS_ELAISN1_MOS15_0022-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391109279744/PS_ELAISN1_MOS15_0022-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0023-img/07-try/PS_ELAISN1_MOS15_0023-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391142834176/PS_ELAISN1_MOS15_0023-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0023-img/07-try/PS_ELAISN1_MOS15_0023-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391142834176/PS_ELAISN1_MOS15_0023-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0024-img/07-try/PS_ELAISN1_MOS15_0024-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391176388608/PS_ELAISN1_MOS15_0024-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0024-img/07-try/PS_ELAISN1_MOS15_0024-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391176388608/PS_ELAISN1_MOS15_0024-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0025-img/07-try/PS_ELAISN1_MOS15_0025-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391209943040/PS_ELAISN1_MOS15_0025-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0025-img/07-try/PS_ELAISN1_MOS15_0025-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391209943040/PS_ELAISN1_MOS15_0025-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0026-img/07-try/PS_ELAISN1_MOS15_0026-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391243497472/PS_ELAISN1_MOS15_0026-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0026-img/07-try/PS_ELAISN1_MOS15_0026-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391243497472/PS_ELAISN1_MOS15_0026-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0027-img/07-try/PS_ELAISN1_MOS15_0027-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391277051904/PS_ELAISN1_MOS15_0027-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0027-img/07-try/PS_ELAISN1_MOS15_0027-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391277051904/PS_ELAISN1_MOS15_0027-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0028-img/07-try/PS_ELAISN1_MOS15_0028-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391310606336/PS_ELAISN1_MOS15_0028-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0028-img/07-try/PS_ELAISN1_MOS15_0028-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391310606336/PS_ELAISN1_MOS15_0028-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0029-img/07-try/PS_ELAISN1_MOS15_0029-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391344160768/PS_ELAISN1_MOS15_0029-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/00-visits/0029-img/07-try/PS_ELAISN1_MOS15_0029-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391344160768/PS_ELAISN1_MOS15_0029-xd-mcat.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/01-main/0007-img/07-try/PS_ELAISN1_MOS15-nd-int.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664528044916736/PS_ELAISN1_MOS15-nd-int.fits.gz ...
[Done]
Downloading URL https://mast.stsci.edu/api/v0.1/Download/file?uri=mast:GALEX/url/data/GR6/pipe/01-vsn/08294-PS_ELAISN1_MOS15/d/01-main/0007-img/07-try/PS_ELAISN1_MOS15-xd-mcat.fits.gz to /home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664528044916736/PS_ELAISN1_MOS15-xd-mcat.fits.gz ...
[Done]
2024-11-08 14:11:58.8736 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_detections'
2024-11-08 14:11:58.9185 vasca.field:1391 |DEBUG: Constructed 'tt_detections'.
2024-11-08 14:11:58.9757 vasca.tables:156 |DEBUG: Adding table 'galex_field:tt_coadd_detections'
2024-11-08 14:11:58.9859 vasca.field:1396 |DEBUG: Constructed 'tt_coadd_detections'.
2024-11-08 14:11:58.9873 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664528044916736/PS_ELAISN1_MOS15-nd-int.fits.gz'
2024-11-08 14:11:59.5799 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390404636672/PS_ELAISN1_MOS15_0001-nd-int.fits.gz'
2024-11-08 14:12:00.2145 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390438191104/PS_ELAISN1_MOS15_0002-nd-int.fits.gz'
2024-11-08 14:12:00.8569 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390471745536/PS_ELAISN1_MOS15_0003-nd-int.fits.gz'
2024-11-08 14:12:01.5071 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390505299968/PS_ELAISN1_MOS15_0004-nd-int.fits.gz'
2024-11-08 14:12:02.1561 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390538854400/PS_ELAISN1_MOS15_0005-nd-int.fits.gz'
2024-11-08 14:12:02.8175 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390572408832/PS_ELAISN1_MOS15_0006-nd-int.fits.gz'
2024-11-08 14:12:03.4696 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390605963264/PS_ELAISN1_MOS15_0007-nd-int.fits.gz'
2024-11-08 14:12:04.1322 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390639517696/PS_ELAISN1_MOS15_0008-nd-int.fits.gz'
2024-11-08 14:12:04.8025 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390673072128/PS_ELAISN1_MOS15_0009-nd-int.fits.gz'
2024-11-08 14:12:05.4816 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390706626560/PS_ELAISN1_MOS15_0010-nd-int.fits.gz'
2024-11-08 14:12:06.1633 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390740180992/PS_ELAISN1_MOS15_0011-nd-int.fits.gz'
2024-11-08 14:12:06.8480 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390773735424/PS_ELAISN1_MOS15_0012-nd-int.fits.gz'
2024-11-08 14:12:07.5297 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664390975062016/PS_ELAISN1_MOS15_0018-nd-int.fits.gz'
2024-11-08 14:12:08.2451 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391075725312/PS_ELAISN1_MOS15_0021-nd-int.fits.gz'
2024-11-08 14:12:08.9442 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391109279744/PS_ELAISN1_MOS15_0022-nd-int.fits.gz'
2024-11-08 14:12:09.6463 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391142834176/PS_ELAISN1_MOS15_0023-nd-int.fits.gz'
2024-11-08 14:12:10.3402 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391176388608/PS_ELAISN1_MOS15_0024-nd-int.fits.gz'
2024-11-08 14:12:11.0494 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391209943040/PS_ELAISN1_MOS15_0025-nd-int.fits.gz'
2024-11-08 14:12:11.7595 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391243497472/PS_ELAISN1_MOS15_0026-nd-int.fits.gz'
2024-11-08 14:12:12.4719 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391277051904/PS_ELAISN1_MOS15_0027-nd-int.fits.gz'
2024-11-08 14:12:13.1954 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391310606336/PS_ELAISN1_MOS15_0028-nd-int.fits.gz'
2024-11-08 14:12:13.9107 vasca.field:117 |DEBUG: Loading skypmap from file: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/mastDownload/GALEX/2597664391344160768/PS_ELAISN1_MOS15_0029-nd-int.fits.gz'
2024-11-08 14:12:14.6436 vasca.tables:211 |INFO: Writing file with name '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/VASCA_GALEX_2597664528044916736_NUV_field_data.fits'
2024-11-08 14:12:14.6449 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-08 14:12:14.6687 vasca.tables:236 |DEBUG: Storing image data of shape (22, 3840, 3840)
2024-11-08 14:12:35.8384 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-08 14:12:35.8527 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-08 14:12:35.8650 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-08 14:12:36.0001 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-08 14:12:36.0928 vasca.field:683 |INFO: Loaded new GALEX field '2597664528044916736' with obs_filter 'NUV'from MAST data .
2024-11-08 14:12:36.1027 vasca.tables:156 |DEBUG: Adding table 'region:tt_filters'
This populates the region object with all specified fields, the relevant metadata is stored in tt_fields.
rg.info()
# rg.tt_fields
tt_fields:
<Table length=3>
name dtype unit description
----------------- ------- ---- -------------------------------------------------------
field_id bytes32 Field source ID number
field_name bytes32 Field name
project bytes32 Field project, typically survey name
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
observatory bytes22 Telescope of the observation (e.g. GALEX)
obs_filter bytes8 Filter of the observation (e.g. NUV)
fov_diam float32 deg Field radius or box size (depending on the observatory)
sel bool Selection of rows for VASCA analysis
nr_vis int32 Total number of visits of the field
time_bin_size_sum float32 s Total exposure time
time_start float64 d Start date and time in MJD
time_stop float64 d End time of last exposure
rg_fd_id int64 Region field ID number
{'DATAPATH': 'None', 'INFO': 'Field information table'}
tt_filters:
<Table length=2>
name dtype description
-------------- ------ ---------------------------------------
obs_filter_id int32 Observation filter ID number
obs_filter bytes8 Filter of the observation (e.g. NUV)
obs_filter_idx int32 Filter index in filter dependent arrays
{'INFO': 'Filters, their IDs and index, the last is specific for this region.'}
| field_id | field_name | project | ra | dec | observatory | obs_filter | fov_diam | sel | nr_vis | time_bin_size_sum | time_start | time_stop | rg_fd_id |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Loading ITables v2.2.3 from the init_notebook_mode cell...
(need help?) |
Field-level analysis¶
The field-level analysis incorporates, first, the data reduction and parameter mapping from raw data to VASCA field objects, second, the data quality selection and finally source clustering on the remaining high-quality detections.
The first step is implicitly taken care of by the GALEXField class, where the raw
data is loaded and only the column parameters are kept that are specified in the tables_dict
module. A shortcut is provided through the Region.get_field method which is an
interface to the load method of
any field class.
The configuration for the next two step requires the selection and cluster_det
entries under the observations section.
Data selection¶
Note
A crucial part of VASCA’s flexibility to adapt to raw data of virtually any instrument
comes from the fact that the parameter list used for data quality selection is not
fixed and is allowed to vary for different instruments and filters. The only
requirement is an existent entry in the tables_dict module for any parameter and
a corresponding field class that includes these parameters in the tt_detections
table.
The API for the data selection is provided by the TableCollection.select_rows
method. Each entry under selection maps to this interface. The table parameters
specifies which table to select on. Any selection operation modifies the sel
column of a given table. It contains boolean values so 0 means unselected and
1 means selected.
By specifying the presel_typeparameter, one controls the logic by which an
existing selection is combined with a new one. The sel_type parameter specifies
the logic by which the selection on a set of multiple column parameters is combined.
Parameters range and bitmask provide the column parameter and artifact
bitflag values that are used to make the selection. Using set_range on can choose
to clip values of a certain column to minimum and maximum values.
In combination with sel_type = "is_in" and var parameters, it is possible to
select the rows of given column varin the target table if a value is also present
in the same column of a reference table (ref_table).
import numpy as np
# Updating the observations for GALEX-NUV observations
config["observations"][0].update(
{
"selection": {
# Quality cuts on visit-level detections
"det_quality": {
"table": "tt_detections",
"presel_type": "and",
"sel_type": "and",
"range": {
"s2n": [3.0, np.inf],
"r_fov": [0.0, 0.5],
"ellip_world": [0.0, 0.5],
"size_world": [0.0, 6.0],
"class_star": [0.15, 1.0],
"chkobj_type": [-0.5, 0.5],
"flux_app_ratio": [0.3, 1.05],
},
"bitmask": {
"artifacts": [2, 4, 8, 128, 256],
},
"set_range": {"pos_err": [0.5, 5]},
},
# Quality cuts on field-averaged detections
"coadd_det_quality": {
"table": "tt_detections",
"presel_type": "and",
"sel_type": "and",
"range": {
"s2n": [5.0, np.inf],
"r_fov": [0.0, 0.5],
"ellip_world": [0.0, 0.5],
"size_world": [0.0, 6.0],
"class_star": [0.15, 1.0],
"chkobj_type": [-0.5, 0.5],
"flux_app_ratio": [0.3, 1.05],
},
"bitmask": {
"artifacts": [2, 4, 8, 128, 256],
},
},
# Selection on only those detections wich are part of clusters
"det_association": {
"table": "tt_detections",
"presel_type": "and",
"sel_type": "is_in",
"ref_table": "tt_sources",
"var": "fd_src_id",
},
},
}
)
Clustering¶
Also the field-level clustering configuration showcases VASCA’s modular
approach. In the cluster_det section on specifies the clustering algorithm
which, in principle, can be different for each instrument and filter. Although,
at the moment only mean-shift clustering
is supported by VASCA.
Again, the responsible API is provided by TableCollection.cluster_meanshift.
This method wraps a method provided by the scikit-learn package. The end result is that each field optains
a new tt_visits table that lists all identified sources as defined
by their clustered detections. Sources have at the minimum one and as manny as n_vis
detections.
Mean-shift is well suited for this use case due to several reasons. Most importantly it is that the algorithm doesn’t require the total number of clusters as a parameter. In fact it is determining that number which would be otherwise very difficult to predict from the visit-level detections before having done the clustering.
Another reason is its relatively simple algorithm where only one parameters is
required. It is called the bandwidth which means, translated to the astronomy use
case, the radial size of a typical source on the sky. It should be roughly chosen
to match the instrument’s PSF, which, for GALEX, is about 5 arcseconds. We set
it slightly smaller to limit false associations also considering that the
source center is usually much better constrained than the PSF might suggest.
# Updating the observations for GALEX-NUV observations continued...
config["observations"][0].update(
{
"cluster_det": {
"meanshift": {
"bandwidth": 4,
"seeds": None,
"bin_seeding": False,
"min_bin_freq": 1,
"cluster_all": True,
"n_jobs": None,
"max_iter": 300,
"table_name": "tt_detections",
},
},
},
)
Pipeline flow¶
According to the configuration above, we can finally run the analysis. VASCA
implements parallel processing (multiprocessing.pool.Pool.starmap) for this part of the pipeline
by applying the run_field method in parallel for each field.
First, the parameters for run_field are collected.
import vasca.utils as vutils
# Collect parameters from config
fd_pars: list[list[int, str, Region, dict]] = []
vobs: list[dict] = config["observations"]
obs_nr: int
field_nr: str
# Loop over observation list index
for obs_nr, _ in enumerate(vobs):
# Loop over fields
for field_nr in vobs[obs_nr]["obs_field_ids"]:
# Construct VASCA field ID (prepending instrument/filter identifier)
iprefix: str = vutils.dd_obs_id_add[
vobs[obs_nr]["observatory"] + vobs[obs_nr]["obs_filter"]
]
field_id: str = f"{iprefix}{field_nr}"
# Save parameters outside loop
fd_pars.append([obs_nr, field_id, rg, config])
Second, all fields are processed in parallel.
from multiprocessing.pool import Pool
import vasca.vasca_pipe as vpipe
# Run each field in a separate process in parallel
nr_cpus = config["general"]["nr_cpus"]
logger.info(f"Analyzing {len(fd_pars)} fields on {nr_cpus} parallel threads.")
with Pool(processes=nr_cpus) as pool:
pool_return = pool.starmap(vpipe.run_field_docs, fd_pars)
pool.join()
logger.info("Done analyzing individual fields.")
Show code cell output
2024-11-08 14:12:36.1628 __main__:6 |INFO: Analyzing 3 fields on 3 parallel threads.
2024-11-08 14:12:36.2529 vasca.vasca_pipe:224 |INFO: Analysing field: GNU2597664528044916736
2024-11-08 14:12:36.2536 vasca.vasca_pipe:224 |INFO: Analysing field: GNU2529969795944677376
2024-11-08 14:12:36.2536 vasca.vasca_pipe:224 |INFO: Analysing field: GNU3880803393752530944
2024-11-08 14:12:36.2576 vasca.field:747 |INFO: Loading field '2597664528044916736' with method 'VASCA' for filter 'NUV' and load_products 'ALL'.
2024-11-08 14:12:36.2592 vasca.field:747 |INFO: Loading field '2529969795944677376' with method 'VASCA' for filter 'NUV' and load_products 'ALL'.
2024-11-08 14:12:36.2596 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736'
2024-11-08 14:12:36.2589 vasca.field:747 |INFO: Loading field '3880803393752530944' with method 'VASCA' for filter 'NUV' and load_products 'ALL'.
2024-11-08 14:12:36.2609 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376'
2024-11-08 14:12:36.2610 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca/vasca/vasca/test/resources/GALEX_visits_list.fits'
2024-11-08 14:12:36.2616 vasca.field:533 |DEBUG: Field data path set to: '/home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944'
2024-11-08 14:12:36.2628 vasca.tables:310 |DEBUG: Loading file with name '/home/runner/work/vasca/vasca/docs/tutorial_resources/2597664528044916736/VASCA_GALEX_2597664528044916736_NUV_field_data.fits'
2024-11-08 14:12:36.2626 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca/vasca/vasca/test/resources/GALEX_visits_list.fits'
2024-11-08 14:12:36.2639 vasca.field:534 |DEBUG: Visits data path set to: '/home/runner/work/vasca/vasca/vasca/test/resources/GALEX_visits_list.fits'
2024-11-08 14:12:36.2647 vasca.tables:310 |DEBUG: Loading file with name '/home/runner/work/vasca/vasca/docs/tutorial_resources/2529969795944677376/VASCA_GALEX_2529969795944677376_NUV_field_data.fits'
2024-11-08 14:12:36.2660 vasca.tables:310 |DEBUG: Loading file with name '/home/runner/work/vasca/vasca/docs/tutorial_resources/3880803393752530944/VASCA_GALEX_3880803393752530944_NUV_field_data.fits'
2024-11-08 14:12:36.2773 vasca.tables:321 |DEBUG: Loading table 'tt_fields'
2024-11-08 14:12:36.2807 vasca.tables:321 |DEBUG: Loading table 'tt_fields'
2024-11-08 14:12:36.2821 vasca.tables:321 |DEBUG: Loading table 'tt_fields'
2024-11-08 14:12:36.2984 vasca.tables:321 |DEBUG: Loading table 'tt_visits'
2024-11-08 14:12:36.3095 vasca.tables:321 |DEBUG: Loading table 'tt_visits'
2024-11-08 14:12:36.3095 vasca.tables:321 |DEBUG: Loading table 'tt_visits'
2024-11-08 14:12:36.3165 vasca.tables:321 |DEBUG: Loading table 'tt_detections'
2024-11-08 14:12:36.3330 vasca.tables:321 |DEBUG: Loading table 'tt_detections'
2024-11-08 14:12:36.3330 vasca.tables:321 |DEBUG: Loading table 'tt_detections'
2024-11-08 14:12:36.3826 vasca.tables:321 |DEBUG: Loading table 'tt_coadd_detections'
2024-11-08 14:12:36.3846 vasca.tables:321 |DEBUG: Loading table 'tt_coadd_detections'
2024-11-08 14:12:36.3987 vasca.tables:321 |DEBUG: Loading table 'tt_coadd_detections'
2024-11-08 14:12:37.3287 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU3880803393752530944', None, S
2024-11-08 14:12:37.3313 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'MISGCSN2_10493_0117', None, S
2024-11-08 14:12:37.3336 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-08 14:12:37.3356 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-08 14:12:37.3374 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-08 14:12:37.3393 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-08 14:12:37.3411 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-08 14:12:37.3427 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-08 14:12:37.3443 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-08 14:12:37.3488 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-08 14:12:37.3521 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU3880803393752530944', None, S
2024-11-08 14:12:37.3549 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'MISGCSN2_10493_0117', None, S
2024-11-08 14:12:37.3577 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-08 14:12:37.3593 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-08 14:12:37.3612 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-08 14:12:37.3630 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-08 14:12:37.3646 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-08 14:12:37.3663 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.64270077057597, deg, f
2024-11-08 14:12:37.3681 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.1235646075049, deg, f
2024-11-08 14:12:37.3710 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-08 14:12:37.3724 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-08 14:12:37.3743 vasca.tables:451 |DEBUG: AND selecting 's2n' [3.0, inf], kept: 67.5798%
2024-11-08 14:12:37.3769 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 45.6911%
2024-11-08 14:12:37.3792 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 37.9818%
2024-11-08 14:12:37.3809 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 37.8166%
2024-11-08 14:12:37.3827 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 36.5501%
2024-11-08 14:12:37.3843 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 35.4626%
2024-11-08 14:12:37.3860 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 33.4389%
2024-11-08 14:12:37.3878 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 31.1123%
2024-11-08 14:12:37.3893 vasca.tables:500 |INFO: Total table entries 7264, pre-selected rows 7264, rows after new selection 2260
2024-11-08 14:12:37.3913 vasca.tables:512 |INFO: Setting range for 'pos_err' to '[0.5, 5]'
2024-11-08 14:12:37.3939 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-08 14:12:37.3957 vasca.tables:451 |DEBUG: AND selecting 's2n' [5.0, inf], kept: 23.5683%
2024-11-08 14:12:37.3984 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 15.4460%
2024-11-08 14:12:37.4001 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 14.3172%
2024-11-08 14:12:37.4017 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 14.1520%
2024-11-08 14:12:37.4034 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 12.9130%
2024-11-08 14:12:37.4060 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 11.8254%
2024-11-08 14:12:37.4081 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 11.8254%
2024-11-08 14:12:37.4099 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 11.0683%
2024-11-08 14:12:37.4114 vasca.tables:500 |INFO: Total table entries 7264, pre-selected rows 2260, rows after new selection 804
2024-11-08 14:12:37.4130 vasca.vasca_pipe:257 |INFO: Clustering field detections (GNU3880803393752530944)
2024-11-08 14:12:37.4151 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_detections
2024-11-08 14:12:37.4189 vasca.tables:667 |DEBUG: Clustering 804 detections/sources
2024-11-08 14:12:37.4218 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': None, 'max_iter': 300}'
2024-11-08 14:12:38.1836 vasca.tables:675 |DEBUG: Done with clustering, found 804 clusters
2024-11-08 14:12:38.1868 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_sources'
2024-11-08 14:12:38.1955 vasca.field:250 |DEBUG: Scanning for doubled visit detections
2024-11-08 14:12:38.6078 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-08 14:12:38.6110 vasca.tables:500 |INFO: Total table entries 7264, pre-selected rows 804, rows after new selection 804
2024-11-08 14:12:38.6136 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/fields/field_GNU3880803393752530944.fits'
2024-11-08 14:12:38.6170 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-08 14:12:38.6320 vasca.tables:236 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-08 14:12:41.4479 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-08 14:12:41.4735 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-08 14:12:41.4959 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-08 14:12:41.5611 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-08 14:12:41.6150 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-08 14:12:41.7732 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_detections'
2024-11-08 14:12:41.7786 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_coadd_detections'
2024-11-08 14:12:41.7873 vasca.vasca_pipe:274 |INFO: Done with field: GNU3880803393752530944
2024-11-08 14:12:42.9388 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2529969795944677376', None, S
2024-11-08 14:12:42.9409 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'ELAISN1_09', None, S
2024-11-08 14:12:42.9429 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-08 14:12:42.9446 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-08 14:12:42.9459 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-08 14:12:42.9472 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-08 14:12:42.9482 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-08 14:12:42.9493 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-08 14:12:42.9503 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-08 14:12:42.9530 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-08 14:12:42.9544 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2529969795944677376', None, S
2024-11-08 14:12:42.9554 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'ELAISN1_09', None, S
2024-11-08 14:12:42.9567 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-08 14:12:42.9580 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-08 14:12:42.9593 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-08 14:12:42.9604 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-08 14:12:42.9615 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-08 14:12:42.9626 vasca.field:403 |DEBUG: Getting parameter 'ra': 241.60667709058498, deg, f
2024-11-08 14:12:42.9637 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.531821052495, deg, f
2024-11-08 14:12:42.9659 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-08 14:12:42.9669 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-08 14:12:42.9711 vasca.tables:451 |DEBUG: AND selecting 's2n' [3.0, inf], kept: 49.4597%
2024-11-08 14:12:42.9749 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 33.9603%
2024-11-08 14:12:42.9782 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 29.8416%
2024-11-08 14:12:42.9816 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 29.6152%
2024-11-08 14:12:42.9850 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 26.4344%
2024-11-08 14:12:42.9890 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 25.8593%
2024-11-08 14:12:42.9929 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 25.6896%
2024-11-08 14:12:42.9962 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 24.3663%
2024-11-08 14:12:42.9980 vasca.tables:500 |INFO: Total table entries 170929, pre-selected rows 170929, rows after new selection 41649
2024-11-08 14:12:42.9998 vasca.tables:512 |INFO: Setting range for 'pos_err' to '[0.5, 5]'
2024-11-08 14:12:43.0129 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-08 14:12:43.0212 vasca.tables:451 |DEBUG: AND selecting 's2n' [5.0, inf], kept: 22.2244%
2024-11-08 14:12:43.0258 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 15.2549%
2024-11-08 14:12:43.0294 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 14.1556%
2024-11-08 14:12:43.0356 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 13.9356%
2024-11-08 14:12:43.0412 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 11.5100%
2024-11-08 14:12:43.0467 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 10.9349%
2024-11-08 14:12:43.0507 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 10.9338%
2024-11-08 14:12:43.0541 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 10.3803%
2024-11-08 14:12:43.0561 vasca.tables:500 |INFO: Total table entries 170929, pre-selected rows 41649, rows after new selection 17743
2024-11-08 14:12:43.0578 vasca.vasca_pipe:257 |INFO: Clustering field detections (GNU2529969795944677376)
2024-11-08 14:12:43.0591 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_detections
2024-11-08 14:12:43.0787 vasca.tables:667 |DEBUG: Clustering 17743 detections/sources
2024-11-08 14:12:43.0805 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': None, 'max_iter': 300}'
2024-11-08 14:12:43.1894 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2597664528044916736', None, S
2024-11-08 14:12:43.1913 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'PS_ELAISN1_MOS15', None, S
2024-11-08 14:12:43.1932 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-08 14:12:43.1947 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-08 14:12:43.1963 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-08 14:12:43.1980 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-08 14:12:43.1998 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-08 14:12:43.2020 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-08 14:12:43.2042 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-08 14:12:43.2082 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-08 14:12:43.2105 vasca.field:403 |DEBUG: Getting parameter 'field_id': b'GNU2597664528044916736', None, S
2024-11-08 14:12:43.2127 vasca.field:403 |DEBUG: Getting parameter 'field_name': b'PS_ELAISN1_MOS15', None, S
2024-11-08 14:12:43.2146 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-08 14:12:43.2170 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-08 14:12:43.2192 vasca.field:403 |DEBUG: Getting parameter 'fov_diam': 1.2000000476837158, deg, f
2024-11-08 14:12:43.2209 vasca.field:403 |DEBUG: Getting parameter 'observatory': b'GALEX', None, S
2024-11-08 14:12:43.2231 vasca.field:403 |DEBUG: Getting parameter 'obs_filter': b'NUV', None, S
2024-11-08 14:12:43.2245 vasca.field:403 |DEBUG: Getting parameter 'ra': 242.396514585217, deg, f
2024-11-08 14:12:43.2257 vasca.field:403 |DEBUG: Getting parameter 'dec': 53.9999872930018, deg, f
2024-11-08 14:12:43.2280 vasca.field:324 |DEBUG: Set attributes: {'tt_fields': ['field_id', 'field_name', 'ra', 'dec', 'fov_diam', 'observatory', 'obs_filter', 'center'], 'tt_visits': ['nr_vis', 'time_bin_size_sum', 'time_start', 'time_stop']}
2024-11-08 14:12:43.2293 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-08 14:12:43.2337 vasca.tables:451 |DEBUG: AND selecting 's2n' [3.0, inf], kept: 51.0152%
2024-11-08 14:12:43.2373 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 36.1159%
2024-11-08 14:12:43.2408 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 31.5473%
2024-11-08 14:12:43.2445 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 31.1979%
2024-11-08 14:12:43.2485 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 27.4451%
2024-11-08 14:12:43.2529 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 26.8908%
2024-11-08 14:12:43.2573 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 26.7283%
2024-11-08 14:12:43.2613 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 25.5369%
2024-11-08 14:12:43.2631 vasca.tables:500 |INFO: Total table entries 228387, pre-selected rows 228387, rows after new selection 58323
2024-11-08 14:12:43.2648 vasca.tables:512 |INFO: Setting range for 'pos_err' to '[0.5, 5]'
2024-11-08 14:12:43.2796 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-08 14:12:43.2844 vasca.tables:451 |DEBUG: AND selecting 's2n' [5.0, inf], kept: 22.6051%
2024-11-08 14:12:43.2887 vasca.tables:451 |DEBUG: AND selecting 'r_fov' [0.0, 0.5], kept: 16.0994%
2024-11-08 14:12:43.2929 vasca.tables:451 |DEBUG: AND selecting 'ellip_world' [0.0, 0.5], kept: 14.8756%
2024-11-08 14:12:43.2972 vasca.tables:451 |DEBUG: AND selecting 'size_world' [0.0, 6.0], kept: 14.5376%
2024-11-08 14:12:43.3008 vasca.tables:451 |DEBUG: AND selecting 'class_star' [0.15, 1.0], kept: 11.6535%
2024-11-08 14:12:43.3047 vasca.tables:451 |DEBUG: AND selecting 'chkobj_type' [-0.5, 0.5], kept: 11.0991%
2024-11-08 14:12:43.3083 vasca.tables:451 |DEBUG: AND selecting 'flux_app_ratio' [0.3, 1.05], kept: 11.0974%
2024-11-08 14:12:43.3114 vasca.tables:468 |DEBUG: AND selecting bitmask 'artifacts' removing [2, 4, 8, 128, 256], kept: 10.6136%
2024-11-08 14:12:43.3127 vasca.tables:500 |INFO: Total table entries 228387, pre-selected rows 58323, rows after new selection 24240
2024-11-08 14:12:43.3139 vasca.vasca_pipe:257 |INFO: Clustering field detections (GNU2597664528044916736)
2024-11-08 14:12:43.3150 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_detections
2024-11-08 14:12:43.3314 vasca.tables:667 |DEBUG: Clustering 24240 detections/sources
2024-11-08 14:12:43.3327 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': None, 'max_iter': 300}'
2024-11-08 14:12:54.9335 vasca.tables:675 |DEBUG: Done with clustering, found 2318 clusters
2024-11-08 14:12:54.9354 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_sources'
2024-11-08 14:12:54.9417 vasca.field:250 |DEBUG: Scanning for doubled visit detections
2024-11-08 14:12:59.2411 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-08 14:12:59.2461 vasca.tables:500 |INFO: Total table entries 170929, pre-selected rows 17743, rows after new selection 17743
2024-11-08 14:12:59.2479 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/fields/field_GNU2529969795944677376.fits'
2024-11-08 14:12:59.2501 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-08 14:12:59.2592 vasca.tables:236 |DEBUG: Storing image data of shape (16, 3840, 3840)
2024-11-08 14:12:59.9423 vasca.tables:675 |DEBUG: Done with clustering, found 2475 clusters
2024-11-08 14:12:59.9442 vasca.tables:156 |DEBUG: Adding table 'base_field:tt_sources'
2024-11-08 14:12:59.9513 vasca.field:250 |DEBUG: Scanning for doubled visit detections
2024-11-08 14:13:05.8670 vasca.tables:417 |INFO: Applying selection on table 'tt_detections'
2024-11-08 14:13:05.8721 vasca.tables:500 |INFO: Total table entries 228387, pre-selected rows 24240, rows after new selection 24240
2024-11-08 14:13:05.8739 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/fields/field_GNU2597664528044916736.fits'
2024-11-08 14:13:05.8755 vasca.tables:222 |DEBUG: Storing image data of shape (3840, 3840)
2024-11-08 14:13:05.8844 vasca.tables:236 |DEBUG: Storing image data of shape (22, 3840, 3840)
2024-11-08 14:13:15.2626 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-08 14:13:15.2779 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-08 14:13:15.2900 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-08 14:13:15.4196 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-08 14:13:15.4563 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-08 14:13:15.5650 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_detections'
2024-11-08 14:13:15.5761 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_coadd_detections'
2024-11-08 14:13:15.5858 vasca.vasca_pipe:274 |INFO: Done with field: GNU2529969795944677376
2024-11-08 14:13:27.4921 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-08 14:13:27.5077 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-08 14:13:27.5201 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-08 14:13:27.6979 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-08 14:13:27.7359 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-08 14:13:27.8379 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_detections'
2024-11-08 14:13:27.8528 vasca.tables:183 |DEBUG: Removing unselected rows in table 'tt_coadd_detections'
2024-11-08 14:13:27.8643 vasca.vasca_pipe:274 |INFO: Done with field: GNU2597664528044916736
2024-11-08 14:13:27.9256 __main__:12 |INFO: Done analyzing individual fields.
Finally, the pool results are unpacked and the region object is updated with processed field information.
A memory-saving procedure is used where first all fields are brought to the
scope of the region object by filling the Region.field dictionary from
which the field-level data is taken and stacked in respective region-owned
tables using the Region.add_table_from_fields method. After this step
all fields are discarded from the scope, to be deleted from the garbage collector.
Hint
In VASCA a Region object keeps track of its fields in the Region.tt_fields
table. At any time one can load field data of a specific field using Region.get_field.
from astropy.table import unique
# Loop over processed fields
for field in pool_return:
# Populate region field dictionary with field data
rg.fields[field.field_id] = field
logger.info(f"Added field {field.field_id} from pool to region")
# Add field tables to region
# Visits metadata
rg.add_table_from_fields("tt_visits")
rg.tt_visits = unique(rg.tt_visits, keys="vis_id")
# Clustered sources
rg.add_table_from_fields("tt_sources")
# Visit-level detections
rg.add_table_from_fields("tt_detections", only_selected=False)
# Field-averaged detectsion
rg.add_table_from_fields("tt_coadd_detections")
# Discard fields. All that was needed has been transferred to region tables
del rg.fields
Show code cell output
2024-11-08 14:13:27.9334 __main__:7 |INFO: Added field GNU3880803393752530944 from pool to region
2024-11-08 14:13:27.9344 __main__:7 |INFO: Added field GNU2529969795944677376 from pool to region
2024-11-08 14:13:27.9351 __main__:7 |INFO: Added field GNU2597664528044916736 from pool to region
2024-11-08 14:13:27.9358 vasca.region:152 |DEBUG: Adding table from fields: tt_visits
2024-11-08 14:13:27.9389 vasca.tables:156 |DEBUG: Adding table 'region:tt_visits'
2024-11-08 14:13:27.9418 vasca.region:152 |DEBUG: Adding table from fields: tt_sources
2024-11-08 14:13:27.9543 vasca.tables:156 |DEBUG: Adding table 'region:tt_sources'
2024-11-08 14:13:27.9628 vasca.region:152 |DEBUG: Adding table from fields: tt_detections
2024-11-08 14:13:27.9879 vasca.tables:156 |DEBUG: Adding table 'region:tt_detections'
2024-11-08 14:13:28.0248 vasca.region:152 |DEBUG: Adding table from fields: tt_coadd_detections
2024-11-08 14:13:28.0412 vasca.tables:156 |DEBUG: Adding table 'region:tt_coadd_detections'
Region-level analysis¶
In the final stage of the pipeline, all region-level analysis steps are performed. This stage encompasses three key tasks: first, managing sources located in overlapping sky regions; second, evaluating statistics for use in variability detection; and finally, preparing the pipeline results for writing to disk and generating the VASCA variable source catalog.
Overlapping fields¶
VASCA merges field-level sources in overlapping sky regions in a second clustering step, where the same mean-shift algorithm is used but with a dedicated configuration.
In case field-averaged (co-added) data exists, the field-level detections are merged in the same way, again, with a separate configuration.
config["cluster_src"] = {
"meanshift": {
"bandwidth": 4,
"seeds": None,
"bin_seeding": False,
"min_bin_freq": 1,
"cluster_all": True,
"n_jobs": 1,
"max_iter": 300,
"table_name": "tt_sources",
}
}
config["cluster_coadd_dets"] = {
"meanshift": {
"bandwidth": 4,
"seeds": None,
"bin_seeding": False,
"min_bin_freq": 1,
"cluster_all": True,
"n_jobs": 1,
"max_iter": 300,
"table_name": "tt_coadd_detections",
}
}
# Cluster field sources and co-adds in parallel
ll_cluster = [
[
config["cluster_src"]["meanshift"],
rg.tt_sources,
rg.tt_detections,
False,
],
[
config["cluster_coadd_dets"]["meanshift"],
rg.tt_coadd_detections,
False,
True,
],
]
with Pool(processes=2) as pool:
pool_return = pool.starmap(vpipe.run_cluster_fields, ll_cluster)
pool.join()
# Copy parallelized results into region
for pool_rg in pool_return:
if "tt_sources" in pool_rg._table_names:
rg.tt_sources = pool_rg.tt_sources
rg.tt_detections = pool_rg.tt_detections
else:
rg.add_table(pool_rg.tt_coadd_sources, "region:tt_coadd_sources")
rg.tt_coadd_detections = pool_rg.tt_coadd_detections
Show code cell output
2024-11-08 14:13:28.1626 vasca.tables:156 |DEBUG: Adding table 'tt_sources'
2024-11-08 14:13:28.1679 vasca.tables:156 |DEBUG: Adding table 'tt_detections'
2024-11-08 14:13:28.1707 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_sources
2024-11-08 14:13:28.1740 vasca.tables:667 |DEBUG: Clustering 5597 detections/sources
2024-11-08 14:13:28.1753 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': 1, 'max_iter': 300}'
2024-11-08 14:13:28.1794 vasca.tables:156 |DEBUG: Adding table 'tt_coadd_detections'
2024-11-08 14:13:28.1847 vasca.tables:650 |INFO: Clustering with MeanShift with table: tt_coadd_detections
2024-11-08 14:13:28.1981 vasca.tables:667 |DEBUG: Clustering 54330 detections/sources
2024-11-08 14:13:28.2019 vasca.tables:668 |DEBUG: MeanShift with parameters (bandwith in degrees): '{'bandwidth': 0.0011111111111111111, 'seeds': None, 'bin_seeding': False, 'min_bin_freq': 1, 'cluster_all': True, 'n_jobs': 1, 'max_iter': 300}'
2024-11-08 14:13:31.6802 vasca.tables:675 |DEBUG: Done with clustering, found 4928 clusters
2024-11-08 14:13:33.1204 vasca.tables:156 |DEBUG: Adding table 'region:tt_sources'
2024-11-08 14:13:33.2855 vasca.tables:719 |DEBUG: Merged sources: 669 (11.9528%)
2024-11-08 14:14:04.9471 vasca.tables:675 |DEBUG: Done with clustering, found 45315 clusters
2024-11-08 14:14:08.3000 vasca.tables:156 |DEBUG: Adding table 'region:tt_coadd_sources'
2024-11-08 14:14:08.4113 vasca.tables:156 |DEBUG: Adding table 'region:tt_coadd_sources'
Source statistics¶
The primary statistic used by VASCA to detect variability is the probability of
obtaining a flux with the observed fluctuations under the assumption that the null
hypothesis is true, that is, constant flux (flux_cpval). The default selection is
such that the chance for the observed variability being purely random can be ruled out
at 5-sigma significance.
Additionally the normalized excess variance (flux_nxv) and absolute flux limits
is used to limit a contamination due to potential hidden systematic flux variations.
Similarly a selection on variation of the spatial coordinates (pos_cpval. pos_xv)
is used to reduce the contamination du to false association of visit-levl detections
in the clustering step.
To ensure the statistical correctness only sources with more than three detections are
considers (n_det>3).
Note
The source selection is configured independently for each observational filter. This allows to adapt to potential systematics in a very flexible way.
For a more concrete example on these calculations in VASCA, have a look at the tutorial on Variability Statistics.
config["selection_src"] = {
"src_variability_nuv": {
"table": "tt_sources",
"presel_type": "or",
"sel_type": "and",
"obs_filter": "NUV",
"range": {
"nr_det": [3, np.inf],
"pos_cpval": [0.0001, np.inf],
"pos_xv": [-np.inf, 2],
"flux_cpval": [-0.5, 0.000000573303],
"flux_nxv": [0.001, np.inf],
"flux": [0.144543, 575.43],
},
},
"src_coadd_diff_nuv": {
"table": "tt_sources",
"presel_type": "or",
"sel_type": "and",
"obs_filter": "NUV",
"range": {
"nr_det": [2, np.inf],
"pos_cpval": [0.0001, np.inf],
"pos_xv": [-np.inf, 2],
"coadd_ffactor": [2.0, np.inf],
"coadd_fdiff_s2n": [7, np.inf],
},
},
}
An additional selection may be possible if co-added data is available. In this case the association between VASCA sources and the field-averaged input data can be made. This serves as cross-check since most static sources should be recovered in the clustering step and match the field average.
config["assoc_src_coadd"] = {
"dist_max": 1, # Associate nearest source below this distance in arc_sec OR
"dist_s2n_max": 3, # Associate nearest source with this distance in units of "squared summed position error"
}
import astropy.units as uu
# Calculate source statistics
rg.set_src_stats(src_id_name="rg_src_id")
rg.set_src_stats(src_id_name="coadd_src_id")
# Match sources to coadd sources
rg.cross_match(
dist_max=config["assoc_src_coadd"]["dist_max"] * uu.arcsec,
dist_s2n_max=config["assoc_src_coadd"]["dist_s2n_max"],
)
# Select variable sources, deselect all sources before and
# make sure all tables containting the region source ID
# are syncronized to this selection
rg.tt_sources["sel"] = False
rg.select_from_config(
config["selection_src"]
) # Loops over TableCollection.select_rows()
rg.synch_src_sel(remove_unselected=False)
# Set source ID mapping table
rg.set_src_id_info()
Show code cell output
2024-11-08 14:14:08.4345 vasca.tables:789 |DEBUG: Calculating source statistics.
2024-11-08 14:14:10.7694 vasca.tables:789 |DEBUG: Calculating source statistics.
2024-11-08 14:14:21.9943 vasca.tables:1099 |INFO: Cross matching table tt_sources
2024-11-08 14:14:22.0248 vasca.tables:402 |INFO: Applying source selection 'src_variability_nuv'
2024-11-08 14:14:22.0257 vasca.tables:417 |INFO: Applying selection on table 'tt_sources'
2024-11-08 14:14:22.0269 vasca.tables:451 |DEBUG: AND selecting 'nr_det' [3, inf], kept: 64.1031%
2024-11-08 14:14:22.0276 vasca.tables:451 |DEBUG: AND selecting 'pos_cpval' [0.0001, inf], kept: 59.5170%
2024-11-08 14:14:22.0288 vasca.tables:451 |DEBUG: AND selecting 'pos_xv' [-inf, 2], kept: 59.3750%
2024-11-08 14:14:22.0294 vasca.tables:451 |DEBUG: AND selecting 'flux_cpval' [-0.5, 5.73303e-07], kept: 1.2378%
2024-11-08 14:14:22.0301 vasca.tables:451 |DEBUG: AND selecting 'flux_nxv' [0.001, inf], kept: 1.1364%
2024-11-08 14:14:22.0308 vasca.tables:451 |DEBUG: AND selecting 'flux' [0.144543, 575.43], kept: 1.1364%
2024-11-08 14:14:22.0317 vasca.tables:500 |INFO: Total table entries 4928, pre-selected rows 0, rows after new selection 56
2024-11-08 14:14:22.0323 vasca.tables:402 |INFO: Applying source selection 'src_coadd_diff_nuv'
2024-11-08 14:14:22.0329 vasca.tables:417 |INFO: Applying selection on table 'tt_sources'
2024-11-08 14:14:22.0346 vasca.tables:451 |DEBUG: AND selecting 'nr_det' [2, inf], kept: 71.4692%
2024-11-08 14:14:22.0354 vasca.tables:451 |DEBUG: AND selecting 'pos_cpval' [0.0001, inf], kept: 66.8831%
2024-11-08 14:14:22.0360 vasca.tables:451 |DEBUG: AND selecting 'pos_xv' [-inf, 2], kept: 66.4976%
2024-11-08 14:14:22.0371 vasca.tables:451 |DEBUG: AND selecting 'coadd_ffactor' [2.0, inf], kept: 0.0812%
2024-11-08 14:14:22.0377 vasca.tables:451 |DEBUG: AND selecting 'coadd_fdiff_s2n' [7, inf], kept: 0.0609%
2024-11-08 14:14:22.0385 vasca.tables:500 |INFO: Total table entries 4928, pre-selected rows 56, rows after new selection 59
2024-11-08 14:14:22.0397 vasca.region:805 |DEBUG: Synchronizing selection in table tt_sources
2024-11-08 14:14:22.0407 vasca.region:805 |DEBUG: Synchronizing selection in table tt_detections
2024-11-08 14:14:22.0419 vasca.region:805 |DEBUG: Synchronizing selection in table tt_coadd_sources
2024-11-08 14:14:22.0431 vasca.region:412 |DEBUG: Setting table tt_src_id_map.
2024-11-08 14:14:22.9331 vasca.tables:156 |DEBUG: Adding table 'region:tt_src_id_map'
# View all table attributes that have been added to the region object
rg.info()
Show code cell output
tt_fields:
<Table length=3>
name dtype unit description
----------------- ------- ---- -------------------------------------------------------
field_id bytes32 Field source ID number
field_name bytes32 Field name
project bytes32 Field project, typically survey name
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
observatory bytes22 Telescope of the observation (e.g. GALEX)
obs_filter bytes8 Filter of the observation (e.g. NUV)
fov_diam float32 deg Field radius or box size (depending on the observatory)
sel bool Selection of rows for VASCA analysis
nr_vis int32 Total number of visits of the field
time_bin_size_sum float32 s Total exposure time
time_start float64 d Start date and time in MJD
time_stop float64 d End time of last exposure
rg_fd_id int64 Region field ID number
{'DATAPATH': 'None', 'INFO': 'Field information table'}
tt_filters:
<Table length=2>
name dtype description
-------------- ------ ---------------------------------------
obs_filter_id int32 Observation filter ID number
obs_filter bytes8 Filter of the observation (e.g. NUV)
obs_filter_idx int32 Filter index in filter dependent arrays
{'INFO': 'Filters, their IDs and index, the last is specific for this region.'}
tt_visits:
<Table length=39>
name dtype unit description
-------------- ------- ---- -----------------------------------------
vis_id uint64 Visit ID number
time_bin_start float64 d Visit exposure start date and time in MJD
time_bin_size float32 s Visit exposure time in s
sel bool Selection of rows for VASCA analysis
obs_filter_id int32 Observation filter ID number
{'INFO': 'Visit information table'}
tt_sources:
<Table length=4928>
name dtype shape unit description
--------------- ------- ----- --------- -------------------------------------------------------------
fd_src_id int32 Source ID associated to the visit detection
nr_det int32 (1,) Number of detections
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
pos_err float32 arcsec Sky coordinate position error
pos_xv float32 arcsec2 Sky position excess variance
pos_var float32 arcsec2 Sky position variance
pos_cpval float32 Sky position quality
pos_rchiq float32 Sky position reduced chisquared of the constant mean
coadd_src_id int64 Co-add source ID number
coadd_dist float32 arcsec Angular distance to associated source
obs_filter_id int32 (1,) Observation filter ID number
sel bool
flux float32 (1,) 1e-06 Jy Flux density
flux_err float32 (1,) 1e-06 Jy Flux density error
flux_nxv float32 (1,) Flux normalised excess variance
flux_var float32 (1,) 1e-12 Jy2 Flux variance
flux_cpval float32 (1,) Probability value for a constant flux from the chisquare test
flux_rchiq float32 (1,) Flux reduced chisquared of the constant mean
coadd_ffactor float32 Source flux divided by flux of the associated co-add source
coadd_fdiff_s2n float32 Signal to noise of the flux difference
rg_fd_id int64 Region field ID number
rg_src_id int32 Region source ID number
nr_fd_srcs int32 Number of field sources
{'INFO': 'Source infomation table', 'CLUSTALG': 'None'}
tt_detections:
<Table length=42787>
name dtype unit description
-------------- ------- -------- ------------------------------------------------------------------------------------------------
vis_id uint64 Visit ID number
fd_src_id int32 Source ID associated to the visit detection
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
pos_err float32 arcsec Sky coordinate position error
flux float32 1e-06 Jy Flux density
flux_err float32 1e-06 Jy Flux density error
flux_app_ratio float32 Flux ratio measured at a different apertures (3.8 arcsec / 6 arcsec radius for GALEX)
s2n float32 Flux signal to noise
obs_filter_id int32 Observation filter ID number
sel bool
r_fov float32 deg Distance from center of FOV in degrees
artifacts int64 Logical OR of artifact flags
class_star float32 Point-source probability: 0.0 (resolved), 1.0 (unresolved, mcat file filter_CLASS_STAR variable)
chkobj_type int32 Detection matched to a known star (bright_match=1, mcat file chkobj_type variable)
size_world float32 arcsec Mean RMS of the ellipse size: (major axis + minor axis) / 2
ellip_world float32 Ellipticity of the detection or source
flux_auto float32 Jy Flux from SExtractor auto option
flux_auto_err float32 Jy Flux error from a fixed circular aperture (3.8 arcsec radius for GALEX)
rg_fd_id int64 Region field ID number
rg_src_id int32 Region source ID number
{'INFO': 'Visit detections table', 'Name': 'tt_detections'}
tt_coadd_detections:
<Table length=54330>
name dtype unit description
-------------- ------- -------- -------------------------------------------------------------------------------------
det_id int64 Reference source ID number
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
pos_err float32 arcsec Sky coordinate position error
flux float32 1e-06 Jy Flux density
flux_err float32 1e-06 Jy Flux density error
flux_app_ratio float32 Flux ratio measured at a different apertures (3.8 arcsec / 6 arcsec radius for GALEX)
s2n float32 Flux signal to noise
obs_filter_id int32 Observation filter ID number
sel bool Selection of rows for VASCA analysis
rg_fd_id int64 Region field ID number
coadd_src_id int64 Co-add source ID number
{'INFO': 'Reference detections table', 'Name': 'tt_coadd_detections'}
tt_coadd_sources:
<Table length=45315>
name dtype shape unit description
--------------- ------- ----- --------- -------------------------------------------------------------
fd_src_id int32 Source ID associated to the visit detection
nr_det int32 (1,) Number of detections
ra float64 deg Sky coordinate Right Ascension (J2000)
dec float64 deg Sky coordinate Declination (J2000)
pos_err float32 arcsec Sky coordinate position error
pos_xv float32 arcsec2 Sky position excess variance
pos_var float32 arcsec2 Sky position variance
pos_cpval float32 Sky position quality
pos_rchiq float32 Sky position reduced chisquared of the constant mean
coadd_src_id int64 Co-add source ID number
coadd_dist float32 arcsec Angular distance to associated source
obs_filter_id int32 (1,) Observation filter ID number
sel bool
flux float32 (1,) 1e-06 Jy Flux density
flux_err float32 (1,) 1e-06 Jy Flux density error
flux_nxv float32 (1,) Flux normalised excess variance
flux_var float32 (1,) 1e-12 Jy2 Flux variance
flux_cpval float32 (1,) Probability value for a constant flux from the chisquare test
flux_rchiq float32 (1,) Flux reduced chisquared of the constant mean
coadd_ffactor float32 Source flux divided by flux of the associated co-add source
coadd_fdiff_s2n float32 Signal to noise of the flux difference
rg_fd_id int64 Region field ID number
nr_fd_dets int32 Number of field detections
rg_src_id int32 Region source ID number
{'INFO': 'Source infomation table', 'CLUSTALG': 'None'}
tt_src_id_map:
<Table length=5597>
name dtype description
--------- ----- -------------------------------------------
rg_src_id int32 Region source ID number
rg_fd_id int64 Region field ID number
fd_src_id int32 Source ID associated to the visit detection
sel bool Selection of rows for VASCA analysis
{'INFO': 'Map between region and field source IDs'}
Show code cell source
# View all sources that passed the selection
df_sources = (
vutils.select_obs_filter(rg.tt_sources, obs_filter_id=1)
.to_pandas()
.apply(lambda x: x.str.decode("utf-8") if x.dtype == "O" else x)
)
df_select = df_sources.query("sel")
show(
df_select,
classes="display nowrap compact",
scrollY="300px",
scrollCollapse=True,
paging=False,
columnDefs=[{"className": "dt-body-left", "targets": "_all"}],
)
| fd_src_id | nr_det | ra | dec | pos_err | pos_xv | pos_var | pos_cpval | pos_rchiq | coadd_src_id | coadd_dist | obs_filter_id | sel | flux | flux_err | flux_nxv | flux_var | flux_cpval | flux_rchiq | coadd_ffactor | coadd_fdiff_s2n | rg_fd_id | rg_src_id | nr_fd_srcs | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Loading ITables v2.2.3 from the init_notebook_mode cell...
(need help?) |
Pipeline Output¶
With a few simple export functions the full region file, the variable source catalog and its pipeline configuration are saved to the pipeline output directory.
This concludes the tutorial. Readers are invited to look into the post-processing notebooks as listed here.
import yaml
# Write region file
rg.write_to_fits(file_name=pipe_dir / f'region_{config["general"]["name"]}.fits')
# Export variable source catalog (only selected sources are included)
rc = rg.get_region_catalog()
rc.write_to_fits(file_name=pipe_dir / f'region_{config["general"]["name"]}_cat.fits')
# Write used config file
yaml_out_name = pipe_dir / f'cfg_ran_{config["general"]["name"]}.yaml'
with open(yaml_out_name, "w") as yaml_file:
yaml.dump(config, yaml_file)
logger.info("Done running VASCA pipeline.")
Show code cell output
2024-11-08 14:14:23.0022 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/region_simple_pipe.fits'
2024-11-08 14:14:23.0048 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-08 14:14:23.0264 vasca.tables:251 |DEBUG: Writing table 'tt_filters'
2024-11-08 14:14:23.0339 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-08 14:14:23.0435 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-08 14:14:23.0832 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-08 14:14:23.1335 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-08 14:14:23.1649 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_sources'
2024-11-08 14:14:23.2265 vasca.tables:251 |DEBUG: Writing table 'tt_src_id_map'
2024-11-08 14:14:23.3062 vasca.region:805 |DEBUG: Synchronizing selection in table tt_sources
2024-11-08 14:14:23.3079 vasca.region:805 |DEBUG: Synchronizing selection in table tt_detections
2024-11-08 14:14:23.3093 vasca.region:805 |DEBUG: Synchronizing selection in table tt_coadd_sources
2024-11-08 14:14:23.3105 vasca.region:805 |DEBUG: Synchronizing selection in table tt_src_id_map
2024-11-08 14:14:23.3130 vasca.tables:156 |DEBUG: Adding table 'tt_fields'
2024-11-08 14:14:23.3150 vasca.tables:156 |DEBUG: Adding table 'tt_filters'
2024-11-08 14:14:23.3163 vasca.tables:156 |DEBUG: Adding table 'tt_visits'
2024-11-08 14:14:23.3185 vasca.tables:156 |DEBUG: Adding table 'tt_sources'
2024-11-08 14:14:23.3219 vasca.tables:156 |DEBUG: Adding table 'tt_detections'
2024-11-08 14:14:23.4079 vasca.tables:156 |DEBUG: Adding table 'tt_coadd_detections'
2024-11-08 14:14:23.4121 vasca.tables:156 |DEBUG: Adding table 'tt_coadd_sources'
2024-11-08 14:14:23.4145 vasca.tables:156 |DEBUG: Adding table 'tt_src_id_map'
2024-11-08 14:14:23.4155 vasca.tables:211 |INFO: Writing file with name '/opt/hostedtoolcache/Python/3.11.10/x64/lib/python3.11/site-packages/docs/tutorial_resources/vasca_pipeline/simple_pipe/region_simple_pipe_cat.fits'
2024-11-08 14:14:23.4178 vasca.tables:251 |DEBUG: Writing table 'tt_fields'
2024-11-08 14:14:23.4390 vasca.tables:251 |DEBUG: Writing table 'tt_filters'
2024-11-08 14:14:23.4469 vasca.tables:251 |DEBUG: Writing table 'tt_visits'
2024-11-08 14:14:23.4567 vasca.tables:251 |DEBUG: Writing table 'tt_sources'
2024-11-08 14:14:23.4995 vasca.tables:251 |DEBUG: Writing table 'tt_detections'
2024-11-08 14:14:23.5322 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_detections'
2024-11-08 14:14:23.5628 vasca.tables:251 |DEBUG: Writing table 'tt_coadd_sources'
2024-11-08 14:14:23.6003 vasca.tables:251 |DEBUG: Writing table 'tt_src_id_map'
2024-11-08 14:14:23.6942 __main__:15 |INFO: Done running VASCA pipeline.
# To be continued ...