The NOIRLab Source Catalog (NSC) is a catalog of nearly all of the public imaging data in NOIRLab’s Astro Data Archive. These images from telescopes in both hemispheres nearly cover the entire sky, but data are principally (~98%) from the Dark Energy Camera (DECam) on the Blanco 4-meter telescope at Cerro Tololo Inter-American Observatory (CTIO). The Asteroid Institute’s THOR run detailed here was performed on the second data release (DR2), spanning 412,116 images taken from October 2012 through October 2019.

Institutional collaborators had indicated that NSC would likely contain observations of many undiscovered asteroids, but the specifics of the dataset present some major challenges. Firstly, the data reduction pipeline did not include a difference imaging step, and so static sources are included. To complicate matters, the data are deep enough (limiting magnitude of 23.5 in some cases) that existing static source catalogs are of limited utility in ruling out sources via a positional crossmatch. This necessitated our writing a bespoke static source filtering regime that, while it did retain a small number of static sources, did reduce the point source database from 68bn points to a more manageable 1.7bn

Again complicating matters is the sheer depth of the survey. NSC has many deep stares, or repeat observations of crowded fields. While this is a domain in which THOR excels, the computational complexity, especially in clustering, can grow quite large. This is one of the key reasons we needed to employ such large scale compute in order to process the dataset in a reasonable timeframe. These computational realities heavily informed the compute architecture described below.

The analysis pipeline starts with data ingestion. NOIRLab Source Catalog (DR2) is one of many telescope catalogs that are ingested into a Unified Dataset, though it is the only dataset we used for this discovery run. These data are filtered and normalized to ideally contain only point source detections that represent likely moving sources. Those point sources stored in two formats: first in BigQuery for easy access and large scale analysis. The second destination is an indexed binary format, used as part of our Precovery service (Precovery DB).

The linking is performed by THOR (for more information on THOR, check out the repository) on thousands of simultaneous containers inside Google Kubernetes Engine. An orchestrator partitions out the data, generates the test orbits based on the input partition, and initializes a container for each test orbit. As each test orbit completes, it uploads results to a cloud storage bucket. This process is check-pointed and runs on spot instances to minimize costs.

The recovered orbits produced by THOR then go through an additional merge and extend process called Iterative Precovery and Orbit Determination (IPOD). This step removes duplicate orbits, and uses our Precovery service to extend the arcs of the orbits. The refined orbits are then uploaded to cloud storage.

The list of refined orbits represents the candidates discoveries, but due to the nature of this dataset, there are many false positives. Chiefly these false positives are cosmic ray hits which survived source extraction, and static sources that evaded our filters. To confirm our quality discovery candidates, millions of cutout images are fetched from NOIRLab and stored in cloud storage in order to perform validation. The images and detections in the discovery candidates are sent to a machine learning model that scores the individual detections. These results are sorted by those scores, and then validated by a human expert. Finally, discoveries are submitted to the Minor Planet Center for review.