Modeling, Systems Engineering, and Project Management for Astronomy IV, 2010
The Large Synoptic Survey Telescope (LSST) project has evolved from just a few staff members in 2... more The Large Synoptic Survey Telescope (LSST) project has evolved from just a few staff members in 2003 to about 100 in 2010; the affiliation of four founding institutions has grown to 32 universities, government laboratories, and industry. The public private collaboration aims to complete the estimated $450 M observatory in the 2017 timeframe. During the design phase of the project
Software and Cyberinfrastructure for Astronomy, 2010
Large ground-based and space-based telescopes are expected to make exciting discoveries in the up... more Large ground-based and space-based telescopes are expected to make exciting discoveries in the upcoming decade. These large projects start their construction phase many years before first-light and continue to operate for many years after first-light and usually span multiple countries. The file-storage cyberinfrastructure ("file-storage CI") of these largescale projects has to evolve over several years from a conceptual prototype to
The project for the proposed Large Synoptic Survey Telescope (LSST) performed more than two years... more The project for the proposed Large Synoptic Survey Telescope (LSST) performed more than two years of data collection, site evaluation, and analysis to support the selection of its prime site. LSST assessment was based on using an existing site with existing infrastructure and historical performance information. A large and diverse set of comparative information was compiled for potential sites using results from other site campaigns, measurements from existing large telescopes, new astro-climate measurements, logistical and feasibility information, and from existing satellite and climate databases. Several analyses were performed on these data including the assessment of survey performance using the LSST operation simulator. An independent site selection committee of experts provided recommendations to the Project leading to three finalist sites, one in Mexico, and two in northern Chile. The finalist sites were assessed thoroughly with additional data collection from all-sky cameras...
The LSST produces data products at two cadences. The nightly data products consist of raw images,... more The LSST produces data products at two cadences. The nightly data products consist of raw images, low-latency alerts, and updates to the object, source, and orbit catalogs, all of which are archived and publicly available. Some additional data produced by the nightly pipelines, such as calibrated and subtracted images, are not necessarily archived but can be recreated on demand. At longer intervals, approximately twice a year, data releases are produced. A data release is the result of processing the survey images accumulated to date through the deep detection and classification pipelines, as well as the image processing pipeline. Once produced and validated, a data release is frozen and archived as an entity that can be readily accessed in the future. This will facilitate publishing papers that refer to specific LSST data releases. The principal data products in a release are the object catalog and stacked images. The object catalog includes a full range of object properties as co-measured from all images in all bands which contain the object, light curves over the duration of the survey, and object classifications. The photometry and astrometry in this catalog will be calibrated to a more precise level than available from the nightly data products, and in general the intent is that most science done with LSST will use this catalog directly. The stacked images are used internally for production of template images, as well as for producing RGB images useful for visual understanding of objects and for EPO.
The Large Synoptic Survey Telescope is a large aperture, wide-field, ground-based telescope that ... more The Large Synoptic Survey Telescope is a large aperture, wide-field, ground-based telescope that will survey half the sky every few nights in six visible bands. The LSST system provides an end-to-end astronomical survey facility to acquire, process, analyze, catalog, and preserve the world's largest database of optical astronomical data. LSST will also open the time domain for studies of transient
The Large Synoptic Survey Telescope (LSST) will have a Science Data Quality Analysis (SDQA) subsy... more The Large Synoptic Survey Telescope (LSST) will have a Science Data Quality Analysis (SDQA) subsystem for vetting its unprecedented volume of astronomical image data. The SDQA subsystem inhabits three basic realms: image processing, graphical-user-interface (GUI) tools, and alarms/reporting. During pipeline image processing, SDQA data are computed for the images and astronomical sources extracted from the images, and utilized to grade the images and sources. Alarms are automatically sent, if necessary, to initiate swift response to problems found. Both SDQA data and machine-determined grades are stored in a database. At the end of a data-processing interval, e.g., nightly processing or data-release reprocessing, automatic SDQA reports are generated from SDQA data and grades queried from the database. The SDQA reports summarize the science data quality and provide feedback to telescope, camera, facility, observation-scheduling and data-processing personnel. During operations, GUI tools facilitate visualization of image and SDQA data in a variety of ways that allow a small SDQA-operations team of humans to quickly and easily perform manual SDQA on a substantial fraction of LSST data products, and possibly reassign SDQA grades as a result of the visual inspection.
Proceedings of Spie the International Society For Optical Engineering, 2010
The LSST Data Management System is built on an open source software framework that has middleware... more The LSST Data Management System is built on an open source software framework that has middleware and application layers. The middleware layer provides capabilities to construct, configure, and manage pipelines on clusters of processing nodes, and to manage the data the pipelines consume and produce. It is not in any way specific to astronomical applications. The complementary application layer provides
Software and Cyberinfrastructure for Astronomy, 2010
The Data Management system for the LSST will have to perform near-real-time calibration and analy... more The Data Management system for the LSST will have to perform near-real-time calibration and analysis of acquired images, particularly for transient detection and alert generation; annual processing of the entire dataset for precision calibration, object detection and characterization, and catalog generation; and support of user data access and analysis. Images will be acquired at roughly a 17-second cadence, with alerts generated within one minute. The ten-year survey will result in tens of petabytes of image and catalog data and will require ~250 teraflops of processing to reduce. The LSST project is carrying out a series of Data Challenges (DC) to refine the design, evaluate the scientific and computational performance of candidate algorithms, and address the challenging scaling issues that the LSST dataset will present. This paper discusses the progress of the DCs to date and plans for future DCs. Algorithm development must address dual requirements for the efficient use of computational resources and the accurate, reliable processing of the deep and broad survey data. The DCs incorporate both existing astronomical images and image data resulting from detailed photon-level simulations. The data is used to ensure that the system can scale to the LSST field of view and 3.2 gigapixel camera scale and meet the scientific data quality requirements. Future DCs, carried out in conjunction with the LSST Science Collaborations, are planned to deliver data products verified by computeraided analysis and actual applications as suitable for high-quality science.
The LSST will be operated in survey mode only. The primary goals of LSST operations are to: 1) ma... more The LSST will be operated in survey mode only. The primary goals of LSST operations are to: 1) maintain the end-to-end survey throughput at a level consistent with achieving the goals for the number of individual exposures at the level specified in the Science Requirements Document within 10 years; 2) achieve the specifications on image quality, photometric and astrometric accuracy, etc.; and 3) enable effective use of the data by a broad community of users. In order to maintain the cadence required, the observations, data processing, data quality assurance, archiving, and access for the community must all be highly automated. The LSST operations plan describes the operational model, the facilities required and their locations, and the services that will be provided.
Observatory Operations: Strategies, Processes, and Systems IV, 2012
The time domain has been identified as one of the most important areas of astronomical research f... more The time domain has been identified as one of the most important areas of astronomical research for the next decade. The Virtual Observatory is in the vanguard with dedicated tools and services that enable and facilitate the discovery, dissemination and analysis of time domain data. These range in scope from rapid notifications of time-critical astronomical transients to annotating long-term variables with the latest modelling results. In this paper, we will review the prior art in these areas and focus on the capabilities that the VAO is bringing to bear in support of time domain science. In particular, we will focus on the issues involved with the heterogeneous collections of (ancilllary) data associated with astronomical transients, and the time series characterization and classification tools required by the next generation of sky surveys, such as LSST and SKA.
Managing the evolution of the LSST data management system. [Proceedings of SPIE 6274, 62740P (200... more Managing the evolution of the LSST data management system. [Proceedings of SPIE 6274, 62740P (2006)]. Jeffrey P. Kantor. Abstract. The Large Synoptic Survey Telescope (LSST) is planned to start construction in early 2009 and achieve first light in late 2012. ...
In science observing mode, the LSST telescope and camera system will deliver a 3.2 Gpixel image e... more In science observing mode, the LSST telescope and camera system will deliver a 3.2 Gpixel image every 12 sec, a data rate of about 0.5 GByte/sec. The data processing pipeline must process this incoming data to produce the LSST's primary data products: the calibrated image archive; the detection catalog; the object catalog; and real-time alerts. Additionally, the quality of the incoming data must be rapidly assessed and fed back to the observatory control system. The pipeline must be flexible enough to allow addition of new processing stages and replacement of existing algorithms with improved ones, and must be robust in the face of the failure of hardware components such as disk drives. We present a preliminary design of the primary LSST data products, and of the data processing pipeline. The mapping of the pipeline onto computing hardware is discussed, along with estimates of the computational, I/O, and network bandwidths required.
Modeling, Systems Engineering, and Project Management for Astronomy IV, 2010
The Large Synoptic Survey Telescope (LSST) project has evolved from just a few staff members in 2... more The Large Synoptic Survey Telescope (LSST) project has evolved from just a few staff members in 2003 to about 100 in 2010; the affiliation of four founding institutions has grown to 32 universities, government laboratories, and industry. The public private collaboration aims to complete the estimated $450 M observatory in the 2017 timeframe. During the design phase of the project
Software and Cyberinfrastructure for Astronomy, 2010
Large ground-based and space-based telescopes are expected to make exciting discoveries in the up... more Large ground-based and space-based telescopes are expected to make exciting discoveries in the upcoming decade. These large projects start their construction phase many years before first-light and continue to operate for many years after first-light and usually span multiple countries. The file-storage cyberinfrastructure ("file-storage CI") of these largescale projects has to evolve over several years from a conceptual prototype to
The project for the proposed Large Synoptic Survey Telescope (LSST) performed more than two years... more The project for the proposed Large Synoptic Survey Telescope (LSST) performed more than two years of data collection, site evaluation, and analysis to support the selection of its prime site. LSST assessment was based on using an existing site with existing infrastructure and historical performance information. A large and diverse set of comparative information was compiled for potential sites using results from other site campaigns, measurements from existing large telescopes, new astro-climate measurements, logistical and feasibility information, and from existing satellite and climate databases. Several analyses were performed on these data including the assessment of survey performance using the LSST operation simulator. An independent site selection committee of experts provided recommendations to the Project leading to three finalist sites, one in Mexico, and two in northern Chile. The finalist sites were assessed thoroughly with additional data collection from all-sky cameras...
The LSST produces data products at two cadences. The nightly data products consist of raw images,... more The LSST produces data products at two cadences. The nightly data products consist of raw images, low-latency alerts, and updates to the object, source, and orbit catalogs, all of which are archived and publicly available. Some additional data produced by the nightly pipelines, such as calibrated and subtracted images, are not necessarily archived but can be recreated on demand. At longer intervals, approximately twice a year, data releases are produced. A data release is the result of processing the survey images accumulated to date through the deep detection and classification pipelines, as well as the image processing pipeline. Once produced and validated, a data release is frozen and archived as an entity that can be readily accessed in the future. This will facilitate publishing papers that refer to specific LSST data releases. The principal data products in a release are the object catalog and stacked images. The object catalog includes a full range of object properties as co-measured from all images in all bands which contain the object, light curves over the duration of the survey, and object classifications. The photometry and astrometry in this catalog will be calibrated to a more precise level than available from the nightly data products, and in general the intent is that most science done with LSST will use this catalog directly. The stacked images are used internally for production of template images, as well as for producing RGB images useful for visual understanding of objects and for EPO.
The Large Synoptic Survey Telescope is a large aperture, wide-field, ground-based telescope that ... more The Large Synoptic Survey Telescope is a large aperture, wide-field, ground-based telescope that will survey half the sky every few nights in six visible bands. The LSST system provides an end-to-end astronomical survey facility to acquire, process, analyze, catalog, and preserve the world's largest database of optical astronomical data. LSST will also open the time domain for studies of transient
The Large Synoptic Survey Telescope (LSST) will have a Science Data Quality Analysis (SDQA) subsy... more The Large Synoptic Survey Telescope (LSST) will have a Science Data Quality Analysis (SDQA) subsystem for vetting its unprecedented volume of astronomical image data. The SDQA subsystem inhabits three basic realms: image processing, graphical-user-interface (GUI) tools, and alarms/reporting. During pipeline image processing, SDQA data are computed for the images and astronomical sources extracted from the images, and utilized to grade the images and sources. Alarms are automatically sent, if necessary, to initiate swift response to problems found. Both SDQA data and machine-determined grades are stored in a database. At the end of a data-processing interval, e.g., nightly processing or data-release reprocessing, automatic SDQA reports are generated from SDQA data and grades queried from the database. The SDQA reports summarize the science data quality and provide feedback to telescope, camera, facility, observation-scheduling and data-processing personnel. During operations, GUI tools facilitate visualization of image and SDQA data in a variety of ways that allow a small SDQA-operations team of humans to quickly and easily perform manual SDQA on a substantial fraction of LSST data products, and possibly reassign SDQA grades as a result of the visual inspection.
Proceedings of Spie the International Society For Optical Engineering, 2010
The LSST Data Management System is built on an open source software framework that has middleware... more The LSST Data Management System is built on an open source software framework that has middleware and application layers. The middleware layer provides capabilities to construct, configure, and manage pipelines on clusters of processing nodes, and to manage the data the pipelines consume and produce. It is not in any way specific to astronomical applications. The complementary application layer provides
Software and Cyberinfrastructure for Astronomy, 2010
The Data Management system for the LSST will have to perform near-real-time calibration and analy... more The Data Management system for the LSST will have to perform near-real-time calibration and analysis of acquired images, particularly for transient detection and alert generation; annual processing of the entire dataset for precision calibration, object detection and characterization, and catalog generation; and support of user data access and analysis. Images will be acquired at roughly a 17-second cadence, with alerts generated within one minute. The ten-year survey will result in tens of petabytes of image and catalog data and will require ~250 teraflops of processing to reduce. The LSST project is carrying out a series of Data Challenges (DC) to refine the design, evaluate the scientific and computational performance of candidate algorithms, and address the challenging scaling issues that the LSST dataset will present. This paper discusses the progress of the DCs to date and plans for future DCs. Algorithm development must address dual requirements for the efficient use of computational resources and the accurate, reliable processing of the deep and broad survey data. The DCs incorporate both existing astronomical images and image data resulting from detailed photon-level simulations. The data is used to ensure that the system can scale to the LSST field of view and 3.2 gigapixel camera scale and meet the scientific data quality requirements. Future DCs, carried out in conjunction with the LSST Science Collaborations, are planned to deliver data products verified by computeraided analysis and actual applications as suitable for high-quality science.
The LSST will be operated in survey mode only. The primary goals of LSST operations are to: 1) ma... more The LSST will be operated in survey mode only. The primary goals of LSST operations are to: 1) maintain the end-to-end survey throughput at a level consistent with achieving the goals for the number of individual exposures at the level specified in the Science Requirements Document within 10 years; 2) achieve the specifications on image quality, photometric and astrometric accuracy, etc.; and 3) enable effective use of the data by a broad community of users. In order to maintain the cadence required, the observations, data processing, data quality assurance, archiving, and access for the community must all be highly automated. The LSST operations plan describes the operational model, the facilities required and their locations, and the services that will be provided.
Observatory Operations: Strategies, Processes, and Systems IV, 2012
The time domain has been identified as one of the most important areas of astronomical research f... more The time domain has been identified as one of the most important areas of astronomical research for the next decade. The Virtual Observatory is in the vanguard with dedicated tools and services that enable and facilitate the discovery, dissemination and analysis of time domain data. These range in scope from rapid notifications of time-critical astronomical transients to annotating long-term variables with the latest modelling results. In this paper, we will review the prior art in these areas and focus on the capabilities that the VAO is bringing to bear in support of time domain science. In particular, we will focus on the issues involved with the heterogeneous collections of (ancilllary) data associated with astronomical transients, and the time series characterization and classification tools required by the next generation of sky surveys, such as LSST and SKA.
Managing the evolution of the LSST data management system. [Proceedings of SPIE 6274, 62740P (200... more Managing the evolution of the LSST data management system. [Proceedings of SPIE 6274, 62740P (2006)]. Jeffrey P. Kantor. Abstract. The Large Synoptic Survey Telescope (LSST) is planned to start construction in early 2009 and achieve first light in late 2012. ...
In science observing mode, the LSST telescope and camera system will deliver a 3.2 Gpixel image e... more In science observing mode, the LSST telescope and camera system will deliver a 3.2 Gpixel image every 12 sec, a data rate of about 0.5 GByte/sec. The data processing pipeline must process this incoming data to produce the LSST's primary data products: the calibrated image archive; the detection catalog; the object catalog; and real-time alerts. Additionally, the quality of the incoming data must be rapidly assessed and fed back to the observatory control system. The pipeline must be flexible enough to allow addition of new processing stages and replacement of existing algorithms with improved ones, and must be robust in the face of the failure of hardware components such as disk drives. We present a preliminary design of the primary LSST data products, and of the data processing pipeline. The mapping of the pipeline onto computing hardware is discussed, along with estimates of the computational, I/O, and network bandwidths required.
Uploads
Papers by Jeffrey Kantor