Exploring No-SQL alternatives for ALMA monitoring system

Software and Cyberinfrastructure for Astronomy II, 2012

The Atacama Large Millimeter /submillimeter Array (ALMA) will be a unique research instrument composed of at least 66 reconfigurable high-precision antennas, located at the Chajnantor plain in the Chilean Andes at an elevation of 5000 m. Each antenna contains instruments capable of receiving radio signals from 31.3 GHz up to 950 GHz. These signals are correlated inside a Correlator and the spectral data are finally saved into the Archive system together with the observation metadata. This paper describes the progress in the development of the ALMA operation support software, which aims to increase the efficiency of the testing, distribution, deployment and operation of the core observing software. This infrastructure has become critical as the main array software evolves during the construction phase. In order to support and maintain the core observing software, it is essential to have a mechanism to align and distribute the same version of software packages across all systems. This is achieved rigorously with weekly based regression tests and strict configuration control. A build farm to provide continuous integration and testing in simulation has been established as well. Given the large amount of antennas, it is imperative to have also a monitoring system to allow trend analysis of each component in order to trigger preventive maintenance activities. A challenge for which we are preparing this year consists in testing the whole ALMA software performing complete end-to-end operation, from proposal submission to data distribution to the ALMA Regional Centers. The experience gained during deployment, testing and operation support will be presented.

The Atacama Large Millimeter Array (ALMA) is a project to build a radio interferometric telescope containing a large number of antennas (nominally 64) at a high site in Chile operating in the mm and sub-mm spectral region. With the addition of a Japan in the last year, ALMA is now a global partnership with participation by institutes in Asia, Europe, and North America. The scope of the ALMA Software includes all aspects: observing script creation through GUIs, dynamic scheduling depending on weather and instrumental parameters, instrument control (including the correlator device, capable of producing data at more than 1 GB/s), data handling and formatting, data archiving and retrieval, and automatic and manual data processing systems. The scope has recently been increased to support telescope operations (e.g., referee support). This ambitious scope is being implemented by a very distributed team, with approximately 60 members at institutes in 10 countries. This paper will describe s...

The design of a real-time Linux application utilizing Real-Time Application Interface (RTAI) to process real-time data from the radio astronomy correlator for the Atacama Large Millimeter Array (ALMA) is described. The correlator is a custom-built digital signal processor which computes the cross-correlation function of two digitized signal streams. ALMA will have 64 antennas with 2080 signal streams each with a sample rate of 4 giga-samples per second. The correlator's aggregate data output will be 1 gigabyte per second. The software is defined by hard deadlines with high input and processing data rates, while requiring interfaces to non real-time external computers. The designed computer system – the Correlator Data Processor or CDP, consists of a cluster of 17 SMP computers, 16 of which are compute nodes plus a master controller node all running real-time Linux kernels. Each compute node uses an RTAI kernel module to interface to a 32-bit parallel inter-face which accepts raw...

Advances in Space …, 2010

The tracking of large-scale interplanetary (IP) disturbances traveling from the Sun to the Earth is a key issue in space weather studies. The Mexican Array Radio Telescope (MEXART) applies the Interplanetary Scintillation (IPS) technique to detect these solar wind disturbances and it will participate in a global warning network of space weather forecasting. We describe the data storage and computational processes carried out to manage the instrument’s real time data. These procedures are important for the MEXART calibration, operation and the scientific data reduction.

TELEIOS is a recent European project that addresses the need for scalable access to petabytes of Earth Observation data and the discovery and exploitation of knowledge that is hidden in them. TELEIOS builds on scientific database technologies (array databases, SciQL, data vaults) and Semantic Web technologies (stRDF and stSPARQL) implemented on top of a state of the art column store database system (MonetDB). We demonstrate a first prototype of the TELEIOS Virtual Earth Observatory (VEO) architecture, using a forest fire monitoring application as example

The Worldwide LHC Computing Grid (WLCG) today includes more than 150 computing centres where more than 2 million jobs are being executed daily and petabytes of data are transferred between sites. Monitoring the computing activities of the LHC experiments, over such a huge heterogeneous infrastructure, is extremely demanding in terms of computation, performance and reliability. Furthermore, the generated monitoring flow is constantly increasing, which represents another challenge for the monitoring systems. While existing solutions are traditionally based on Oracle for data storage and processing, recent developments evaluate NoSQL for processing large-scale monitoring datasets. NoSQL databases are getting increasingly popular for processing datasets at the terabyte and petabyte scale using commodity hardware. In this contribution, the integration of NoSQL data processing in the Experiment Dashboard framework is described along with first experiences of using this technology for monitoring the LHC computing activities.

Proceedings of the 9th International Conference on Cognitive Radio Oriented Wireless Networks, 2014

This paper describes the RF measurement storage and database architecture for long-term continuously running spectrum observatories in the US (at Chicago and Blacksburg) and in Turku, Finland. It also describes how the measurement band plan has been collaboratively created taking into account the specifications and limits of the receiving system. Collecting measurements from different geographical locations makes it possible to perform spatial and temporal analysis of the spectrum usage in different parts of the world. As the multi-site measurements are aggregated into one central database, very high requirements are set on the database architecture, equipment, capacity, speed and number of data connections.

Observatory Operations: Strategies, Processes, and Systems V, 2014

The Atacama Large Millimeter /submillimeter Array (ALMA) will be a unique research instrument composed of at least 66 reconfigurable high-precision antennas, located at the Chajnantor plain in the Chilean Andes at an elevation of 5000 m. The observatory has another office located in Santiago of Chile, 1600 km from the Chajnantor plain. In the Atacama desert, the wonderful observing conditions imply precarious living conditions and extremely high operation costs: i.e: flight tickets, hospitality, infrastructure, water, electricity, etc. It is clear that a purely remote operational model is impossible, but we believe that a mixture of remote and local operation scheme would be beneficial to the observatory, not only in reducing the cost but also in increasing the observatory overall efficiency. This paper describes the challenges and experience gained in such experimental proof of the concept. The experiment was performed over the existing 100 Mbps bandwidth, which connects both sites through a third party telecommunication infrastructure. During the experiment, all of the existent capacities of the observing software were validated successfully, although room for improvement was clearly detected. Network virtualization, MPLS configuration, L2TPv3 tunneling, NFS adjustment, operational workstations design are part of the experiment.

Scientific and Statistical …, 2007

This paper presents our experiences in porting the Sloan Digital Sky Survey (SDSS)/SkyServer to the state-of-the-art open source database system MonetDB/SQL. SDSS acts as a well-documented benchmark for scientific database management. We have achieved a fully functional prototype for the personal SkyServer, to be downloaded from our site. The lessons learned are 1) the column store approach of MonetDB demonstrates a great potential in the world of scientific databases. However, the application also ...

2020

After eight observing Cycles, the Atacama Large Millimeter-submillimeter Array (ALMA) is capable of observing in eight different bands (covering a frequency range from 84 to 950 GHz), with 66 antennas and two correlators. For the current Cycle (7), ALMA offers up to 4300 hours for the 12-m array, and 3000 hours on both the 7-m of the Atacama Compact Array (ACA) and TP Array plus 750 hours in a supplemental call. From the customer perspective (i.e., the astronomical community), ALMA is an integrated product service provider, i.e. it observes in service mode, processes and delivers the data obtained. The Data Management Group (DMG) is in charge of the processing, reviewing, and delivery of the ALMA data and consists of approximately 60 experts in data reduction, from the ALMA Regional Centers (ARCs) and the Joint ALMA Observatory (JAO), distributed in fourteen countries. Prior to their delivery, the ALMA data products go through a thorough quality assurance (QA) process, so that the a...