Inspired by the success of BESDIRAC, the distributed computing environment based on DIRAC for BES... more Inspired by the success of BESDIRAC, the distributed computing environment based on DIRAC for BESIII experiment, several other experiments operated by Institute of High Energy Physics (IHEP), such as Circular Electron Positron Collider (CEPC), Jiangmen Underground Neutrino Observatory (JUNO), Large High Altitude Air Shower Observatory (LHAASO) and Hard X-ray Modulation Telescope (HXMT) etc, are willing to use DIRAC to integrate the geographically distributed computing resources available by their collaborations. In order to minimize manpower and hardware cost, we extended the BESDIRAC platform to support multi-VO scenario, instead of setting up a self-contained distributed computing environment for each VO. This makes DIRAC as a service for the community of those experiments. To support multi-VO, the system architecture of BESDIRAC is adjusted for scalability. The VOMS and DIRAC servers are reconfigured to manage users and groups belong to several VOs. A lightweight storage resource manager StoRM is employed as the central SE to integrate local and grid data. A frontend system is designed for user's massive job splitting, submission and management, with plugins to support new VOs. A monitoring and accounting system is also considered to easy the system administration and VO related resources usage accounting.
Distributed computing is necessary nowadays for high energy physics experiments to organize heter... more Distributed computing is necessary nowadays for high energy physics experiments to organize heterogeneous computing resources all over the world to process enormous amounts of data. The BESIII experiment in China, has established its own distributed computing system, based on DIRAC, as a supplement to local clusters, collecting cluster, grid, desktop and cloud resources from collaborating member institutes around the world. The system consists of workload management and data management to deal with the BESIII Monte Carlo production workflow in a distributed environment. A dataset-based data transfer system has been developed to support data movements among sites. File and metadata management tools and a job submission frontend have been developed to provide a virtual layer for BESIII physicists to use distributed resources. Moreover, the paper shows the experience to cope with lack of grid experience and low manpower among the BESIII community.
9th International Conference "Distributed Computing and Grid Technologies in Science and Education", Dec 13, 2021
The modern Big Data ecosystem provides tools to build a flexible platform for processing data str... more The modern Big Data ecosystem provides tools to build a flexible platform for processing data streams and batch datasets. Supporting both the functioning of modern giant particle physics experiments and the services necessary for the work of many individual physics researchers results in generating and transferring large amounts of semi-structured data. Thus, it is promising to apply cutting-edge technologies to study these data flows and make the services' provisioning more effective. In this work, we describe the structure and implementation of our data analysis platform, built on the Apache Spark cluster. With the official support for GPU computing now available in Spark version 3, we propose a change in the architecture to utilize these more performant resources while keeping the platform's functionality provided by using mainstream Big Data software. Furthermore, the necessity for GPU support entails a change in the computing resource management infrastructure from Apache Mesos to Kubernetes. Finally, to demonstrate the features and operation of the system, we use the task of network packet analysis for security monitoring and anomaly detection in both batch and stream modes.
In this paper we report on LCG Monte-Carlo Data Base (MCDB) and software which has been developed... more In this paper we report on LCG Monte-Carlo Data Base (MCDB) and software which has been developed to operate MCDB. The main purpose of the LCG MCDB project is to provide a storage and documentation system for sophisticated event samples simulated for the LHC collaborations by experts. In many cases, the modern Monte-Carlo simulation of physical processes requires expert knowledge in Monte-Carlo generators or significant amount of CPU time to produce the events. MCDB is a knowledgebase mainly dedicated to accumulate simulated events of this type. The main motivation behind LCG MCDB is to make the sophisticated MC event samples available for various physical groups. All the data from MCDB is accessible in several convenient ways. LCG MCDB is being developed within the CERN LCG Application Area Simulation project.
Kompʹûternye issledovaniâ i modelirovanie, Jun 1, 2015
The Compact Muon Solenoid (CMS) is a high-performance general-purpose detector at the Large Hadro... more The Compact Muon Solenoid (CMS) is a high-performance general-purpose detector at the Large Hadron Collider (LHC) at CERN. A distributed data analysis system for processing and further analysis of CMS experimental data has been developed and this model foresees the obligatory usage of modern grid-technologies. The CMS Computing Model makes use of the hierarchy of computing centers (Tiers). The Joint Institute for Nuclear Research (JINR) takes an active part in the CMS experiment. In order to provide a proper computing infrastructure for the CMS experiment at JINR and for Russian institutes collaborating in CMS, Tier-1 center for the CMS experiment is constructing at JINR. The main tasks and services of the CMS Tier-1 at JINR are described. The status and perspectives of the Tier1 center for the CMS experiment at JINR are presented.
Physics of Particles and Nuclei Letters, Sep 1, 2016
The BES-III experiment at the IHEP CAS, Beijing, is running at the high-luminosity e+e-collider B... more The BES-III experiment at the IHEP CAS, Beijing, is running at the high-luminosity e+e-collider BEPC-II to study physics of charm quarks and tau leptons. The world largest samples of J/psi and psi' events are already collected, a number of unique data samples in the energy range 2.5-4.6 GeV have been taken. The data volume is expected to increase by an order of magnitude in the coming years. This requires to move from a centralized computing system to a distributed computing environment, thus allowing the use of computing resources from remote sites -members of the BES-III Collaboration. In this report the general information, latest results and development plans of the BES-III distributed computing system are presented.
The paper summarizes general information, the latest results and development plans of the BES-III... more The paper summarizes general information, the latest results and development plans of the BES-III distributed computing system. The BES-III experiment at the Institute of High Energy Physics (Beijing, China) is aimed at the precision measurements in e+e-annihilation at the energy range from 2.0 to 4.6 GeV. The world's largest samples of J/psi and psi' events and unique samples of XYZ data have been already collected. To process the BES-III experiment's data the distributed computing infrastructure based on DIRAC interware has been setup and became operational since 2013. The work was done in order to tune and enhance DIRAC itself for BES-III purposes: monitoring system and data transfer system were developed. Experience of the BES-III development team may be very useful for medium scale HEP experiments aiming to apply distributed data processing in their computing model.
Physics of Particles and Nuclei Letters, Sep 1, 2016
An overview of the JINR Tier-1 centre for the CMS experiment at the LHC is given. Special emphasi... more An overview of the JINR Tier-1 centre for the CMS experiment at the LHC is given. Special emphasis is placed on the main tasks and services of the CMS Tier-1 at JINR. In February 2015 the JINR CMS Tier-1 resources were increased to the level that was outlined in JINR’s rollout plan: CPU 2400 cores (28800 HEP-Spec06), 2.4 PB disks, and 5.0 PB tapes. The first results of the Tier-1 operations received during the LHC Run-2 start are presented.
Proceedings of The 6th International Workshop on Deep Learning in Computational Physics — PoS(DLCP2022)
The article discusses the methods and algorithms that underlie the analytical platform for automa... more The article discusses the methods and algorithms that underlie the analytical platform for automated monitoring and analysis of the labor market in the Russian Federation, as well as the analysis of the higher education system's compliance with the labor market's current needs. The study involved natural language processing methods and Big Data technologies. The general scheme corresponds to end-to-end processing-from data collection and storage, their transformation, analysis, and modeling, to visualization of results and decision-making. The analytical core of the system is a module for intellectual analysis of the texts of job advertisements in the labor market. The vacancies are collected from the most complete databases in Russia (namely HeadHunter, Work in Russia and SuperJob). Job descriptions of vacancies are matched with the official list of professions of the Ministry of Labor and Social Protection of Russia using semantic analysis based on neural models trained on large arrays of texts. Also, using semantic analysis, automated monitoring and intellectual analysis of the staffing needs of the all-Russian and regional labor markets are carried out according to the range of specialties of the university. Data gathering has been ongoing from 2015 up to now.
Multifunctional Information and Computing Complex (MICC) is one of the basic scientific facilitie... more Multifunctional Information and Computing Complex (MICC) is one of the basic scientific facilities of Joint Institute for Nuclear Research (JINR). It provides a 24×7 implementation of a vast range of competitive research conducted at JINR at a global level. MICC consists of four major components: grid-infrastructure, central computing complex, JINR private cloud and high performance heterogeneous cluster HybriLIT. All major components rely on the network and engineering infrastructure. It is important to monitor all of the components on three levels: hardware level, network level, and service level. Currently there are many monitoring systems applied for MICC. These systems are used by different groups and built on different technologies. All monitoring systems are mostly independent despite the fact that some systems collect the same monitoring data. Their independence makes it difficult to see the overall effectiveness and bottlenecks of MICC because the data are scattered among m...
Last years, the prospects for digital transformation of economic processes were actively discusse... more Last years, the prospects for digital transformation of economic processes were actively discussed. It is quite a complex problem having no solution with traditional methods. Opportunities of the qualitative development of the transformation are illustrated by the example of use of Big Data analytics, particularly text analysis, for the assessment of the needs of regional labour markets in the man-power. The problem is solved using the developed by the authors the automated information system of monitoring of matching the staffing needs of employers with the training level. The system presented use the information gathering from open data sources and provides additional opportunities to identify qualitative and quantitative interrelation between the education and the labour market. The system is targeted at a wide range of users: authorities and management of regions and municipalities; the management of universities, companies, recruitment agencies; graduates and prospective students.
The paper describes the cooperation of JINR specialists and Czech colleagues in the ˇeld of deplo... more The paper describes the cooperation of JINR specialists and Czech colleagues in the ˇeld of deployment and development of modern grid technologies within the framework of the WLCG global infrastructure project. Special attention is paid to the development and usage of monitoring systems.
This is the summary and introduction to the proceedings contributions for the Les Houches 2009 “T... more This is the summary and introduction to the proceedings contributions for the Les Houches 2009 “Tools and Monte Carlo” working group
The author would like to thank the ARDA/ASAP team for a magnificent grid tool they have created, ... more The author would like to thank the ARDA/ASAP team for a magnificent grid tool they have created, and A. Nikitenko and R. Kinnunen for reading the manuscript. 27. DATA-DRIVEN BACKGROUND DETERMINATION IN THE CHANNEL H → W W → lνlν WITH NO HARD JETS 55
9th International Conference "Distributed Computing and Grid Technologies in Science and Education", 2021
Started in natural sciences, the high demand for analyzing a vast amount of complex data reached ... more Started in natural sciences, the high demand for analyzing a vast amount of complex data reached suchresearch areas as economics and social sciences. Big Data methods and technologies provide newefficient tools for research. In this paper, we discuss the main principles and architecture of the digitalanalytical platform aimed to support socio-economic applications. Integrating specific open-sourcesolutions, the platform intended to cover full-cycle data analysis and machine learning experiments,from data gathering to visualization. One of the system's primary goals is to deliver the advantage ofthe cloud and distributed computing and GPU accelerators with Big Data analysis techniques. Theauthors present the approach of building the platform from low-level services such as storage, virtualinfrastructure, pass-through authentication, up to data flows processing, analysis experiments, andresults representation.
This paper provides a brief overview of modern methods and approaches used for automatic processi... more This paper provides a brief overview of modern methods and approaches used for automatic processing of text information. In English-language literature, this area of science is called NLP-Natural Language Processing. The very name suggests that the subject of analysis (and for many tasks – and synthesis) are materials presented in one of the natural languages (and for a number of tasks – in several languages simultaneously), i.e. national languages of communication between people. Programming languages are not included in this group. In Russian-language literature, this area is called Computer (or mathematical) linguistics. NLP (computational linguistics) usually includes speech analysis along with text analysis, but in this review speech analysis does not consider. The review used materials from original works, monographs, and a number of articles published the «Open Systems.DBMS» journal.
2020 International Scientific and Technical Conference Modern Computer Network Technologies (MoNeTeC), 2020
Megascience projects like the Worldwide LHC Computing Grid (WLCG) usually require global, natural... more Megascience projects like the Worldwide LHC Computing Grid (WLCG) usually require global, naturally distributed systems to operate the vast experimental data. For data processing, storage, and analysis, WLCG unites the assets of about 180 data centers in 50 countries, and the total storage volume exceeds 1 Exabyte. Joint Institute for Nuclear Research is profoundly engaged in the integration and development of distributed heterogeneous resources and the development of Big Data technologies to provide the infrastructure and the methods for contemporary scientific megaprojects in such data- and computing-intensive science domains as high energy physics, astrophysics, bioinformatics, and others. One of the network’s critical tasks here is to provide high-speed and reliable access to the distributed resources worldwide and to allow using Big Data analytical platforms to the full. The paper provides a review of the JINR network infrastructure and its main parts - JINR Local Area Network (Backbone), Network of the Multifunctional Information and Computing Complex, and JINR Telecommunication Channels, as a crucial part of JINR Informational and Computer Infrastructure which serves needs of numerous large scale projects and experiments of megascience level.
While scientific data is usually processed with specialized software, more and more institutions ... more While scientific data is usually processed with specialized software, more and more institutions make use of new developments in Big Data for processing all the ancillary data that accompany experimental research and scientific computing. One common use case is monitoring of the experiments and the IT infrastructure, but a well-realized Big Data platform can be used to extract useful knowledge from many available data streams. This work describes our efforts to lay out and implement a general-purpose Big Data processing framework capable of both batch and stream operation for use in our institute. As the core, we chose Apache Spark running on cluster resources controlled by Apache Mesos. To demonstrate the functionality of the platform in both modes, we chose a sample problem of local network packet analysis. TCP and IP header fields were aggregated by the source IP and converted to the AGgregate and Mode (AGM) form. After encoding into numerical features, clusterization and princip...
Inspired by the success of BESDIRAC, the distributed computing environment based on DIRAC for BES... more Inspired by the success of BESDIRAC, the distributed computing environment based on DIRAC for BESIII experiment, several other experiments operated by Institute of High Energy Physics (IHEP), such as Circular Electron Positron Collider (CEPC), Jiangmen Underground Neutrino Observatory (JUNO), Large High Altitude Air Shower Observatory (LHAASO) and Hard X-ray Modulation Telescope (HXMT) etc, are willing to use DIRAC to integrate the geographically distributed computing resources available by their collaborations. In order to minimize manpower and hardware cost, we extended the BESDIRAC platform to support multi-VO scenario, instead of setting up a self-contained distributed computing environment for each VO. This makes DIRAC as a service for the community of those experiments. To support multi-VO, the system architecture of BESDIRAC is adjusted for scalability. The VOMS and DIRAC servers are reconfigured to manage users and groups belong to several VOs. A lightweight storage resource manager StoRM is employed as the central SE to integrate local and grid data. A frontend system is designed for user's massive job splitting, submission and management, with plugins to support new VOs. A monitoring and accounting system is also considered to easy the system administration and VO related resources usage accounting.
Distributed computing is necessary nowadays for high energy physics experiments to organize heter... more Distributed computing is necessary nowadays for high energy physics experiments to organize heterogeneous computing resources all over the world to process enormous amounts of data. The BESIII experiment in China, has established its own distributed computing system, based on DIRAC, as a supplement to local clusters, collecting cluster, grid, desktop and cloud resources from collaborating member institutes around the world. The system consists of workload management and data management to deal with the BESIII Monte Carlo production workflow in a distributed environment. A dataset-based data transfer system has been developed to support data movements among sites. File and metadata management tools and a job submission frontend have been developed to provide a virtual layer for BESIII physicists to use distributed resources. Moreover, the paper shows the experience to cope with lack of grid experience and low manpower among the BESIII community.
9th International Conference "Distributed Computing and Grid Technologies in Science and Education", Dec 13, 2021
The modern Big Data ecosystem provides tools to build a flexible platform for processing data str... more The modern Big Data ecosystem provides tools to build a flexible platform for processing data streams and batch datasets. Supporting both the functioning of modern giant particle physics experiments and the services necessary for the work of many individual physics researchers results in generating and transferring large amounts of semi-structured data. Thus, it is promising to apply cutting-edge technologies to study these data flows and make the services' provisioning more effective. In this work, we describe the structure and implementation of our data analysis platform, built on the Apache Spark cluster. With the official support for GPU computing now available in Spark version 3, we propose a change in the architecture to utilize these more performant resources while keeping the platform's functionality provided by using mainstream Big Data software. Furthermore, the necessity for GPU support entails a change in the computing resource management infrastructure from Apache Mesos to Kubernetes. Finally, to demonstrate the features and operation of the system, we use the task of network packet analysis for security monitoring and anomaly detection in both batch and stream modes.
In this paper we report on LCG Monte-Carlo Data Base (MCDB) and software which has been developed... more In this paper we report on LCG Monte-Carlo Data Base (MCDB) and software which has been developed to operate MCDB. The main purpose of the LCG MCDB project is to provide a storage and documentation system for sophisticated event samples simulated for the LHC collaborations by experts. In many cases, the modern Monte-Carlo simulation of physical processes requires expert knowledge in Monte-Carlo generators or significant amount of CPU time to produce the events. MCDB is a knowledgebase mainly dedicated to accumulate simulated events of this type. The main motivation behind LCG MCDB is to make the sophisticated MC event samples available for various physical groups. All the data from MCDB is accessible in several convenient ways. LCG MCDB is being developed within the CERN LCG Application Area Simulation project.
Kompʹûternye issledovaniâ i modelirovanie, Jun 1, 2015
The Compact Muon Solenoid (CMS) is a high-performance general-purpose detector at the Large Hadro... more The Compact Muon Solenoid (CMS) is a high-performance general-purpose detector at the Large Hadron Collider (LHC) at CERN. A distributed data analysis system for processing and further analysis of CMS experimental data has been developed and this model foresees the obligatory usage of modern grid-technologies. The CMS Computing Model makes use of the hierarchy of computing centers (Tiers). The Joint Institute for Nuclear Research (JINR) takes an active part in the CMS experiment. In order to provide a proper computing infrastructure for the CMS experiment at JINR and for Russian institutes collaborating in CMS, Tier-1 center for the CMS experiment is constructing at JINR. The main tasks and services of the CMS Tier-1 at JINR are described. The status and perspectives of the Tier1 center for the CMS experiment at JINR are presented.
Physics of Particles and Nuclei Letters, Sep 1, 2016
The BES-III experiment at the IHEP CAS, Beijing, is running at the high-luminosity e+e-collider B... more The BES-III experiment at the IHEP CAS, Beijing, is running at the high-luminosity e+e-collider BEPC-II to study physics of charm quarks and tau leptons. The world largest samples of J/psi and psi' events are already collected, a number of unique data samples in the energy range 2.5-4.6 GeV have been taken. The data volume is expected to increase by an order of magnitude in the coming years. This requires to move from a centralized computing system to a distributed computing environment, thus allowing the use of computing resources from remote sites -members of the BES-III Collaboration. In this report the general information, latest results and development plans of the BES-III distributed computing system are presented.
The paper summarizes general information, the latest results and development plans of the BES-III... more The paper summarizes general information, the latest results and development plans of the BES-III distributed computing system. The BES-III experiment at the Institute of High Energy Physics (Beijing, China) is aimed at the precision measurements in e+e-annihilation at the energy range from 2.0 to 4.6 GeV. The world's largest samples of J/psi and psi' events and unique samples of XYZ data have been already collected. To process the BES-III experiment's data the distributed computing infrastructure based on DIRAC interware has been setup and became operational since 2013. The work was done in order to tune and enhance DIRAC itself for BES-III purposes: monitoring system and data transfer system were developed. Experience of the BES-III development team may be very useful for medium scale HEP experiments aiming to apply distributed data processing in their computing model.
Physics of Particles and Nuclei Letters, Sep 1, 2016
An overview of the JINR Tier-1 centre for the CMS experiment at the LHC is given. Special emphasi... more An overview of the JINR Tier-1 centre for the CMS experiment at the LHC is given. Special emphasis is placed on the main tasks and services of the CMS Tier-1 at JINR. In February 2015 the JINR CMS Tier-1 resources were increased to the level that was outlined in JINR’s rollout plan: CPU 2400 cores (28800 HEP-Spec06), 2.4 PB disks, and 5.0 PB tapes. The first results of the Tier-1 operations received during the LHC Run-2 start are presented.
Proceedings of The 6th International Workshop on Deep Learning in Computational Physics — PoS(DLCP2022)
The article discusses the methods and algorithms that underlie the analytical platform for automa... more The article discusses the methods and algorithms that underlie the analytical platform for automated monitoring and analysis of the labor market in the Russian Federation, as well as the analysis of the higher education system's compliance with the labor market's current needs. The study involved natural language processing methods and Big Data technologies. The general scheme corresponds to end-to-end processing-from data collection and storage, their transformation, analysis, and modeling, to visualization of results and decision-making. The analytical core of the system is a module for intellectual analysis of the texts of job advertisements in the labor market. The vacancies are collected from the most complete databases in Russia (namely HeadHunter, Work in Russia and SuperJob). Job descriptions of vacancies are matched with the official list of professions of the Ministry of Labor and Social Protection of Russia using semantic analysis based on neural models trained on large arrays of texts. Also, using semantic analysis, automated monitoring and intellectual analysis of the staffing needs of the all-Russian and regional labor markets are carried out according to the range of specialties of the university. Data gathering has been ongoing from 2015 up to now.
Multifunctional Information and Computing Complex (MICC) is one of the basic scientific facilitie... more Multifunctional Information and Computing Complex (MICC) is one of the basic scientific facilities of Joint Institute for Nuclear Research (JINR). It provides a 24×7 implementation of a vast range of competitive research conducted at JINR at a global level. MICC consists of four major components: grid-infrastructure, central computing complex, JINR private cloud and high performance heterogeneous cluster HybriLIT. All major components rely on the network and engineering infrastructure. It is important to monitor all of the components on three levels: hardware level, network level, and service level. Currently there are many monitoring systems applied for MICC. These systems are used by different groups and built on different technologies. All monitoring systems are mostly independent despite the fact that some systems collect the same monitoring data. Their independence makes it difficult to see the overall effectiveness and bottlenecks of MICC because the data are scattered among m...
Last years, the prospects for digital transformation of economic processes were actively discusse... more Last years, the prospects for digital transformation of economic processes were actively discussed. It is quite a complex problem having no solution with traditional methods. Opportunities of the qualitative development of the transformation are illustrated by the example of use of Big Data analytics, particularly text analysis, for the assessment of the needs of regional labour markets in the man-power. The problem is solved using the developed by the authors the automated information system of monitoring of matching the staffing needs of employers with the training level. The system presented use the information gathering from open data sources and provides additional opportunities to identify qualitative and quantitative interrelation between the education and the labour market. The system is targeted at a wide range of users: authorities and management of regions and municipalities; the management of universities, companies, recruitment agencies; graduates and prospective students.
The paper describes the cooperation of JINR specialists and Czech colleagues in the ˇeld of deplo... more The paper describes the cooperation of JINR specialists and Czech colleagues in the ˇeld of deployment and development of modern grid technologies within the framework of the WLCG global infrastructure project. Special attention is paid to the development and usage of monitoring systems.
This is the summary and introduction to the proceedings contributions for the Les Houches 2009 “T... more This is the summary and introduction to the proceedings contributions for the Les Houches 2009 “Tools and Monte Carlo” working group
The author would like to thank the ARDA/ASAP team for a magnificent grid tool they have created, ... more The author would like to thank the ARDA/ASAP team for a magnificent grid tool they have created, and A. Nikitenko and R. Kinnunen for reading the manuscript. 27. DATA-DRIVEN BACKGROUND DETERMINATION IN THE CHANNEL H → W W → lνlν WITH NO HARD JETS 55
9th International Conference "Distributed Computing and Grid Technologies in Science and Education", 2021
Started in natural sciences, the high demand for analyzing a vast amount of complex data reached ... more Started in natural sciences, the high demand for analyzing a vast amount of complex data reached suchresearch areas as economics and social sciences. Big Data methods and technologies provide newefficient tools for research. In this paper, we discuss the main principles and architecture of the digitalanalytical platform aimed to support socio-economic applications. Integrating specific open-sourcesolutions, the platform intended to cover full-cycle data analysis and machine learning experiments,from data gathering to visualization. One of the system's primary goals is to deliver the advantage ofthe cloud and distributed computing and GPU accelerators with Big Data analysis techniques. Theauthors present the approach of building the platform from low-level services such as storage, virtualinfrastructure, pass-through authentication, up to data flows processing, analysis experiments, andresults representation.
This paper provides a brief overview of modern methods and approaches used for automatic processi... more This paper provides a brief overview of modern methods and approaches used for automatic processing of text information. In English-language literature, this area of science is called NLP-Natural Language Processing. The very name suggests that the subject of analysis (and for many tasks – and synthesis) are materials presented in one of the natural languages (and for a number of tasks – in several languages simultaneously), i.e. national languages of communication between people. Programming languages are not included in this group. In Russian-language literature, this area is called Computer (or mathematical) linguistics. NLP (computational linguistics) usually includes speech analysis along with text analysis, but in this review speech analysis does not consider. The review used materials from original works, monographs, and a number of articles published the «Open Systems.DBMS» journal.
2020 International Scientific and Technical Conference Modern Computer Network Technologies (MoNeTeC), 2020
Megascience projects like the Worldwide LHC Computing Grid (WLCG) usually require global, natural... more Megascience projects like the Worldwide LHC Computing Grid (WLCG) usually require global, naturally distributed systems to operate the vast experimental data. For data processing, storage, and analysis, WLCG unites the assets of about 180 data centers in 50 countries, and the total storage volume exceeds 1 Exabyte. Joint Institute for Nuclear Research is profoundly engaged in the integration and development of distributed heterogeneous resources and the development of Big Data technologies to provide the infrastructure and the methods for contemporary scientific megaprojects in such data- and computing-intensive science domains as high energy physics, astrophysics, bioinformatics, and others. One of the network’s critical tasks here is to provide high-speed and reliable access to the distributed resources worldwide and to allow using Big Data analytical platforms to the full. The paper provides a review of the JINR network infrastructure and its main parts - JINR Local Area Network (Backbone), Network of the Multifunctional Information and Computing Complex, and JINR Telecommunication Channels, as a crucial part of JINR Informational and Computer Infrastructure which serves needs of numerous large scale projects and experiments of megascience level.
While scientific data is usually processed with specialized software, more and more institutions ... more While scientific data is usually processed with specialized software, more and more institutions make use of new developments in Big Data for processing all the ancillary data that accompany experimental research and scientific computing. One common use case is monitoring of the experiments and the IT infrastructure, but a well-realized Big Data platform can be used to extract useful knowledge from many available data streams. This work describes our efforts to lay out and implement a general-purpose Big Data processing framework capable of both batch and stream operation for use in our institute. As the core, we chose Apache Spark running on cluster resources controlled by Apache Mesos. To demonstrate the functionality of the platform in both modes, we chose a sample problem of local network packet analysis. TCP and IP header fields were aggregated by the source IP and converted to the AGgregate and Mode (AGM) form. After encoding into numerical features, clusterization and princip...
Uploads
Papers by Sergey Belov