Integrating the Palantir Grid Meta-Information System with GRMS
Ivan Rodero
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
Grids allow large scale resource-sharing across different administrative domains. Those diverse resources are likely to join or quit the Grid at any moment or possibly to break down. In such environments there is a wide rage of types of resources to be monitored, with different nature, characteristics and so on. These issues make the task of monitoring complex to treat, and it is difficult to provide a generals ways for accessing to all this information. In this paper we describe the Palantir meta-information system that has been designed for uniforming the access to different monitoring and information systems and abstracting the complexity of the underlying systems. We present the API that Palantir provides to the end users, a general architectural description of its components, and the description of how GRMS providing job monitoring information can be integrated as a part of this meta-Information System.
Related papers
A Grid job monitoring system
Journal of Physics: Conference Series, 2010
This paper presents a web-based Job Monitoring framework for individual Grid sites that allows users to follow in detail their jobs in quasi-real time. The framework consists of several independent components : (a) a set of sensors that run on the site CE and worker nodes and update a database, (b) a simple yet extensible web services framework and (c) an Ajax powered web interface having a look-and-feel and control similar to a desktop application. The monitoring framework supports LSF, Condor and PBS-like batch systems. This is one of the first monitoring systems where an X.509 authenticated web interface can be seamlessly accessed by both end-users and site administrators. While a site administrator has access to all the possible information, a user can only view the jobs for the Virtual Organizations (VO) he/she is a part of. The monitoring framework design supports several possible deployment scenarios. For a site running a supported batch system, the system may be deployed as a whole, or existing site sensors can be adapted and reused with the web services components. A site may even prefer to build the web server independently and choose to use only the Ajax powered web interface. Finally, the system is being used to monitor a glideinWMS instance. This broadens the scope significantly, allowing it to monitor jobs over multiple sites.
Information and monitoring services within a grid environment
The R-GMA (Relational Grid Monitoring Architecture) was developed within the EU DataGrid project, to bring the power of SQL to an information and monitoringsystem for the grid. It provides producer and consumerservices to both publish and retrieve information fromanywhere within a grid environment. Users within a Virtual Organization may define their own tables dynamically into which to publish data.Within the DataGrid project R-GMA was used for theinformation system, making details about grid resources available for use by other middleware components. R-GMA has also been used for monitoring grid jobs by members of the CMS and D0 collaborations where information about jobs is published from within a jobwrapper, transported across the grid by R-GMA and made available to users. An accounting package for processing PBS logging data and sending it to one or more GridOperation Centres using R-GMA has been written and isbeing deployed within LCG. There are many otherexisting and potential applications.R-GMA is currently being re-engineered to fit into a Web Service environment as part of the EU Enabling Grids for E-science in Europe (EGEE) project. Improvements being developed include fine grainedauthorization, an improved user interface and measures to ensure superior scaling behaviour.
Production services for information and monitoring in the grid
UK e-Science All Hands meeting
R-GMA is a realization of the Grid Monitoring Architecture (GMA) that also exploitsthe power of the relational data model and the SQL query language. The biggest challenge during the development of R-GMA was to ensure that it could be scaled to operate in a large grid reliably. The system is being used in areas as diverse as resource discovery, job logging and bookkeeping, network monitoring and accounting. A version of R-GMA is being developed within the follow-on European project EGEE. Work continues within GGF to define information services for OGSA on the basis of experience with R-GMA.
Job monitoring in an interactive grid analysis environment
2004
The grid is emerging as a great computational resource but its dynamic behavior makes the Grid environment unpredictable. Systems and networks can fail, and the introduction of more users can result in resource starvation. Once a job has been submitted for execution on the grid, monitoring becomes essential for a user to see that the job is completed in an efficient way, and to detect any problems that occur while the job is running. In current environments once a user submits a job he loses direct control over the job and the system behaves like a batch system: the user submits the job and later gets a result back. The o nly information a user can obtain about a job is whether it is scheduled, running, cancelled or finished. Today users are becoming increasingly interested in such analysis g rid environments in which they can check the progress of the job, obtain intermediate results, terminate the job based on the progress of job or intermediate results, steer the job to other nodes to achieve better performance and check the resources consumed by the job. In order to fulfill their requirements of interactivity a mechanism is needed that can provide the user with real time access to information about different attributes of a job. In this paper we present the design of a Job Monitoring Service, a web service that will provide interactive remote job monitoring by allowing users to access different attributes of a job once it has been submitted to the interactive Grid Analysis Environment [1].
Relational Grid Monitoring Architecture (R-GMA)
Computing Research Repository, 2003
We describe R-GMA (Relational Grid Monitoring Architecture) which has been developed within the European DataGrid Project as a Grid Information and Monitoring System. Is is based on the GMA from GGF, which is a simple Consumer-Producer model. The special strength of this implementation comes from the power of the relational model. We offer a global view of the information as if each Virtual Organisation had one large relational database. We provide a number of different Producer types with different characteristics; for example some support streaming of information. We also provide combined Consumer/Producers, which are able to combine information and republish it. At the heart of the system is the mediator, which for any query is able to find and connect to the best Producers for the job. We have developed components to allow a measure of inter-working between MDS and R-GMA. We have used it both for information about the grid (primarily to find out about what services are available at any one time) and for application monitoring. R-GMA has been deployed in various testbeds; we describe some preliminary results and experiences of this deployment.
Related topics
Related papers
Monitoring Grid Resources: JMX in Action
2004
Abstract: This paper summarizes research on monitoring GRID resources, which resulted in theimplementation of the JIMS system. It contains an overview of the most important architectural andsoftware concepts that make the constructed system flexible and user-friendly. The paper evaluatesJMX and Web Service technologies as foundations for implementing monitoring systems. Particularattention has been paid to system adaptability, autoconfiguration and interoperability.Keywords: JMX, Grid, Monitoring, Distributed System, SOA, Web Services 1. Introduction Monitoring distributed computer system resources is an integral part of anymanagement activity. The grid computing concept [1] addresses issues related to ac-cessibility and transparent sharing of distributed computational, storage and com-munication resources among groups of users, putting the management aspects at theforefront of grid research. Recently, the grid research community has been inspiredby a new approach based on SOA (Servic...
The Relational Grid Monitoring Architecture: Mediating Information about the Grid
Journal of Grid Computing, 2004
We have developed and implemented the Relational Grid Monitoring Architecture (R-GMA) as part of the DataGrid project, to provide a flexible information and monitoring service for use by other middleware components and applications.
R-GMA: An Information Integration System for Grid Monitoring
2003
Computational Grids are distributed systems that provide access to computational resources in a transparent fashion. Collecting and providing information about the status of the Grid itself is called Grid monitoring. We describe R-GMA (Relational Grid Monitoring Architecture) as a solution to the Grid monitoring problem. It uses a local as view approach to information integration and will be a component of the European Union’s DataGrid. The R-GMA architecture and mechanisms are general and could be used in other areas where there is a need for publishing and querying information in a distributed fashion.
GRID2003 Monitoring, Metrics, and Grid Cataloging System
2004
Grid computing involves the close coordination of many different sites which offer distinct computational and storage resources to the Grid user community. The resources at each site need to be monitored continuously. Static and dynamic site information needs to be presented to the user community in a simple and efficient manner. This paper will present both the design and implementation of the Grid3 monitoring infrastructure and the design details and the functionalities of a new application called the GridCat. The Grid3 monitoring architecture follows a useroriented design that specifies standard metrics and utilizes underlying monitoring tools to collect them into a diversified framework. Then existing tools can be integrated, their functionality extended and new tools developed. The primary tools used include the ACDC Job Monitoring system from University at Buffalo, Ganglia, a preliminary version of GridCat, Globus MDS, the University of Chicago Grid telemetry MDViewer, and US ...
Architecture for Highly Configurable Dashboards for Operations Monitoring and Support
Studies in Informatics and Control, 2018
Nowadays, information systems make use of a series of different components and platforms. Being able to monitor the health status of such components and platforms, along with the overall health status of the operation supported by such information systems is of paramount importance. The immediate solution for these monitoring needs is to use vendor specific applications, which results in having to look and understand several different interfaces. As it will be explained, there are available solutions that integrate such needs on a single platform. However, these solutions failed to provide a monitoring dashboard for the entire system operation, with highly customizable dashboard interface, easily defined metrics and easy mobile application availability. This paper describes an architecture for configurable dashboards capable of presenting heterogeneous metrics, side by side, regardless of their origin. A prototype, based on such architecture, is presented as a proof of concept. This prototype is being used to monitor the Cape Verde's Justice Information System. The result, based on the proposed architecture and prototype, is a custom tool with a single configuration file that can be adapted to different thresholds, metrics and monitoring scenarios.
The CanonicalProducer: An instrument monitoring component of the Relational Grid Monitoring Architecture (R-GMA
Scientific Programming, 2005
We describe how the R-GMA (Relational Grid Monitoring Architecture) can be used to allow for instrument monitoring in a Grid environment. The R-GMA has been developed within the European DataGrid Project (EDG) as a Grid Information and Monitoring System. It is based on the Grid Monitoring Architecture (GMA) from the Global Grid Forum (GGF), which is a simple Consumer-Producer model. The special strength of this implementation comes from the power of the relational model. It offers a global view of the information as if each Virtual Organisation had one large relational database. It provides a number of different Producer types with different characteristics; for example some support streaming of information. We describe the R-GMA component that allows for instrument monitoring, the CanonicalProducer. We also describe an example use of this approach in the European CrossGrid project, SANTA-G, a network monitoring tool.
GridICE: a monitoring service for Grid systems
Future Generation Computer Systems, 2005
Grid systems follow a new paradigm of distributed computing that enables the coordination of resources and services that are not subject to centralized control, can dynamically join and leave virtual pools, and are assigned to users by means of an explicit assignment functionality. The monitoring of a Grid is a multi-institutional and Virtual Organization (VO)-oriented service. It must deal with the dynamics, diversity, and geographical distribution of the resources available to Virtual Organizations, and the various levels of abstraction for modeling them. This paper presents the requirements, architecture and implementation of GridICE, a monitoring service for Grid systems. The suitability of this tool in real-life scenarios is analyzed and discussed.
R-GMA: A Relational Grid Information and Monitoring System
We describe R-GMA (Relational Grid Monitoring Architecture) which is being developed within the European DataGrid Project as a Grid Information and Monitoring System for both the grid itself and for use by applications. Is is based on the GMA from GGF, which is a simple Consumer-Producer model. The special strength of this implementation comes from the power of the relational model. We offer a global view of the information as if each VO had one large relational database. We provide a number of different Producer types with different characteristics; for example some support streaming of information. We also provide combined Consumer/Producers, which are able to combine information and republish it. At the heart of the system is the mediator, which for any query is able to find and connect to the best Producers to do the job. In addition to having some of our own sensors able to publish information, we are able to invoke MDS info-provider scripts and publish the resulting information via R-GMA.
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.