The Open Grid Services Architecture (OGSA)was developed in the Open Grid Forum. The Open Grid Forum (OGF) is a community of users, developers, and vendors leading the global standardization effort for grid computing. The OGF community consists of thousands of individuals in industry and research, representing over 400 organizations in more than 50 countries. Together we work to accelerate adoption of grid computing worldwide because we believe grids will lead to new discoveries, new opportunities, and better business practices.

The work of OGF is carried out through community-initiated working groups, which develop standards and specifications in cooperation with other leading standards organizations, software vendors, and users. OGF is funded through its Organizational Members, including technology companies and academic and government research institutions. OGF hosts several events each year to further develop grid-related specifications and use cases and to share best practices.

As described in the IEEE Computer article, "An OGSA Primer", there are a number of different specifications and profiles that come together to realize the many different grid use cases. Below we provide a short description of each, as well as a link to the specification. OGF documents have a document number, e.g., GFD.80 means grid forum document 80.

The documents are organized below as follows: overview material, infrastructure specifications, execution management, data, security, examples of the standards in use, and links to projects that have implemented the specifications.

Overview material

OGSA is a classic middleware architecture designed to fit within a traditional three layer distributed systems model with an access layer, a services layer, and a resources layer.

The access layer provides a means for users to interact with the grid, whether it is via a special-purpose-built application, a Web portal or gateway, a set of APIs, or via integration into the local host operating system.

The services layer provides a standard set of interfaces that implementers of the access layer can depend on, and realize what Notkin termed a “layer of virtual homogeneity�. This is the heart of OGSA – and indeed OGF, specifications and profiles.

The resources or provisioning layer provides the physical or virtual resources that are being manipulated, whether they are computing systems, flat files, relational databases, or virtual organizations. Different vendors or implementers may provide different implementations of the grid services that map to their particular resources.



GFD.80 The Open Grid Services Architecture, Version 1.5 Obsoletes GFD.30 INFO I. Foster, H. Kishimoto, A. Savva, D. Berry, A. Grimshaw, B. Horn, F. Maciel, F. Siebenlist, R. Subramaniam, J. Treadwell, J. Von Reich 2006-09-05

GFD.120 Open Grid Services Architecture® Glossary of Terms Version 1.6 Obsoletes GFD.81 INFO J. Treadwell 2007-12-12.

GFD.141 Independent Software Vendors (ISV) Remote Computing Usage Primer INFO S. Newhouse, A. Grimshaw 2008-10-07. This document describes in detail how OGF specifications can be used to implement high throughput computing use cases. The target auidience is independent software vendors who want to exploit grid resources in their products.

Infrastructure

Infrastructure services are a set of common functionalities required by higher-level services. As OGSA builds on Web services technologies, the Web Services Description Language defines service interfaces. Infrastructure includes standards such as Web Services Resource Framework (WS-RF), WS-Management, and WS-Addressing.

Infrastructure services are concerned with resource naming, communication, and reflection. In OGSA, a resource is a physical or logical entity that supports the use or operation of a computing application or environment. Resources are often stateful and provide a capability or capacity such as servers, networks, memory, applications, and databases. Resources might also handle dynamic entities, such as processes, print jobs, database query results, and virtual organizations.

Naming. Suppose there are two resources, A and B, and that A wishes to interact with B. How does A refer to B? OGSA defines a multilayer naming scheme of addresses, identities, and pathnames.

OGSA uses the WS-Addressing specification for addressing. This specification defines an XML data structure called the end point reference (EPR) that encapsulates the information the client needs to message a service. The EPR includes a network protocol and address, an extensible metadata section to convey information such as security policy, and an opaque section for session/resource identifiers.

The Open Grid Forum’s (OGF’s) WS-Naming Specification specification profiles WS-Addressing to provide identities and name rebinding. An optional EPR metadata element called end point identifier (EPI) is a URI that is unique in space and time. Clients can compare the EPIs contained in two or more EPRs. If the EPIs are the same, the EPRs are said to point to the same entity. The semantics of the underlying service determine sameness. If the EPIs are different, nothing can be inferred.

Name rebinding is the ability to get a new EPR if communication with a web service pointed to by the old EPR fails, for example if the host the web service was running on has failed or become disconnected from the network. The rebinding aspects of WS-Naming EPRs facilitate the implementation of the traditional distributed system transparencies: migration, failure, replication, and so forth. Embedded in an EPR’s metadata element is an optional resolver EPR. A client can call the resolver EPR to acquire a new EPR for the service. For example, suppose there is a client C and service EPR S that contains a resolver EPR R. Suppose S fails or migrates. Subsequent invocations of S by C will fault because S has moved or failed. C can then invoke the resolution function to acquire a new EPR for S, S', that is, S' = S.R.resolve(). This capability is critical for reaching our reliability, availability, and performance objectives.

Reflection and metadata. Reflection here refers to the ability to discover properties or attributes of grid resources or services, for example, the port-types, security mechanisms, and the provenance of data. Examples in use include WSRF-RP 88 and WS-Metadata Exchange. The OGSA WS-RF Base Profile addresses selected WSRF-RP specifications including the operation getResourceProperties, which returns an XML document with the metadata associated with a resource.

GFD.109 WS-Naming Specification P-REC A. Grimshaw, D. Snelling 2007-07-03

GFD.101 Resource Namespace Service Specification P-REC M. Pereira, O. Tatebe, L. Luan, T. Anderson 2007-05-03

WS-RF

WS-Notification

WS-Eventing

Execution Management

Execution management is concerned with describing, starting, and managing the execution of jobs. This is one of the most active areas of OGF work, and OGSA work in particular.

GFD.136 Job Submission Description Language (JSDL) Specification v1.0 P-REC A. Anjomshoaa, F. Brisard, M. Drescher, D. Fellows, A. Ly, S. McGough, D. Pulsipher, A. Savva 2008-07-28 JSDL documents are XML documents that describe a job: the resource requirements such as memory, number and type of CPUs, supported libraries, etc.; the input and output files, where they can be found, file access protocols to be used when staging data in and out; and the parameters to be passed to the application. If the application is not installed for a particular execution environment it must first be installed. Often this is accomplished by staging-in the application as well as the input data files. JSDL files are given to execution services to execute the described job.

GFD.108 OGSA® Basic Execution Service Version 1.0 REC I. Foster, A. Grimshaw, P. Lane, W. Lee, M. Morgan, S. Newhouse, S. Pickles, D. Pulsipher, C. Smith, M. Theimer 2007-08- OGSA-BES is a simple interface for creating new jobs, monitoring them, and managing their lifetime in addition to providing information useful for making scheduling decisions. On top of the JSDL and OGSA-BES specifications the HPC Profile Group inside the OGF has defined a number of specifications and profiles on existing specifications which further aide in interoperability.

D.115 JSDL SPMD Application Extension, Version 1.0 P-REC A. Savva 2007-08-28

GFD.114 HPC Basic Profile, Version 1.0 REC B. Dillaway, M. Humphrey, C. Smith, M. Theimer, G. Wasson 2007-08-28 HPC-BP defines a simplified Application element which can be used inside of JSDL documents to more easily annotate how a sequential application should be run (what the executable is called, what the arguments are for the command line, etc.).

GFD.124 Interoperability Experiences with the High Performance Computing Basic Profile (HPCBP), Version 1.0 EXP G. Wasson 2008-02-21

GFD.135 HPC File Staging Profile, Version 1.0 P-REC G. Wasson, M. Humphrey 2008-06-28

GFD.98 Usage Record - Format Recommendation P-REC R. Mach, R. Lepro-Metz, S. Jackson, L. McGinnis 2007-02-22

Data Management

GFD.121 OGSA® Data Architecture INFO D. Berry, A. Luniewski, M. Antonioletti 2007-12-05

GFD.88 ByteIO OGSA® WSRF Basic Profile Rendering 1.0 P-REC M. Morgan 2007-01-12 GFD.87 ByteIO Specification 1.0 P-REC M. Morgan 2007-01-12

GFD.77 Interoperability Testing for DAIS Working Group Specifications INFO S. Lynden, N. Paton, D. Pearson 2006-09-05

GFD.76 Web Services Data Access and Integration - The Relational Realisation (WS-DAIR) Specification, Version 1.0 P-REC M. Antonioletti, B. Collins, A. Krause, S. Laws, J. Magowan, S. Malaika, N. Paton 2006-09-05

GFD.75 Web Services Data Access and Integration - The XML Realization (WS-DAIX) Specification, Version 1.0 P-REC M. Antonioletti, S. Hastings, A. Krause, S. Langella, S. Lynden, S. Laws, S. Malaika, N. Paton 2006-09-05

GFD.74 Web Services Data Access and Integration - The Core (WS-DAI) Specification, Version 1.0 P-REC M. Antonioletti, M. Atkinson, A. Krause, S. Laws, S. Malaika, N. Paton, D. Pearson, G. Riccardi 2006-09-05

Security

The security of grid infrastructure rests on its ability to protect its assets (information, data, services, etc) from various sources of misuse. Following the Open Systems Interconnect threat model, the types of security threat to which networked resources are vulnerable include disclosure or theft of resources, modification (including destruction) of resources, and resource service interruption. The purpose of a grid security architecture is to define the security services and related mechanisms that can be appropriately applied to mitigate such threats. The goal of these services and mechanisms is to make the costs of unauthorized behavior greater than the potential value of doing so. Without a strong commitment to meaningful security, many potential adopters would be unable to participate because of undue risk and/or legal restrictions.

Grid security architecture is complicated by the reality that participants from different administrative domains will “come to the table� with distinct trust and security infrastructures. This presents challenges for interoperability, even if all resource interfaces and security mechanisms are well-specified by de facto community standards. For example, although the service implementations of a particular application (e.g., a job execution service) may conform to the same interface, each provider is free to choose the security mechanisms that satisfy its particular security requirements. The orthogonality between service interface and security has two important implications: security tokens from one domain may not carry any syntactic or semantic meaning within another, and different peers may require different compositions of secure communication mechanisms (e.g., specific encryption or signature actions).

With site autonomy in mind, OGSA-basedarchitectures are designed with a flexible security architecture in order to support the integration of existing trust infrastructures rather than force users to adopt a new, singular security model. Genesis II provides services and mechanisms to federate and broker existing identity, to provide suitable forms of new identity as necessary, and to configure and discover the secure communication requirements of communicating peers. This is fundamental to realizing the vision of resource sharing in an environment lacking a single source of authority.

WS I Basic Security Profile

WS-Trust

WS-Security

WS-SecurityPolicy

WS-SecureConversation

GFD.74 WS-Secure Addressing Profile

Security Assertion Markup Language (SAML)

eXtensible Access Control Markup Language (XACML)

Examples

The OGSA service specifications and profiles by themselves are of relatively little value to end-users. It is how they are combined and presented to the user that makes them really useful. One example of how the specifications can be combined to realize high throughput computing use cases is described in the ISV Usage Primer. Another example is in the Cross Campus Grid (XCG) homed at the University of Virginia that uses Genesis II as its base middleware layer.

XCG using Genesis II

In the XCG all resources are visible via a single logical namespace. The namespace is implemented using RNS (Resource Namespace Service) resources that map path strings, e.g., “/GENEBANK/databases/bio/sequences/pir21.sqf� to WS-Addressing EndPoint References (EPRs)). An EPR is essentially a Web Services pointer and plays the role of an inode in the directory system. To access a resource anywhere in the XCG all that is needed is the pathname and XCG does the rest.

Many different types of resources can be pointed to by EPRs and thus by pathnames. Examples include files and directories. Less obvious examples include relational databases, execution services, secure token services (identities and groups). For example, an execution service that implements OGSA-BES can also implement the RNS directory interface. With this implementation all that is necessary to execute a job on that service is to copy the JSDL file into the directory representing the execution service. The job then appears in the directory as a file, and job status can be read directly from the file without special tooling.

The Genesis II access layer partitions into two broad categories. The first, user access, deals with how the user interacts with the grid in order to retrieve and manipulate data and compute resources. This category of access constitutes the largest majority of user interaction and generally represents a user’s day-to-day activities. The second category, provisioning, includes the less frequent but equally important aspect of provisioning data or compute information into the grid. This category constitutes the activities that a user sometimes has to perform, but typically not on a daily basis. In both cases all access is accomplished and mediated using OGF specifications.

Accessing Data and Compute Resources in the Grid – The File System Way

The file system paradigm of data access is one of the most ubiquitous in human-computer interaction. Whether you are browsing through folders and double-clicking on files, or cd’ing through directories and cat’ing files, the notion of a hierarchical arrangement of files and directories has become second nature to virtually every computer user in the world. Genesis II exploits this by providing file system drivers for each of its supported platforms. In Linux this takes the form of a FUSE file system driver, and in Windows, an Installable File System (or IFS) driver.

Genesis II grid resources implement one or both of the RNS or ByteIO port-types. These port-types, modeled on traditional POSIX-like APIs, describe respectively a directory-like interface for accessing name-endpoint mappings and a file-like interface for reading and writing data, thus permitting Genesis II to treat each resource as if it were a directory or a file. This makes it relatively easy to implement file system drivers which interact with these resources. Once mounted, access to Genesis II is no different then access to any other file or directory on your local machine.

This access extends beyond simple file data. Because every resource in Genesis II implements a file or directory interface, even resource types that do not represent pure data or directory concepts are mapped as files and directories. This notion of treating resources as files and directories first appeared in Plan 9 and today is in use in the /proc file system. Treating grid resources not usually thought of as files or directories extends the file system based access layer into the compute realm, allowing users to manage compute resources like jobs, hosts, and queues using familiar filesystem abstractions.

Genesis II includes two user level file system drivers (an IFS for Windows and FUSE for Linux) that allow a grid user to map the grid environment into the native OS. Once done, grid access is nearly indistinguishable from local access. The two figures below show respectively Windows and Linux sessions where a user is browsing and manipulating a grid environment. It is easy to see that the sessions are no different than they would be if the user was working in a local file system. In fact, in the Linux window, you can see how native binary applications (like sum) which normally would not be able to use the grid at all without source code modification, can now use grid resources straight out of the box.

Linux FUSE access to Grid resources. The legacy program "sum" can access Grid data resources directly without modification.



A view of the "G" for Grid drive using the Windows IFS driver for Grid.

Another important outcome of treating grid resources as if they were files and directories is that complicated grid operations that normally would take obscure command line incantations can now be performed with simple, easy-to-understand file system access patterns. By simply copying a job description file into a grid compute resource, a user indicates that he or she wishes to submit that job to the grid. Figure 3 shows an example of this where a user is in the process of dragging-and-dropping a JSDL document into a Genesis II queue resource via the Windows IFS driver. Once copied, the queue begins managing the job described in that document without the user issuing any grid specific commands.



Using Native Filesystem Tools to Launch Grid Jobs. The JSDL file gnomad.xml is being dragged and dropped onto the windows-R-queue. The queue takes over, finds a resource to execute the job, stages data as necessary, starts and monitors the job, restarts if necessary.

An important side benefit to this approach is that it makes it very easy for users and system administrators alike to produce programs and scripts to automatically submit jobs to grid resources. Consider the typical HPC use case where a user wants to run a sequential application many times over against a set of input data. A simple script can easily produce the job description files for this batch of jobs and can then submit those jobs via a standard OS file copy operation, thereby fully integrating the grid experience with the traditional user’s local computing experience.

Provisioning Data and Compute Resources

Genesis II users provision data into the grid using one of several techniques. The ability to treat the grid as a file system (as discussed above) implies an obvious and simple way to provision data into the grid – namely to simply copy the data in. However, while this mechanism is sufficient for small numbers of small files, it does not scale well. To address this shortcoming, Genesis II provides an alternative mechanism for provisioning data into the grid called Export Dir. Export Dir is a set of service interfaces that implement the RNS and the ByteIO port-types which redirect all web service based operations into the host machine’s local file system. A user exports a local directory structure into the grid and from that moment forward, that directory structure appears in the grid as on disk. All access to files and directories from the grid are realized in the local file system. Further, since all access to the data is resolved locally, the grid views and the local views of the exported directory are always in sync with one another. A similar approach has been prototyped with relational databases and the WS-DAIR specification.

Provisioning compute resources is accomplished by mapping OGSA-BES resources into the global namespace. A variety of execution resources have been included this way, from stand-alone hosts to small queue-controlled clusters to large queue controlled systems such as the machines at the national centers. ExportDir. While the Genesis II RNS and ByteIO service implementations provide the basic versions of their respective specifications, the specifications are combined together to create a new service, ExportDir, that offers a more advanced data-sharing technology to users. Using ExportDir, users can share directory trees from local Windows or Linux file system with other grid users. Subject to access control, other grid users can read, write, create, and destroy files and directories in the exported directory structure. When a user selects a directory structure on their local machine to share into the grid, the ExportDir service “wraps� that directory and each subdirectory inside with an RNS interface, and each contained file with a ByteIO interface. All operations that take place through the grid interface are reflected into the actual file system that the user shared out, and all changes made to the local file system on that machine are in turn reflected into the grid.



Export Dir Tools. Mapping the local directory /home/morgan/data into the grid.

Replicated ExportDir. Genesis II also includes a replicated version of ExportDir called G-REX. The G-REX service increases the availability of shared data in the face of failure by transparently replicating exported directory structures. This replication scheme does not disrupt compliance with the RNS and ByteIO as it is based on a complimentary grid specification, WS-Naming. WS-Naming extends the basic Web Services addressing model with identities and a re-binding mechanism. Under this specification, EPRs additionally contain an endpoint identifier element which serves as a globally unique (both in space and time) abstract name for that resource and a list of zero or more resolver endpoints. The endpoint identifier element gives clients a way of identifying and comparing addressing endpoints (such as replicas) without requiring them to communicate with those endpoints while the resolver list indicates a number of services that clients can use to bind those endpoint identifiers to new WS-Addressing EPRs (such as from a failing service endpoint to its replica). In addition to increasing the availability of shared data, this re-binding scheme can be used to enhance the performance of data access. By knowledgeably choosing which replicas provide access to shared data, we can reduce access time. In this way, we can utilize replication with WS-Naming to further enrich the data sharing experience in the Genesis II grid platform while maintaining standards compliance with RNS and ByteIO.

Projects:

Genesis II is an open source implementation of a number of OGSA and related specifications, including JDSL, OGSA-BES, HPC Basic Profile, HPC BP File Staging Extensions, RNS, OGSA-ByteIO, OGSA WR-BSP, WS-Naming, and WS-SecureAddressing.

BES++ - A collaborative open source implementation of BES and JSDL.

Windows HPC Server 2008 implements the HPC Basic Profile, JSDL, and OGSA-BES.