Tags:
create new tag
view all tags

New EMI Requirements and Work Plans

Taken from an email from Andres Aeschlimann (to operations@swing-grid.ch, Feb 7th 14:51), for the EMI Roadmap there is https://wiki.egi.eu/wiki/EGI_Operations#EMI

Here is the EMI 1 release plan: https://twiki.cern.ch/twiki/bin/view/EMI/InternalDeliverableEmi1ReleaseDevPlans

Release schedule of different components: https://savannah.cern.ch/task/?group=emi-rel-sched&func=browse&set=open&msort=0&advsrch=0&morder=bug_id%3C&order=bug_id#results

EMI Services documentation: https://twiki.cern.ch/twiki/bin/view/EMI/EMIServicesDocs

Compute Area Work Plan

ARC, gLite CREAM (with CEMon and BLAH) and UNICORE provide the same core functionality; they are implemented in different and often incompatible ways. They want to address some of these issues. In particular the plan is:

  • To define and adopt job execution standard/agreed interfaces, with all three representatives, work in progress.
  • To adopt information service related standards. GLUE v. 2.0.
  • To harmonize the used authorization mechanisms. gLite Argus service.
  • To harmonize the parallel and MPI job support. Provision of a standard EMI unified layer
  • To consolidate and harmonize the compute area clients and APIs. The main objectives are to improve: usability, maintainability and portability to different architectures and operating systems.
  • To assess the use of a messaging system between the EMI job management components. It's not clear if it's needed, but it's under investigation (maybe messages between WMS and Cream?)
  • To remove GSI, and replace it with standard SSL/TLS. An overall strategy of the EMI project is to remove the Globus proprietary GSI protocol in the EMI components and replace it with standard SSL/TLS readily available in target operating systems

The document further describes how to accomplish these targets for each CE model, please take a look at it for further details.

Evolution plans:

  • ARC-CE: Dynamic management of RTEs through the environment Janitor, accounting hooks, and Data Staging components need a serious evolution (this is a pure research and development activity)
  • ARC clients: Extending the resource discovery module and enhancing the job submission interface, language binding on non-linux platforms, enhancement and extension to support the wide area of storage protocols and information systems, and support job management for non Grid resources (local, for testing)
  • gLite Job Management services: Prevent jobs starting many hours after submission (WMS should be able to migrate jobs within different CEs, with a timeout), better support for MPI jobs, see below.
  • gLite MPI: Better control of user processes (how to map the processes to physical resources), better hooking management in MPI-Start, enhancement of scheduler module (better support for different LRMSs).
  • Unicore: more flexible generation of site-specific scripts.

Data Area Work Plan

These documents are to be published as of final versions by the middle of 2011. Migration of components begin afterwards, more or less within one year. For precise references and plans please check the official document.

There is a Harmonization process that consists of:

  • Catalogue syncronization, within SEs and LFCs, using a new, under development, message-passing mechanism.
  • Consolidation of Storage Resource Manager (SRM) protocol, EMI data will clarify the SRM v2.2 specification by adding more user-friendly documentation
  • Replacing the Globus httpg security protocol with the SSL/X509 (https) standard. They want to move some components to SSL/X509 away from GSI. the outcome of the latter goal is not clear, but EMI will investigate solutions to guarantee interoperability
  • Providing standard access to data through a mounted file system (NFS4.1)
  • Providing standard access to data via http(s) and WebDAV. May not affect us.
  • Publishing GLUE 2.0 information
  • Integration of the ARGUS EMI authorization system. All components agreed to access the ARGUS authorization at least for obtaining user blacklisting, Some SE's can go even further.
  • Consolidating data access client libraries. Together with ARC. This includes but is not limited to storage control (SRM), storage access (e.g. gsiFTP and http) and information protocols.

Evolution Plans:

  • Monitoring and accounting. Consolidate activities and offer mechanisms to better monitor user access and user accounting. Also proper definition of what accounting would mean for data is still missing.
  • Maintainability and Usability. Most of the services are planning to provide or improve Web interfaces or Command Line interfaces, work done by themselves.
  • Evolution in wide area protocols. Gridftp v2 under development in Globus, and available in dCache. NFSv4.1 not considered.

EMI_data_area_view.jpg

Security Area Work Plan

They divide the work into Product Teams, each taking care of their business. The dates they give are mentioned explicitly as approximate, so delays are very likely to happen.

Harmonization plan:

  • ARC Security Utils Product Team, introducing changes in update-crls, nordugridmap and arcproxy.
  • Argus Product Team. Argus 1.2 release is out, they are adding functionality required by other components and middleware stacks.
  • VOMS Product Team. The long-term strategy is to use a common SAML assertion source and subsequently a common SAML profile, resulting in a first Globus-free client and server release (VOMS 2.0). Do all VOMS changes we've seen lately have relation with this?
  • gLite Security Product Team. Worked on components: Trustmanager/Util-java, Delegation, Site Central Authorization Service (SCAS, to be replaced by Argus), Local Centre Authorization Service (LCAS)/ Local Credential MAPping Service (LCMAPS), gLExec, Hydra (key store service), Security Token Service (STS), Short Lived Credential Serviced (SLCS) and Pseudonymity
  • UNICORE Security Product Team, to integrate it with Argus (test the basics, evaluation of opensaml, API for a common authentication library)
  • CESNET Security Product Team. No harmonization plans.
  • GSI Removal. GSI removal problem needs more discussion across more than one technical area, since there are things still to understand. Not clear yet.
  • Common Authentication Libraries. Once the Common Authentication libraries are produced, it will become the task of the individual product teams to integrate these libraries into their components. For next EMI Release.
  • Common SAML Profile. SAML assertions are used in production by UNICORE and are seen as a way forward in order to “future-proof” gLite and ARC for client requests
  • Compute Area Authorization. Integrate CEs to use a common XACML profile.

Evolution (for the first year):

  • ARC Security Utils Product Team. The arcproxy utility will have to respond to client requests for features. The ARGUS Authorization system will need to be integrated to the ARC Security Utils as will VOMS-SAML. The support for MyProxy will be improved. The integration of the Argus policies into the mapfile generator should be completed.
  • Argus Product Team. No new developments. In a Cloud context the EES would configure, start, manage and stop VMs taking into account the policies as defined within the Argus system.
  • VOMS Product Team. C++ APIs will be rewritten, VOMS-Admin component test suite also rewritten.
  • gLite Security Product Team. Phase out gLExec. The plan is to create a set of modules following the Pluggable Authentication Modules (PAM) standard that will interact with Argus in order to test their suitability. Integrate Hydra concept to Cloud resources. Release Pseudonymity (anonymous yet auditable identity for users), Security Token Service (STS) development.
  • UNICORE Security Product Team. Verification performed of the UNICORE server side with respect to support for DN-based identification of entities everywhere
  • CESNET Security Product Team. May be that some work on the delegation tool that may be needed in response to requirements from other middleware areas.

Infrastructure Area Work Plan

Covers a wide range of domains including Information Services, Accounting, Service Monitoring and Instrumentation, Virtualization and Clouds, and Messaging.

Status

  • Information Services. GLUE2.06 is defined. There is a validation script for sites (gstat-validation). Three levels of BDII (resource, site and top) and there are command line tools (lcg-info and lcg-infosites). ARC Infoserver has migrated to the Classic Infoserver to use BDII and GLUE2. The Common Information Provider (CIP) service uses XML supporting GLUE2.
  • Logging and Bookkeeping. Provides an integrated view of job status. Not sure still where is it running.
  • Accounting. There is APEL and DGAS to provide accounting records.
  • Messaging. They use Apache ActiveMQ to pass information, has been used for high-level cross-grid applications for a while. Not sure where either.
  • Service Monitoring and Management. Neither Nagios or SAM, together with GLUE2 provide a sufficiently fine-grained and consistent insight into the inner workings of the services for sysadmins.
  • Virtualization and clouds. The main user communities that EMI aims to support are becoming increasingly interested in Virtualization and Cloud computing technologies.

Plans

  • Information Services. BDII (topl), ARC-LDAP and CIP to be migrated to GLUE2. Investigation into Information Transportation Mechanisms between BDII-site and top. Publishing information about storage resources
  • Logging & Bookkeeping Development. IPv6 compliance, Adoption of the common logging format, GLUE2, Elementary native support for CREAM jobs, log sandbox transfer progress, Configuration tuned to allow collocation with WMS, Advanced authorization, Exporting notification.
  • Accounting. Proper accounting of MPI jobs, Extend the APEL client to provide summary publication, New sensor module for DGAS, storage accounting for DGAS using ActiveMQ
  • Messaging Development. Paper survey of the various options for an EMI messaging service (April 2011). Gather User Requirements.
  • Service Monitoring and Management. Investigating Requirements. Apr11.
  • Virtualization and clouds. Investigate the possible usage of Virtualization. Apr11.

-- PabloFernandez - 2011-02-08

Edit | Attach | Watch | Print version | History: r3 < r2 < r1 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r3 - 2011-02-09 - PabloFernandez
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback