New EMI Requirements and Work Plans

Taken from an email from Andres Aeschlimann (to operations@swing-grid.ch, Feb 7th 14:51), for the EMI Roadmap there is https://wiki.egi.eu/wiki/EGI_Operations#EMI

Here is the EMI 1 release plan: https://twiki.cern.ch/twiki/bin/view/EMI/InternalDeliverableEmi1ReleaseDevPlans

• DJRA1.1.1 – Compute Area Work Plan and Status Report https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA111
• DJRA1.2.1 – Data Area Work Plan and Status Report https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA121
• DJRA1.3.1 – Security Area Work Plan and Status Report https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA131
• DJRA1.4.1 - Infrastructure Area Work Plan and Status Report https://twiki.cern.ch/twiki/bin/view/EMI/DeliverableDJRA141

Release schedule of different components: https://savannah.cern.ch/task/?group=emi-rel-sched&func=browse&set=open&msort=0&advsrch=0&morder=bug_id%3C&order=bug_id#results

EMI Services documentation: https://twiki.cern.ch/twiki/bin/view/EMI/EMIServicesDocs

Compute Area Work Plan

ARC, gLite CREAM (with CEMon and BLAH) and UNICORE provide the same core functionality; they are implemented in different and often incompatible ways. They want to address some of these issues. In particular the plan is:

• To define and adopt job execution standard/agreed interfaces, with all three representatives, work in progress.
• To adopt information service related standards. GLUE v. 2.0.
• To harmonize the used authorization mechanisms. gLite Argus service.
• To harmonize the parallel and MPI job support. Provision of a standard EMI unified layer
• To consolidate and harmonize the compute area clients and APIs. The main objectives are to improve: usability, maintainability and portability to different architectures and operating systems.
• To assess the use of a messaging system between the EMI job management components. It's not clear if it's needed, but it's under investigation (maybe messages between WMS and Cream?)
• To remove GSI, and replace it with standard SSL/TLS. An overall strategy of the EMI project is to remove the Globus proprietary GSI protocol in the EMI components and replace it with standard SSL/TLS readily available in target operating systems

The document further describes how to accomplish these targets for each CE model, please take a look at it for further details.

Evolution plans:

• ARC-CE: Dynamic management of RTEs through the environment Janitor, accounting hooks, and Data Staging components need a serious evolution (this is a pure research and development activity)
• ARC clients: Extending the resource discovery module and enhancing the job submission interface, language binding on non-linux platforms, enhancement and extension to support the wide area of storage protocols and information systems, and support job management for non Grid resources (local, for testing)
• gLite Job Management services: Prevent jobs starting many hours after submission (WMS should be able to migrate jobs within different CEs, with a timeout), better support for MPI jobs, see below.
• gLite MPI: Better control of user processes (how to map the processes to physical resources), better hooking management in MPI-Start, enhancement of scheduler module (better support for different LRMSs).
• Unicore: more flexible generation of site-specific scripts.

Data Area Work Plan

These documents are to be published as of final versions by the middle of 2011. Migration of components begin afterwards, more or less within one year. For precise references and plans please check the official document.

There is a Harmonization process that consists of:

• Catalogue syncronization, within SEs and LFCs, using a new, under development, message-passing mechanism.
• Consolidation of Storage Resource Manager (SRM) protocol, EMI data will clarify the SRM v2.2 specification by adding more user-friendly documentation
• Replacing the Globus httpg security protocol with the SSL/X509 (https) standard. They want to move some components to SSL/X509 away from GSI. the outcome of the latter goal is not clear, but EMI will investigate solutions to guarantee interoperability
• Providing standard access to data through a mounted file system (NFS4.1)
• Providing standard access to data via http(s) and WebDAV. May not affect us.
• Publishing GLUE 2.0 information
• Integration of the ARGUS EMI authorization system. All components agreed to access the ARGUS authorization at least for obtaining user blacklisting, Some SE's can go even further.
• Consolidating data access client libraries. Together with ARC. This includes but is not limited to storage control (SRM), storage access (e.g. gsiFTP and http) and information protocols.

Evolution Plans:

• Monitoring and accounting. Consolidate activities and offer mechanisms to better monitor user access and user accounting. Also proper definition of what accounting would mean for data is still missing.
• Maintainability and Usability. Most of the services are planning to provide or improve Web interfaces or Command Line interfaces, work done by themselves.
• Evolution in wide area protocols. Gridftp v2 under development in Globus, and available in dCache. NFSv4.1 not considered.

Security Area Work Plan

They divide the work into Product Teams, each taking care of their business. The dates they give are mentioned explicitly as approximate, so delays are very likely to happen.

Harmonization plan:

• ARC Security Utils Product Team, introducing changes in update-crls, nordugridmap and arcproxy.
• Argus Product Team. Argus 1.2 release is out, they are adding functionality required by other components and middleware stacks.
• VOMS Product Team. The long-term strategy is to use a common SAML assertion source and subsequently a common SAML profile, resulting in a first Globus-free client and server release (VOMS 2.0). Do all VOMS changes we've seen lately have relation with this?
• gLite Security Product Team. Worked on components: Trustmanager/Util-java, Delegation, Site Central Authorization Service (SCAS, to be replaced by Argus), Local Centre Authorization Service (LCAS)/ Local Credential MAPping Service (LCMAPS), gLExec, Hydra (key store service), Security Token Service (STS), Short Lived Credential Serviced (SLCS) and Pseudonymity
• UNICORE Security Product Team, to integrate it with Argus (test the basics, evaluation of opensaml, API for a common authentication library)
• CESNET Security Product Team. No harmonization plans.
• GSI Removal. GSI removal problem needs more discussion across more than one technical area, since there are things still to understand. Not clear yet.
• Common Authentication Libraries. Once the Common Authentication libraries are produced, it will become the task of the individual product teams to integrate these libraries into their components. For next EMI Release.
• Common SAML Profile. SAML assertions are used in production by UNICORE and are seen as a way forward in order to “future-proof” gLite and ARC for client requests
• Compute Area Authorization. Integrate CEs to use a common XACML profile.

Evolution (for the first year):

• ARC Security Utils Product Team. The arcproxy utility will have to respond to client requests for features. The ARGUS Authorization system will need to be integrated to the ARC Security Utils as will VOMS-SAML. The support for MyProxy will be improved. The integration of the Argus policies into the mapfile generator should be completed.
• Argus Product Team. No new developments. In a Cloud context the EES would configure, start, manage and stop VMs taking into account the policies as defined within the Argus system.
• VOMS Product Team. C++ APIs will be rewritten, VOMS-Admin component test suite also rewritten.
• gLite Security Product Team. Phase out gLExec. The plan is to create a set of modules following the Pluggable Authentication Modules (PAM) standard that will interact with Argus in order to test their suitability. Integrate Hydra concept to Cloud resources. Release Pseudonymity (anonymous yet auditable identity for users), Security Token Service (STS) development.
• UNICORE Security Product Team. Verification performed of the UNICORE server side with respect to support for DN-based identification of entities everywhere
• CESNET Security Product Team. May be that some work on the delegation tool that may be needed in response to requirements from other middleware areas.

Infrastructure Area Work Plan

TO BE DONE.

3. Status   9
3.1. Information Services   9
3.1.1 GLUE Information Model   9
3.1.1.1 Information Validation   9
3.1.2 gLite Information System   9
3.1.2.1 BDII Service (resource-level)   10
3.1.2.2 BDII Service (site-level)   10
3.1.2.3 BDII Service (top-level)   10
3.1.2.4 Client Side Tools   10
3.1.3 ARC LDAP-based Infosys   10
3.1.3.1 Classic Infoserver    10
3.1.3.2 Classic Infoindex (EGIIS)   11
3.1.4 UNICORE Registry   11
3.1.5 Common Information Provider (CIP)   11
3.2. Logging and Bookkeeping   11
3.3. Accounting   12
3.3.1 APEL Client   12
3.3.2 DGAS Client   12
3.4. Messaging   12
3.5. Service Monitoring and Management   12
3.6. Virtualization and clouds   13
4. Plans   14
4.1. Information Services   14
4.1.1 gLite Information System Development   14
4.1.2 ARC LDAP-based Infosys Development   14
4.1.3 Common Information Provider Development   15
4.1.4 EMI Registry   15
4.2. Logging & Bookkeeping Development   15
4.3. Accounting   16
4.3.1 APEL Client Development   16
4.3.2 DGAS Client   17
4.4. Messaging Development   18
4.5. Service Monitoring and Management   19
4.6. Virtualization and clouds   19

-- PabloFernandez - 2011-02-08