Tags:
tag this topic
create new tag
view all tags
<!-- keep this as a security measure: * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup #uncomment this if you want the page only be viewable by the internal people # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup --> ---+!! %TOPIC% %TOC% ---++ Symptoms Summary: %FORMFIELD{"Symptom summary"}% ---++ Occurrences At what times did this problem occur (used to estimate frequency): | 2011-08-05 | After wiping Scratch FS | ---++ Observations <!-- #collect here the information which may help to better understand the state of the system or services, e.g. #log excerpts, strace output, etc. #this also may help to identify the problem if similar conditions arise again --> This happens when the CreamCE thinks there is a delegated proxy in the Cream Sandbox, but it was deleted from there for whatever reason. You can detect it happening because jobs fail inmediately after starting (exit_status=1) and Cream reports this error in *glite-ce-cream.log*" <verbatim>05 Aug 2011 16:45:49,917 INFO org.glite.ce.creamapi.jobmanagement.cmdexecutor.AbstractJobExecutor (AbstractJobExecutor.java:2163) - (Worker Thread 49) JOB CREAM459165977 STATUS CHANGED: RUNNING => DONE-FAILED [failureReason=Cannot move ISB (retry_copy ${globus_transfer_cmd} gsiftp://rb1.cyf-kr.edu.pl:2811/var/glite/SandboxDir/x2/ https_3a_2f_2flb.reef.man.poznan.pl_3a9000_2fx2r6TFgNA_5f7pIJOPpag2Sg/input/h1mcLauncher_perl_ftt.sh file:///lustre/scratch/home/egee/honeprd/home_cre01_459165977/CREAM459165977/h1mcLauncher_perl_ftt.sh): Problem to detect the lifetime of the proxy] [localUser=honeprd] [gridJobId=https://lb.reef.man.poznan.pl:9000/x2r6TFgNA_7pIJOPpag2Sg] [workerNode=wn122.lcg.cscs.ch] [delegationId=13041121772E840574rb12Ecyf2Dkr2Eedu2Epl] </verbatim> ---++ Solution or Workaround This can be solved by removing the entry from the CreamCE where it says there's an existing delegated proxy. If you run this script in the CreamCE log dir: <verbatim>grep lifetime glite-ce-cream.log glite-ce-cream.log.? | grep DONE-FAILED | egrep -o "delegationId=[^]]*" | \ sed 's/dele.*\=//' | sort -u | while read i; do echo "delete from t_credential where dlg_id='$i';"; done</verbatim> It will tell you the sql commands you need to execute in the cream DB: =mysql -u cream -p delegationdb= ---++ Deleting delegated proxies on bad files Another possibility is to detect bad files and delete the entries in the database. This may be useful:<verbatim>for i in `find /lustre/scratch/CREAM_CE/cream02 -type d -maxdepth 3`; \ do ls -la $i/proxy;done 2> /dev/null | awk '/\?/ {print $7}' | while read i; do echo "delete from t_credential where dlg_id='$i';"; done</verbatim> ---++ Monitoring for this condition <!-- #how can this condition be recognized automatically, if at all? --> -- Main.PabloFernandez - 2011-08-05
IssueForm
Affected Service
cream CE
Symptom summary
Cannot move ISB, Problem to detect the lifetime of the proxy
Reason Understood
yes
Solution Exists
yes
Obsolete
no
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r4
<
r3
<
r2
<
r1
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r4 - 2011-08-19
-
PabloFernandez
LCGTier2
Log In
(Topic)
LCGTier2 Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Users
Entry point / Contact
RoadMap
ATLAS Pages
CMS Pages
CMS User Howto
CHIPP CB
Outreach
Technical
Cluster details
Services
Hardware and OS
Tools & Tips
Monitoring
Logs
Maintenances
Meetings
Tests
Issues
Blog
Home
Site map
CmsTier3 web
LCGTier2 web
PhaseC web
Main web
Sandbox web
TWiki web
LCGTier2 Web
Users
Groups
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
P
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Warning: Can't find topic "".""
Account
Log In
E
dit
A
ttach
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback