CMSSiteLog15 < LCGTier2

<!-- keep this as a security measure:
   * Set ALLOWTOPICCHANGE = Main.TWikiAdminGroup,Main.LCGAdminGroup
   * Set ALLOWTOPICRENAME = Main.TWikiAdminGroup,Main.LCGAdminGroup
   #uncomment this if you want the page only be viewable by the internal people
   # * Set ALLOWTOPICVIEW = Main.TWikiAdminGroup,Main.LCGAdminGroup
-->

%TOC%

%ICON{arrowleft}% Go to [[CMSSiteLog14][previous page]] / [[CMSSiteLog16][next page]] of CMS site log %M%

---+ 24. 06. 2009 Migration of se05_cms dcache pool files to other pools with the shelltools

---++ Introduction

Note: We do this manually with the shelltools, since the copy cell cannot be trusted in this 1.8-x version of dcache, and we will only update dcache to the 1.9-x versions in July.

This uses my DcacheShellutils tools.

All of these steps are done in the directory =/home/staff/dfeich/shellutils/work/se05_cms-migration=

Check that se05_cms is read only, so that we can prevent new files appearing on it:
<pre>
dc_get_pool_list.sh -r |grep se05_cms
se05_cms  (enabled=true;active=12;rdOnly=true;links=0;pgroups=1;hsm=[];mode=enabled)
</pre>

se05 is one of the older pools from a time that predates space management. At that time we only had _precious_ and _cached_ files (cached meaning that dcache was allowed to remove these copies when it needed space). Later we got space management, and it seems that space managed files get the _sticky_ characteristic.
This can be nicely seen from the dCache web GUI's [[http://%SEHOST%:2288/usageInfo][pool usage page]], where all these categories get different color bars. The old pools all still have some precious files (red), while the new ones only have sticky (pink) files. There are almost no cached files (green), because cached replicas are only generated by dcache, if there is a underlying HSM behind it (... I, think).

Let's get lists of all files on the pool, and of oll files that are precious and sticky. *Note that this differentiation is done by the se05_cms pool cell's =rep ls= command!* So the correctness depends on the correctness of that command!

<pre>
dc_get_rep_ls.sh se05_cms > se05_cms-ID.lst

dc_get_rep_ls.sh -l p se05_cms > se05_cms-PRECIOUS-ID.lst

dc_get_rep_ls.sh -l s se05_cms > se05_cms-STICKY-ID.lst

dc_get_rep_ls.sh -c se05_cms > se05_cms-CACHED-ID.lst  # careful. just use "-c", not "-l c"
</pre>

Let's count the entries
<pre>
wc -l se05_cms-ID.lst se05_cms-PRECIOUS-ID.lst se05_cms-STICKY-ID.lst
   5526 se05_cms-ID.lst
  1267 se05_cms-PRECIOUS-ID.lst
  4546 se05_cms-STICKY-ID.lst
</pre>

So, 1267 + 4546 = 5813. Seems that some sticky files are also precious at the same time. This may derive from the fact that an earlier migration, shortly after the space management was introduces, manually marked some files as precious...


---++ Finding out about sticky, cached and precious files, and how to correctly create new copies of files

I first need to find out how I can reliably differentiate between sticky, cached and precious files using the admin shell, and how copies with the right properties can be created. For this, I'll do a little research with the shelltools.

Let's look at the characteristics of every kind of file....

Precious:
<verbatim>
tail -10 se05_cms-PRECIOUS-ID.lst | dc_get_rep_ls.sh -r -i se05_cms | grep -v admin

00020000000000000000E150 <-P---------(0)[0]> 70901468 si={cms:cms}
0002000000000000000543B0 <-P---------(0)[0]> 2463153484 si={cms:cms}
0002000000000000000744A8 <-P---------(0)[0]> 808370799 si={cms:cms}
0002000000000000000F1018 <-P---------(0)[0]> 432776 si={cms:cms}
00020000000000000001C380 <-P---------(0)[0]> 56193849 si={cms:cms}
00020000000000000004DF40 <-P---------(0)[0]> 2476990936 si={cms:cms}
00020000000000000000C498 <-P---------(0)[0]> 69011452 si={cms:cms}
0002000000000000000207D8 <-P---------(0)[0]> 59985791 si={cms:cms}
0002000000000000000EA730 <-P---------(0)[0]> 427697 si={cms:cms}
00020000000000000024E818 <-P---------(0)[0]> 495136770 si={cms:cms}
</verbatim>

Sticky:
<verbatim>
tail -10 se05_cms-STICKY-ID.lst | dc_get_rep_ls.sh -r -i se05_cms | grep -v admin

000200000000000001B46118 <C-------X--(0)[0]> 1968743916 si={cms:cms}
000200000000000001AD9838 <C-------X--(0)[0]> 1629486174 si={cms:cms}
0002000000000000011D7320 <C-------X--(0)[0]> 167411059 si={cms:cms}
00020000000000000224AC90 <C-------X--(0)[0]> 25856206 si={cms:cms}
000200000000000000E82D50 <C-------X--(0)[0]> 35728076 si={cms:cms}
000200000000000001C0E0D8 <C-------X--(0)[0]> 739802749 si={cms:cms}
0002000000000000023BB850 <C-------X--(0)[0]> 1092259150 si={cms:cms}
000200000000000001AD1D18 <C-------X--(0)[0]> 2172298863 si={cms:cms}
0002000000000000011C3880 <C-------X--(0)[0]> 171887245 si={cms:cms}
000200000000000001BFA3E8 <C-------X--(0)[0]> 498125871 si={cms:cms}
</verbatim>

Hmmm.... I noted from pre-space management times, that =<C-------X--(0)[0]>= marked _cached_ files, so it seems that these flags are not adequate in differentiating between _sticky_ and simply _cached_ files. Bad. *Added later: Seems I made a wrong conclusion at that time... look further below. cached files lack the -X- flag!. This makes it easy again*


I found in the admin interface of the pool, the =rep sticky ls <pnfsID>= command. Maybe this can be used. Let's test the theory with the =dc_generic_cellcommand.sh= on some of the _precious_ files.
<verbatim>
head -20 se05_cms-PRECIOUS-ID.lst | dc_generic_cellcommand.sh -f -c 'rep sticky ls $n' se05_cms

    dCache Admin (VII) (user=admin)


[storage01.lcg.cscs.ch] (local) admin > cd se05_cms
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000BB020
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000013FE0
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000024CB50
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000018E3B60
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000EC520
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000062BB8
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000078648
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000A6AA8
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000241E0
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000858D0
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000011B148
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000E8730
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000004DEF0
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000018B6FE0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000008F958
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000008E68
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000F11D8
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000018987B0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001A896A8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000008A98
[storage01.lcg.cscs.ch] (se05_cms) admin > ..
[storage01.lcg.cscs.ch] (local) admin > logoff
dmg.util.CommandExitException: (0) Done
</verbatim>

And now the same for the _sticky_ files:

<verbatim>
head -20 se05_cms-STICKY-ID.lst | dc_generic_cellcommand.sh -f -c 'rep sticky ls $n' se05_cms

    dCache Admin (VII) (user=admin)


[storage01.lcg.cscs.ch] (local) admin > cd se05_cms
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000137A2C8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001A980A8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000015D2568
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001C29030
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000023B65E0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000002144460
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001B482A0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000121F038
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001C9C7B8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001ABE618
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000E76DB8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001B09E00
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000013651A8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000018E3B60
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000136ED30
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001C4FD00
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001B51810
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001ECA7B8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001D2E990
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001C221C0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > ..
[storage01.lcg.cscs.ch] (local) admin > logoff
dmg.util.CommandExitException: (0) Done
</verbatim>


In chapter 18 of the dcache book (_Pinning Files to a Pool_), I can see that the characteristic _sticky_ can apply to a single replicate of a file on a pool, or to all copies of a file on all pools. In the first case, this is a attribute of the file instance in the pool, in the other global case, the information is stored within the pnfs data base:
<pre>
rep set sticky pnfsid on|off      # pool cell command to make file inside this pool sticky

set sticky pnfsid   # global effect. Can be used from the top entry point into
                           # the admin cell (e.g. after doing "cd ..")
</pre>


Let's test how we can copy a file and make it sticky

<verbatim>
srmcp -debug file:///`pwd`/se05_cms-ID.lst srm://storage01.lcg.cscs.ch:8443/srm/managerv1?SFN=//pnfs/lcg.cscs.ch/cms/local_tests/derek-sticky-test

echo /pnfs/lcg.cscs.ch/cms/local_tests/derek-sticky-test | dc_get_ID_from_pnfsnamelist.sh
00020000000000000243CD00 /pnfs/lcg.cscs.ch/cms/local_tests/derek-sticky-test

echo 00020000000000000243CD00 | dc_get_cacheinfo_from_IDlist.sh
00020000000000000243CD00 se19_cms


echo 00020000000000000243CD00 | dc_get_rep_ls.sh -r -i se19_cms | grep -v admin
00020000000000000243CD00 <C-------X--(0)[0]> 138150 si={cms:cms}

echo 00020000000000000243CD00 | dc_ppcopy_files.sh -f se19_cms se18_cms

echo 00020000000000000243CD00 | dc_get_cacheinfo_from_IDlist.sh
00020000000000000243CD00 se18_cms,se19_cms

echo 00020000000000000243CD00 | dc_get_rep_ls.sh -r -i se18_cms | grep -v admin
00020000000000000243CD00 <C----------(0)[0]> 138150 si={cms:cms}

</verbatim>

Ok. Let's try to make the new copy on pool se18_cms sticky

<verbatim>
echo 00020000000000000243CD00 | dc_generic_cellcommand.sh -f -c 'rep set sticky $n on' se18_cms
</verbatim>

And voila! Here we correctly get our sticky file on this pool  :-D
<verbatim>
echo 00020000000000000243CD00 | dc_get_rep_ls.sh -r -i se18_cms | grep -v admin
00020000000000000243CD00 <C-------X--(0)[0]> 138150 si={cms:cms}
</verbatim>

---+++ special case of precious AND sticky files on this old pool

Let's find files which are precious and sticky

<pre>
for n in `cat se05_cms-PRECIOUS-ID.lst`; do grep -q $n se05_cms-STICKY-ID.lst; if test $? -ne 0; then echo $n; fi; done >precious-and-sticky.lst
</pre>

Now we have a look at the flags for these files that are sticky and precious at the same time.
<verbatim>
cat precious-and-sticky.lst | dc_get_rep_ls.sh -i -r  se05_cms | grep -v admin

0002000000000000000BB020 <-P---------(0)[0]> 179755831 si={cms:cms}
000200000000000000013FE0 <-P---------(0)[0]> 1510974283 si={cms:cms}
00020000000000000024CB50 <-P---------(0)[0]> 1756795340 si={cms:cms}
0002000000000000000EC520 <-P---------(0)[0]> 472898 si={cms:cms}
000200000000000000062BB8 <-P---------(0)[0]> 19387237 si={cms:cms}
000200000000000000078648 <-P---------(0)[0]> 262656755 si={cms:cms}
0002000000000000000A6AA8 <-P---------(0)[0]> 134572991 si={cms:cms}
0002000000000000000241E0 <-P---------(0)[0]> 102410208 si={cms:cms}
0002000000000000000858D0 <-P---------(0)[0]> 162611125 si={cms:cms}
00020000000000000011B148 <-P---------(0)[0]> 450574 si={cms:cms}
...
</verbatim>


Let's look at files which are only sticky
<verbatim>
for n in `cat se05_cms-PRECIOUS-ID.lst`; do grep -q $n se05_cms-STICKY-ID.lst; if test $? -eq 0; then echo $n; fi; done > only-sticky.lst

head -20 only-sticky.lst | dc_get_rep_ls.sh -r -i se05_cms | grep -v admin

0002000000000000018E3B60 <-P------X--(0)[0]> 3336915216 si={cms:cms}
0002000000000000018B6FE0 <-P------X--(0)[0]> 787573248 si={cms:cms}
0002000000000000018987B0 <-P------X--(0)[0]> 3222094048 si={cms:cms}
000200000000000001A896A8 <-P------X--(0)[0]> 674796192 si={cms:cms}
0002000000000000018E77A8 <-P------X--(0)[0]> 1390713984 si={cms:cms}
00020000000000000189EF28 <-P------X--(0)[0]> 810449872 si={cms:cms}
0002000000000000018D23A8 <-P------X--(0)[0]> 137643984 si={cms:cms}
0002000000000000018CC2E8 <-P------X--(0)[0]> 700283680 si={cms:cms}
0002000000000000018D2898 <-P------X--(0)[0]> 111847864 si={cms:cms}
</verbatim>



Let's look at files which are only precious

<verbatim>
for n in `cat se05_cms-PRECIOUS-ID.lst`; do grep -q $n precious-and-sticky.lst; if test $? -ne 0; then echo $n; fi; done > only-precious.lst

head -20 only-precious.lst | dc_get_rep_ls.sh -r -i se05_cms | grep -v admin
0002000000000000018E3B60 <-P------X--(0)[0]> 3336915216 si={cms:cms}
0002000000000000018B6FE0 <-P------X--(0)[0]> 787573248 si={cms:cms}
0002000000000000018987B0 <-P------X--(0)[0]> 3222094048 si={cms:cms}
000200000000000001A896A8 <-P------X--(0)[0]> 674796192 si={cms:cms}
0002000000000000018E77A8 <-P------X--(0)[0]> 1390713984 si={cms:cms}
00020000000000000189EF28 <-P------X--(0)[0]> 810449872 si={cms:cms}
0002000000000000018D23A8 <-P------X--(0)[0]> 137643984 si={cms:cms}
0002000000000000018CC2E8 <-P------X--(0)[0]> 700283680 si={cms:cms}
0002000000000000018D2898 <-P------X--(0)[0]> 111847864 si={cms:cms}
...
</verbatim>

Drats! :-(  the flag signature is the same as for the _only precious_ files... something is wrong....

Let's do a check:
<pre>
dc_get_rep_ls.sh -i -l p se05_cms only-precious.lst > op-gaga.lst

wc -l op-gaga.lst only-precious.lst
  317 op-gaga.lst
  317 only-precious.lst
</pre>


<pre>
dc_get_rep_ls.sh -i -l s se05_cms only-precious.lst > check.lst
wc -l check.lst
317 check.lst
</pre>

*NEED TO BETTER RESEARCH THIS.....*

---++ How to migrate the files in a pool

We get the list of files that we want to transfer
<pre>
dc_get_rep_ls.sh se05_cms > se05_cms-ID.lst
</pre>

We split the files into batches of 500 files for the migration
<pre>
split -d -l 500 se05_cms-ID.lst batch_ID
</pre>

Test the migration with the first batch
<pre>
time dc_ppcopy_files.sh se05_cms se15_cms batch_ID00
real    1m52.828s
user    0m0.060s
sys     0m0.140s
</pre>

We can see that the transfer load on the machines increases immediately

<img src="%ATTACHURLPATH%/se05_src.png" alt="se05_src.png" width='227' height='119' />
<img src="%ATTACHURLPATH%/se15_dst.png" alt="se15_dst.png" width='227' height='119' />

The transfers seem to fill one Gb connection. Every thumper has 4 aggregated 1Gb ethernet connections, but it may be that the hash function governing the distribution over the physical interfaces evaluates to the same for all these connections, so we effectively are only using a single one.

The thumper's CPU spends lots of time in system/kernel space, probably because the connections are fighting for the network. Based on our experience in writing to the 48 disks, the problem should not be located in disk I/O.

<img src="%ATTACHURLPATH%/se15_cpu.png" alt="se15_cpu.png" width='397' height='168' />

The number of queued transfers can easily be seen by looking at the [[http://%SEHOST%:2288/queueInfo][queue info]] page.

The copies finished around 16:30h.

Now we do a sanity check whether all copies are ok:

<pre>
dc_get_rep_ls.sh -i se15_cms batch_ID00 > batch_ID00.check

wc -l batch_ID00  batch_ID00.check
  500 batch_ID00
  500 batch_ID00.check
</pre>

We do a little check for the flags (should be cached-only files):
<verbatim>
head -10 batch_ID00 | dc_get_rep_ls.sh -r -i se15_cms | grep -v admin

00020000000000000137A2C8 <C----------(0)[0]> 156990918 si={cms:cms}
000200000000000001A980A8 <C----------(0)[0]> 4769081784 si={cms:cms}
0002000000000000015D2568 <C----------(0)[0]> 4162 si={cms:cms}
000200000000000001C29030 <C----------(0)[0]> 22260183 si={cms:cms}
0002000000000000023B65E0 <C----------(0)[0]> 1187411081 si={cms:cms}
000200000000000002144460 <C----------(0)[0]> 2578597264 si={cms:cms}
000200000000000001B482A0 <C----------(0)[0]> 1782946180 si={cms:cms}
00020000000000000121F038 <C----------(0)[0]> 1572443586 si={cms:cms}
000200000000000001C9C7B8 <C----------(0)[0]> 1921029783 si={cms:cms}
0002000000000000000BB020 <C----------(0)[0]> 179755831 si={cms:cms}
</verbatim>

We set the files to sticky for that pool

<pre>
time dc_generic_cellcommand.sh -c 'rep set sticky $n on' se15_cms batch_ID00
real    0m40.966s
user    0m0.050s
sys     0m0.630s
</pre>

And again, we check the flag values

<verbatim>
 head -10 batch_ID00 | dc_get_rep_ls.sh -r -i se15_cms | grep -v admin

00020000000000000137A2C8 <C-------X--(0)[0]> 156990918 si={cms:cms}
000200000000000001A980A8 <C-------X--(0)[0]> 4769081784 si={cms:cms}
0002000000000000015D2568 <C-------X--(0)[0]> 4162 si={cms:cms}
000200000000000001C29030 <C-------X--(0)[0]> 22260183 si={cms:cms}
0002000000000000023B65E0 <C-------X--(0)[0]> 1187411081 si={cms:cms}
000200000000000002144460 <C-------X--(0)[0]> 2578597264 si={cms:cms}
000200000000000001B482A0 <C-------X--(0)[0]> 1782946180 si={cms:cms}
00020000000000000121F038 <C-------X--(0)[0]> 1572443586 si={cms:cms}
000200000000000001C9C7B8 <C-------X--(0)[0]> 1921029783 si={cms:cms}
0002000000000000000BB020 <C-------X--(0)[0]> 179755831 si={cms:cms}
</verbatim>

Seems ok :-D

Fotis did the next steps:
<pre>
time dc_ppcopy_files.sh se05_cms se11_cms batch_ID01
time dc_ppcopy_files.sh se05_cms se12_cms batch_ID02
time dc_ppcopy_files.sh se05_cms se13_cms batch_ID03
time dc_ppcopy_files.sh se05_cms se14_cms batch_ID04
</pre>

The full list of commands and pool destinations is, or should be:
<pre>
time ./dc_ppcopy_files.sh se05_cms se15_cms batch_ID00 # executed, sticky'ed & verified
time ./dc_ppcopy_files.sh se05_cms se11_cms batch_ID01 # executed
time ./dc_ppcopy_files.sh se05_cms se12_cms batch_ID02 # executed
time ./dc_ppcopy_files.sh se05_cms se13_cms batch_ID03 # executed
time ./dc_ppcopy_files.sh se05_cms se14_cms batch_ID04 # executed
time ./dc_ppcopy_files.sh se05_cms se15_cms batch_ID05 # executed
time ./dc_ppcopy_files.sh se05_cms se16_cms batch_ID06 # executed
time ./dc_ppcopy_files.sh se05_cms se17_cms batch_ID07 # executed
time ./dc_ppcopy_files.sh se05_cms se18_cms batch_ID08 # executed
time ./dc_ppcopy_files.sh se05_cms se19_cms batch_ID09 # executed
time ./dc_ppcopy_files.sh se05_cms se20_cms batch_ID10 # executed
time ./dc_ppcopy_files.sh se05_cms se20_cms batch_ID11 # executed
</pre>


---+++ Tests whether everything has worked out

---++++ Have all files been copied?
Let's do an independent test whether all has worked out correctly. Every file should now have at least two replicas, one one the original pool se05_cms, and one on another pool. So we just get the cacheinfo for every file, and look whether there are files which only have one replicate.
<pre>
dc_get_cacheinfo_from_IDlist.sh se05_cms-ID.lst > checklist

# In the command's output multiple pools are separated by a colon ",". So we can search lines without colons to find suspicious entries
#   (i.e. files with only one or no phys replicate)
grep -v ',' checklist
0002000000000000019B9C30

# let's investigate this single ID
echo 0002000000000000019B9C30 | dc_get_cacheinfo_from_IDlist.sh
0002000000000000019B9C30

# let's find the filename
echo 0002000000000000019B9C30 | dc_get_pnfsname_from_IDlist.sh
0002000000000000019B9C30 /pnfs/lcg.cscs.ch/cms/local/stiegerb/hfegsignal50/hfegsignal_69.root
</pre>

Ok. This file is from a local user area. Maybe the user has done something to it. Problem is now that we have an orphaned pnfs entry without a file beneath it. Usually one should ask the user in respect to this.
But this is a small matter in comparison to the whole migration that we did :-D

---++++ Are all files sticky?

use something like
<pre>
dc_get_rep_ls.sh -r -i se12_atlas filelist.lst > check_se12_atlas_rep.lst
grep "C--" check_se12_atlas_rep.lst | grep -v '\-X\-'
</pre>

---++ Example how to transfer back misplaced ATLAS files from CMS pools

For all affected pools, we generate pool file lists and pnfs mappings, e.g.
<pre>
dc_get_rep_ls.sh se12_cms > se12_cms_id.lst
dc_get_pnfsname_from_IDlist.sh se12_cms_id.lst > se12_cms_pnfs.lst
</pre>


We filter out files belonging to atlas and generate lists with the misplaced files per pool
<pre>
for n in *pnfs.lst; do grep "lcg.cscs.ch/atlas" $n > wrongatlas_"${n}"; done
</pre>

*The next steps demonstrate the sequence for a single pool. We transfer the files from the node's CMS pool to the ATLAS pool on the same node.*

   1. issue copy commands (careful: target files end up as cached only). This proved successful for these lists of 5000 files! <pre>
time dc_ppcopy_files.sh -f se12_cms se12_atlas wrongatlas_se12_cms_pnfs.lst 
real    13m23.419s
</pre>
   1. for checking whether all files were duplicated <pre>
time dc_get_cacheinfo_from_IDlist.sh wrongatlas_se12_cms_pnfs.lst > wrongatlas_se12_cms_cacheinfo.lst
real    8m9.301s

grep -v 'se12_atlas' wrongatlas_se12_cms_cacheinfo.lst
</pre>
      * If there are missing files, again pipe their IDs into a ppcopy command.
   1. for dc_generic_cellcommand we need a list of IDs only <pre>
cut -f1 -d' ' wrongatlas_se12_cms_pnfs.lst > wrongatlas_se12_cms_id.lst
</pre>
   1. set files sticky on new pool <pre>
time dc_generic_cellcommand.sh -f -c 'rep set sticky $n on' se12_atlas wrongatlas_se12_cms_id.lst
real    9m16.211s
</pre>
   1. for checking whether all files have been set sticky <pre>
time dc_get_rep_ls.sh -r -i se12_atlas wrongatlas_se12_cms_id.lst > check_se12_atlas_rep.lst
real    8m20.484s

grep "C--" check_se12_atlas_rep.lst | grep -v '\-X\-'
</pre>
   1. If all looks ok, remove the misplaced files from the CMS pool *CAREFUL! This is irreversible!!!!* <pre>
dc_rep_rm_list.sh -y -f se12_cms wrongatlas_se12_cms_id.lst
</pre>

List of corrected pools
| *Pool* | *status* | *problems* |
| se12 | done ||
| se13 | done ||
| se14 | done ||
| se15 | done ||
| se16 | done ||
| se17 | done |0003000000000000043E6DE0 0003000000000000043E6DB8 0003000000000000043E7778|
| se18 | done |0003000000000000043E7F38 0003000000000000043E7D70 0003000000000000043E78E0|
| se19 | done |000300000000000003B99948 0003000000000000043E7F10 000300000000000000FD6388 0003000000000000043E7B40|
| se20 | done ||

----------------

%ICON{arrowleft}% Go to [[CMSSiteLog14][previous page]] / [[CMSSiteLog16][next page]] of CMS site log %M%
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who
png	se05_src.png	r1	manage	6.2 K	2009-06-25 - 13:32	DerekFeichtinger
png	se15_cpu.png	r1	manage	12.0 K	2009-06-25 - 13:39	DerekFeichtinger
png	se15_dst.png	r1	manage	7.5 K	2009-06-25 - 13:32	DerekFeichtinger
This topic: LCGTier2 > WebHome > CMSInfoPages > CMSSiteLog > CMSSiteLog15
Topic revision: r15 - 2009-09-17 - DerekFeichtinger