Tags:
create new tag
view all tags

Arrow left Go to previous page / next page of CMS site log MOVED TO...

24. 06. 2009 Migration of se05_cms dcache pool files to other pools with the shelltools

Introduction

Note: We do this manually with the shelltools, since the copy cell cannot be trusted in this 1.8-x version of dcache, and we will only update dcache to the 1.9-x versions in July.

This uses my DcacheShellutils tools.

All of these steps are done in the directory /home/staff/dfeich/shellutils/work/se05_cms-migration

Check that se05_cms is read only, so that we can prevent new files appearing on it:

dc_get_pool_list.sh -r |grep se05_cms
se05_cms  (enabled=true;active=12;rdOnly=true;links=0;pgroups=1;hsm=[];mode=enabled)

se05 is one of the older pools from a time that predates space management. At that time we only had precious and cached files (cached meaning that dcache was allowed to remove these copies when it needed space). Later we got space management, and it seems that space managed files get the sticky characteristic. This can be nicely seen from the dCache web GUI's pool usage page, where all these categories get different color bars. The old pools all still have some precious files (red), while the new ones only have sticky (pink) files. There are almost no cached files (green), because cached replicas are only generated by dcache, if there is a underlying HSM behind it (... I, think).

Let's get lists of all files on the pool, and of oll files that are precious and sticky. Note that this differentiation is done by the se05_cms pool cell's rep ls command! So the correctness depends on the correctness of that command!

dc_get_rep_ls.sh se05_cms > se05_cms-ID.lst

dc_get_rep_ls.sh -l p se05_cms > se05_cms-PRECIOUS-ID.lst

dc_get_rep_ls.sh -l s se05_cms > se05_cms-STICKY-ID.lst

dc_get_rep_ls.sh -c se05_cms > se05_cms-CACHED-ID.lst  # careful. just use "-c", not "-l c"

Let's count the entries

wc -l se05_cms-ID.lst se05_cms-PRECIOUS-ID.lst se05_cms-STICKY-ID.lst
   5526 se05_cms-ID.lst
  1267 se05_cms-PRECIOUS-ID.lst
  4546 se05_cms-STICKY-ID.lst

So, 1267 + 4546 = 5813. Seems that some sticky files are also precious at the same time. This may derive from the fact that an earlier migration, shortly after the space management was introduces, manually marked some files as precious...

Finding out about sticky, cached and precious files, and how to correctly create new copies of files

I first need to find out how I can reliably differentiate between sticky, cached and precious files using the admin shell, and how copies with the right properties can be created. For this, I'll do a little research with the shelltools.

Let's look at the characteristics of every kind of file....

Precious:

tail -10 se05_cms-PRECIOUS-ID.lst | dc_get_rep_ls.sh -r -i se05_cms | grep -v admin

00020000000000000000E150 <-P---------(0)[0]> 70901468 si={cms:cms}
0002000000000000000543B0 <-P---------(0)[0]> 2463153484 si={cms:cms}
0002000000000000000744A8 <-P---------(0)[0]> 808370799 si={cms:cms}
0002000000000000000F1018 <-P---------(0)[0]> 432776 si={cms:cms}
00020000000000000001C380 <-P---------(0)[0]> 56193849 si={cms:cms}
00020000000000000004DF40 <-P---------(0)[0]> 2476990936 si={cms:cms}
00020000000000000000C498 <-P---------(0)[0]> 69011452 si={cms:cms}
0002000000000000000207D8 <-P---------(0)[0]> 59985791 si={cms:cms}
0002000000000000000EA730 <-P---------(0)[0]> 427697 si={cms:cms}
00020000000000000024E818 <-P---------(0)[0]> 495136770 si={cms:cms}

Sticky:

tail -10 se05_cms-STICKY-ID.lst | dc_get_rep_ls.sh -r -i se05_cms | grep -v admin

000200000000000001B46118 <C-------X--(0)[0]> 1968743916 si={cms:cms}
000200000000000001AD9838 <C-------X--(0)[0]> 1629486174 si={cms:cms}
0002000000000000011D7320 <C-------X--(0)[0]> 167411059 si={cms:cms}
00020000000000000224AC90 <C-------X--(0)[0]> 25856206 si={cms:cms}
000200000000000000E82D50 <C-------X--(0)[0]> 35728076 si={cms:cms}
000200000000000001C0E0D8 <C-------X--(0)[0]> 739802749 si={cms:cms}
0002000000000000023BB850 <C-------X--(0)[0]> 1092259150 si={cms:cms}
000200000000000001AD1D18 <C-------X--(0)[0]> 2172298863 si={cms:cms}
0002000000000000011C3880 <C-------X--(0)[0]> 171887245 si={cms:cms}
000200000000000001BFA3E8 <C-------X--(0)[0]> 498125871 si={cms:cms}

Hmmm.... I noted from pre-space management times, that <C-------X--(0)[0]> marked cached files, so it seems that these flags are not adequate in differentiating between sticky and simply cached files. Bad. Added later: Seems I made a wrong conclusion at that time... look further below. cached files lack the -X- flag!. This makes it easy again

I found in the admin interface of the pool, the rep sticky ls command. Maybe this can be used. Let's test the theory with the dc_generic_cellcommand.sh on some of the precious files.

head -20 se05_cms-PRECIOUS-ID.lst | dc_generic_cellcommand.sh -f -c 'rep sticky ls $n' se05_cms

    dCache Admin (VII) (user=admin)


[storage01.lcg.cscs.ch] (local) admin > cd se05_cms
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000BB020
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000013FE0
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000024CB50
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000018E3B60
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000EC520
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000062BB8
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000078648
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000A6AA8
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000241E0
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000858D0
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000011B148
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000E8730
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000004DEF0
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000018B6FE0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000008F958
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000008E68
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000000F11D8
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000018987B0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001A896A8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000008A98
[storage01.lcg.cscs.ch] (se05_cms) admin > ..
[storage01.lcg.cscs.ch] (local) admin > logoff
dmg.util.CommandExitException: (0) Done

And now the same for the sticky files:

head -20 se05_cms-STICKY-ID.lst | dc_generic_cellcommand.sh -f -c 'rep sticky ls $n' se05_cms

    dCache Admin (VII) (user=admin)


[storage01.lcg.cscs.ch] (local) admin > cd se05_cms
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000137A2C8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001A980A8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000015D2568
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001C29030
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000023B65E0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000002144460
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001B482A0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000121F038
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001C9C7B8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001ABE618
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000000E76DB8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001B09E00
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000013651A8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 0002000000000000018E3B60
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 00020000000000000136ED30
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001C4FD00
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001B51810
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001ECA7B8
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001D2E990
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > rep sticky ls 000200000000000001C221C0
system:-1
[storage01.lcg.cscs.ch] (se05_cms) admin > ..
[storage01.lcg.cscs.ch] (local) admin > logoff
dmg.util.CommandExitException: (0) Done

In chapter 18 of the dcache book (Pinning Files to a Pool), I can see that the characteristic sticky can apply to a single replicate of a file on a pool, or to all copies of a file on all pools. In the first case, this is a attribute of the file instance in the pool, in the other global case, the information is stored within the pnfs data base:

rep set sticky pnfsid on|off      # pool cell command to make file inside this pool sticky

set sticky pnfsid   # global effect. Can be used from the top entry point into
                           # the admin cell (e.g. after doing "cd ..")

Let's test how we can copy a file and make it sticky

srmcp -debug file:///`pwd`/se05_cms-ID.lst srm://storage01.lcg.cscs.ch:8443/srm/managerv1?SFN=//pnfs/lcg.cscs.ch/cms/local_tests/derek-sticky-test

echo /pnfs/lcg.cscs.ch/cms/local_tests/derek-sticky-test | dc_get_ID_from_pnfsnamelist.sh
00020000000000000243CD00 /pnfs/lcg.cscs.ch/cms/local_tests/derek-sticky-test

echo 00020000000000000243CD00 | dc_get_cacheinfo_from_IDlist.sh
00020000000000000243CD00 se19_cms


echo 00020000000000000243CD00 | dc_get_rep_ls.sh -r -i se19_cms | grep -v admin
00020000000000000243CD00 <C-------X--(0)[0]> 138150 si={cms:cms}

echo 00020000000000000243CD00 | dc_ppcopy_files.sh -f se19_cms se18_cms

echo 00020000000000000243CD00 | dc_get_cacheinfo_from_IDlist.sh
00020000000000000243CD00 se18_cms,se19_cms

echo 00020000000000000243CD00 | dc_get_rep_ls.sh -r -i se18_cms | grep -v admin
00020000000000000243CD00 <C----------(0)[0]> 138150 si={cms:cms}

Ok. Let's try to make the new copy on pool se18_cms sticky

echo 00020000000000000243CD00 | dc_generic_cellcommand.sh -f -c 'rep set sticky $n on' se18_cms

And voila! Here we correctly get our sticky file on this pool big grin

echo 00020000000000000243CD00 | dc_get_rep_ls.sh -r -i se18_cms | grep -v admin
00020000000000000243CD00 <C-------X--(0)[0]> 138150 si={cms:cms}

special case of precious AND sticky files on this old pool

Let's find files which are precious and sticky

for n in `cat se05_cms-PRECIOUS-ID.lst`; do grep -q $n se05_cms-STICKY-ID.lst; if test $? -ne 0; then echo $n; fi; done >precious-and-sticky.lst

Now we have a look at the flags for these files that are sticky and precious at the same time.

cat precious-and-sticky.lst | dc_get_rep_ls.sh -i -r  se05_cms | grep -v admin

0002000000000000000BB020 <-P---------(0)[0]> 179755831 si={cms:cms}
000200000000000000013FE0 <-P---------(0)[0]> 1510974283 si={cms:cms}
00020000000000000024CB50 <-P---------(0)[0]> 1756795340 si={cms:cms}
0002000000000000000EC520 <-P---------(0)[0]> 472898 si={cms:cms}
000200000000000000062BB8 <-P---------(0)[0]> 19387237 si={cms:cms}
000200000000000000078648 <-P---------(0)[0]> 262656755 si={cms:cms}
0002000000000000000A6AA8 <-P---------(0)[0]> 134572991 si={cms:cms}
0002000000000000000241E0 <-P---------(0)[0]> 102410208 si={cms:cms}
0002000000000000000858D0 <-P---------(0)[0]> 162611125 si={cms:cms}
00020000000000000011B148 <-P---------(0)[0]> 450574 si={cms:cms}
...

Let's look at files which are only sticky

for n in `cat se05_cms-PRECIOUS-ID.lst`; do grep -q $n se05_cms-STICKY-ID.lst; if test $? -eq 0; then echo $n; fi; done > only-sticky.lst

head -20 only-sticky.lst | dc_get_rep_ls.sh -r -i se05_cms | grep -v admin

0002000000000000018E3B60 <-P------X--(0)[0]> 3336915216 si={cms:cms}
0002000000000000018B6FE0 <-P------X--(0)[0]> 787573248 si={cms:cms}
0002000000000000018987B0 <-P------X--(0)[0]> 3222094048 si={cms:cms}
000200000000000001A896A8 <-P------X--(0)[0]> 674796192 si={cms:cms}
0002000000000000018E77A8 <-P------X--(0)[0]> 1390713984 si={cms:cms}
00020000000000000189EF28 <-P------X--(0)[0]> 810449872 si={cms:cms}
0002000000000000018D23A8 <-P------X--(0)[0]> 137643984 si={cms:cms}
0002000000000000018CC2E8 <-P------X--(0)[0]> 700283680 si={cms:cms}
0002000000000000018D2898 <-P------X--(0)[0]> 111847864 si={cms:cms}

Let's look at files which are only precious

for n in `cat se05_cms-PRECIOUS-ID.lst`; do grep -q $n precious-and-sticky.lst; if test $? -ne 0; then echo $n; fi; done > only-precious.lst

head -20 only-precious.lst | dc_get_rep_ls.sh -r -i se05_cms | grep -v admin
0002000000000000018E3B60 <-P------X--(0)[0]> 3336915216 si={cms:cms}
0002000000000000018B6FE0 <-P------X--(0)[0]> 787573248 si={cms:cms}
0002000000000000018987B0 <-P------X--(0)[0]> 3222094048 si={cms:cms}
000200000000000001A896A8 <-P------X--(0)[0]> 674796192 si={cms:cms}
0002000000000000018E77A8 <-P------X--(0)[0]> 1390713984 si={cms:cms}
00020000000000000189EF28 <-P------X--(0)[0]> 810449872 si={cms:cms}
0002000000000000018D23A8 <-P------X--(0)[0]> 137643984 si={cms:cms}
0002000000000000018CC2E8 <-P------X--(0)[0]> 700283680 si={cms:cms}
0002000000000000018D2898 <-P------X--(0)[0]> 111847864 si={cms:cms}
...

Drats! frown the flag signature is the same as for the only precious files... something is wrong....

Let's do a check:

dc_get_rep_ls.sh -i -l p se05_cms only-precious.lst > op-gaga.lst

wc -l op-gaga.lst only-precious.lst
  317 op-gaga.lst
  317 only-precious.lst

dc_get_rep_ls.sh -i -l s se05_cms only-precious.lst > check.lst
wc -l check.lst
317 check.lst

NEED TO BETTER RESEARCH THIS.....

How to migrate the files in a pool

We get the list of files that we want to transfer

dc_get_rep_ls.sh se05_cms > se05_cms-ID.lst

We split the files into batches of 500 files for the migration

split -d -l 500 se05_cms-ID.lst batch_ID

Test the migration with the first batch

time dc_ppcopy_files.sh se05_cms se15_cms batch_ID00
real    1m52.828s
user    0m0.060s
sys     0m0.140s

We can see that the transfer load on the machines increases immediately

se05_src.png se15_dst.png

The transfers seem to fill one Gb connection. Every thumper has 4 aggregated 1Gb ethernet connections, but it may be that the hash function governing the distribution over the physical interfaces evaluates to the same for all these connections, so we effectively are only using a single one.

The thumper's CPU spends lots of time in system/kernel space, probably because the connections are fighting for the network. Based on our experience in writing to the 48 disks, the problem should not be located in disk I/O.

se15_cpu.png

The number of queued transfers can easily be seen by looking at the queue info page.

The copies finished around 16:30h.

Now we do a sanity check whether all copies are ok:

dc_get_rep_ls.sh -i se15_cms batch_ID00 > batch_ID00.check

wc -l batch_ID00  batch_ID00.check
  500 batch_ID00
  500 batch_ID00.check

We do a little check for the flags (should be cached-only files):

head -10 batch_ID00 | dc_get_rep_ls.sh -r -i se15_cms | grep -v admin

00020000000000000137A2C8 <C----------(0)[0]> 156990918 si={cms:cms}
000200000000000001A980A8 <C----------(0)[0]> 4769081784 si={cms:cms}
0002000000000000015D2568 <C----------(0)[0]> 4162 si={cms:cms}
000200000000000001C29030 <C----------(0)[0]> 22260183 si={cms:cms}
0002000000000000023B65E0 <C----------(0)[0]> 1187411081 si={cms:cms}
000200000000000002144460 <C----------(0)[0]> 2578597264 si={cms:cms}
000200000000000001B482A0 <C----------(0)[0]> 1782946180 si={cms:cms}
00020000000000000121F038 <C----------(0)[0]> 1572443586 si={cms:cms}
000200000000000001C9C7B8 <C----------(0)[0]> 1921029783 si={cms:cms}
0002000000000000000BB020 <C----------(0)[0]> 179755831 si={cms:cms}

We set the files to sticky for that pool

time dc_generic_cellcommand.sh -c 'rep set sticky $n on' se15_cms batch_ID00
real    0m40.966s
user    0m0.050s
sys     0m0.630s

And again, we check the flag values

 head -10 batch_ID00 | dc_get_rep_ls.sh -r -i se15_cms | grep -v admin

00020000000000000137A2C8 <C-------X--(0)[0]> 156990918 si={cms:cms}
000200000000000001A980A8 <C-------X--(0)[0]> 4769081784 si={cms:cms}
0002000000000000015D2568 <C-------X--(0)[0]> 4162 si={cms:cms}
000200000000000001C29030 <C-------X--(0)[0]> 22260183 si={cms:cms}
0002000000000000023B65E0 <C-------X--(0)[0]> 1187411081 si={cms:cms}
000200000000000002144460 <C-------X--(0)[0]> 2578597264 si={cms:cms}
000200000000000001B482A0 <C-------X--(0)[0]> 1782946180 si={cms:cms}
00020000000000000121F038 <C-------X--(0)[0]> 1572443586 si={cms:cms}
000200000000000001C9C7B8 <C-------X--(0)[0]> 1921029783 si={cms:cms}
0002000000000000000BB020 <C-------X--(0)[0]> 179755831 si={cms:cms}

Seems ok big grin

Fotis did the next steps:

time dc_ppcopy_files.sh se05_cms se11_cms batch_ID01
time dc_ppcopy_files.sh se05_cms se12_cms batch_ID02
time dc_ppcopy_files.sh se05_cms se13_cms batch_ID03
time dc_ppcopy_files.sh se05_cms se14_cms batch_ID04

The full list of commands and pool destinations is, or should be:

time ./dc_ppcopy_files.sh se05_cms se15_cms batch_ID00 # executed, sticky'ed & verified
time ./dc_ppcopy_files.sh se05_cms se11_cms batch_ID01 # executed
time ./dc_ppcopy_files.sh se05_cms se12_cms batch_ID02 # executed
time ./dc_ppcopy_files.sh se05_cms se13_cms batch_ID03 # executed
time ./dc_ppcopy_files.sh se05_cms se14_cms batch_ID04 # executed
time ./dc_ppcopy_files.sh se05_cms se15_cms batch_ID05 # executed
time ./dc_ppcopy_files.sh se05_cms se16_cms batch_ID06 # executed
time ./dc_ppcopy_files.sh se05_cms se17_cms batch_ID07 # executed
time ./dc_ppcopy_files.sh se05_cms se18_cms batch_ID08 # executed
time ./dc_ppcopy_files.sh se05_cms se19_cms batch_ID09 # executed
time ./dc_ppcopy_files.sh se05_cms se20_cms batch_ID10 # executed
time ./dc_ppcopy_files.sh se05_cms se20_cms batch_ID11 # executed

Tests whether everything has worked out

Have all files been copied?

Let's do an independent test whether all has worked out correctly. Every file should now have at least two replicas, one one the original pool se05_cms, and one on another pool. So we just get the cacheinfo for every file, and look whether there are files which only have one replicate.
dc_get_cacheinfo_from_IDlist.sh se05_cms-ID.lst > checklist

# In the command's output multiple pools are separated by a colon ",". So we can search lines without colons to find suspicious entries
#   (i.e. files with only one or no phys replicate)
grep -v ',' checklist
0002000000000000019B9C30

# let's investigate this single ID
echo 0002000000000000019B9C30 | dc_get_cacheinfo_from_IDlist.sh
0002000000000000019B9C30

# let's find the filename
echo 0002000000000000019B9C30 | dc_get_pnfsname_from_IDlist.sh
0002000000000000019B9C30 /pnfs/lcg.cscs.ch/cms/local/stiegerb/hfegsignal50/hfegsignal_69.root

Ok. This file is from a local user area. Maybe the user has done something to it. Problem is now that we have an orphaned pnfs entry without a file beneath it. Usually one should ask the user in respect to this. But this is a small matter in comparison to the whole migration that we did big grin

Are all files sticky?

use something like

dc_get_rep_ls.sh -r -i se12_atlas filelist.lst > check_se12_atlas_rep.lst
grep "C--" check_se12_atlas_rep.lst | grep -v '\-X\-'

Example how to transfer back misplaced ATLAS files from CMS pools

For all affected pools, we generate pool file lists and pnfs mappings, e.g.

dc_get_rep_ls.sh se12_cms > se12_cms_id.lst
dc_get_pnfsname_from_IDlist.sh se12_cms_id.lst > se12_cms_pnfs.lst

We filter out files belonging to atlas and generate lists with the misplaced files per pool

for n in *pnfs.lst; do grep "lcg.cscs.ch/atlas" $n > wrongatlas_"${n}"; done

The next steps demonstrate the sequence for a single pool. We transfer the files from the node's CMS pool to the ATLAS pool on the same node.

  1. issue copy commands (careful: target files end up as cached only). This proved successful for these lists of 5000 files!
    time dc_ppcopy_files.sh -f se12_cms se12_atlas wrongatlas_se12_cms_pnfs.lst 
    real    13m23.419s
    
  2. for checking whether all files were duplicated
    time dc_get_cacheinfo_from_IDlist.sh wrongatlas_se12_cms_pnfs.lst > wrongatlas_se12_cms_cacheinfo.lst
    real    8m9.301s
    
    grep -v 'se12_atlas' wrongatlas_se12_cms_cacheinfo.lst
    
    • If there are missing files, again pipe their IDs into a ppcopy command.
  3. for dc_generic_cellcommand we need a list of IDs only
    cut -f1 -d' ' wrongatlas_se12_cms_pnfs.lst > wrongatlas_se12_cms_id.lst
    
  4. set files sticky on new pool
    time dc_generic_cellcommand.sh -f -c 'rep set sticky $n on' se12_atlas wrongatlas_se12_cms_id.lst
    real    9m16.211s
    
  5. for checking whether all files have been set sticky
    time dc_get_rep_ls.sh -r -i se12_atlas wrongatlas_se12_cms_id.lst > check_se12_atlas_rep.lst
    real    8m20.484s
    
    grep "C--" check_se12_atlas_rep.lst | grep -v '\-X\-'
    
  6. If all looks ok, remove the misplaced files from the CMS pool CAREFUL! This is irreversible!!!!
    dc_rep_rm_list.sh -y -f se12_cms wrongatlas_se12_cms_id.lst
    

List of corrected pools

Pool status problems
se12 done
se13 done
se14 done
se15 done
se16 done
se17 done 0003000000000000043E6DE0 0003000000000000043E6DB8 0003000000000000043E7778
se18 done 0003000000000000043E7F38 0003000000000000043E7D70 0003000000000000043E78E0
se19 done 000300000000000003B99948 0003000000000000043E7F10 000300000000000000FD6388 0003000000000000043E7B40
se20 done


Arrow left Go to previous page / next page of CMS site log MOVED TO...

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng se05_src.png r1 manage 6.2 K 2009-06-25 - 13:32 DerekFeichtinger  
PNGpng se15_cpu.png r1 manage 12.0 K 2009-06-25 - 13:39 DerekFeichtinger  
PNGpng se15_dst.png r1 manage 7.5 K 2009-06-25 - 13:32 DerekFeichtinger  
Edit | Attach | Watch | Print version | History: r15 < r14 < r13 < r12 < r11 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r15 - 2009-09-17 - DerekFeichtinger
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback