Go to previous page / next page of Tier3 site log

14. 02. 2014 Missing Billing DB and Billing logs entries about t3fs14 files

Yesterday I ran https://bitbucket.org/fabio79ch/billinginfo2chimera/src that nicely updated the iatime of 1153009 rows in chimera.t_inodes. Execution was:

chimera=# UPDATE t_inodes
chimera-#        SET iatime = iatimebillinginfo.newer_iatime
chimera-#        FROM (
chimera(#               SELECT   max(billinginfo.datestamp) as newer_iatime,
chimera(#                        pnfsid
chimera(#               FROM     billinginfo
chimera(#               group by pnfsid
chimera(#             ) AS iatimebillinginfo
chimera-#
chimera-#        WHERE t_inodes.ipnfsid = iatimebillinginfo.pnfsid
chimera-#        AND   iatime           < iatimebillinginfo.newer_iatime
chimera-# ;

LOG:  duration: 229333.917 ms  statement: UPDATE t_inodes
        SET iatime = iatimebillinginfo.newer_iatime
        FROM (
               SELECT   max(billinginfo.datestamp) as newer_iatime,
                        pnfsid
               FROM     billinginfo
               group by pnfsid
             ) AS iatimebillinginfo
             
        WHERE t_inodes.ipnfsid = iatimebillinginfo.pnfsid
        AND   iatime           < iatimebillinginfo.newer_iatime
;
UPDATE 1153009

and now I get:
chimera=# select count(*) from t_inodes where iatime > ( ictime + interval '12 hours' )  ;
LOG:  duration: 4846.568 ms  statement: select count(*) from t_inodes where iatime > ( ictime + interval '12 hours' )  ;
  count
--------
  567402 

but

some dcache gridftp file transfers are neither reported in the billing.billinginfo table nor in the billing logs on t3dcachedb ; e.g. despite of t3nagios constantly hammering the t3fs14 pools I see stat cases like:

/pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_7   Access: 2013-08-23 14:19:27.000000000 +0200 

to try a different protocol I copied by dccp the file /pnfs/psi.ch/cms/t3-nagios/1MB-test-file_pool_t3fs14_cms_7 and the entry appeared both in the billing.billinginfo table and in the billing logs.

Because these issues are concentrated just on t3fs14 I would restart dCache there, but I need to find an idle moment.

14-02-2014 15:34
restarted billing service on t3dcachedb, I see a lot of CPU used
9407 postgres  20    0 1192M  801M run   456:54 84.70% 95.38% INSERT INTO billinginfo ("transaction",protocol,transfersize,"action",pnfsid,datestamp,errorcode,p2p,errormessage,fullsize,connectiontime,cellname,client,initiator,storageclass,isnew) VALUES

---

Go to previous page / next page of Tier3 site log