RE: {bb} The bb_rename error problem

[brent.mccrackin@bell.ca]

Duplicate entries have most definitely been ruled out.

A busy BB Server has been confirmed as part of the problem :-)  However,
with BBGEN everything completes within the BBSLEEP cycle (if it didn't,
then that would be reported in BBOUT as well - which I haven't seen).

BBLOGSTATUS is dynamic, and has been for a very long time.  I forgot to
mention that in my analysis.

You've said that BBVAR/logs/.somehost.test doesn't exist anymore when
the error occurs.

What if BBVAR/logs/.somehost.test doesn't exist *YET* when bb_rename
comes around to try and rename it?  I suspect that with a very busy I/O
load, the write of the incoming status file to disk hasn't happened or
failed in some manner that wasn't caught, which results in the rename
process subsequently failing.  This also results in the related updates
to the hist and histlogs files being missed.

Either the file is in the filesystem cache waiting to be written to
disk, or the file write has been dropped completely.  Sometimes I've
found the related BBVAR/logs/.somehost.test file appearing after the
error is generated, making me think something else might have had one of
the two files in question locked.

Running BBVAR/logs in tmpfs has helped reduce the error occurrence, and
may be the best option to gain better reliability of the status and
availability reports.  A real RAM disk that is guaranteed to stay in
memory is a better option - one which I am looking into.  Faster disks
is another option being looked into for this problem.


Thanks for taking a look into it!

---
Brent B McCrackin
UNIX Systems Specialist - Bell Sympatico
Brent.McCrackin@Bell.ca   PH: 416-353-0692
"Serenity through viciousness."
 


-----Original Message-----
From: owner-bb@bb4.com [mailto:owner-bb@bb4.com] On Behalf Of
Robert-Andre Croteau
Sent: January 28, 2005 11:47 AM
To: bb@bb4.com
Subject: Re: {bb} The bb_rename error problem


brent.mccrackin@bell.ca wrote:
> I have a very large and very busy BB server, and notice a lot of 
> bb_rename errors logged to BBOUT where new status logs are unable to
be 
> renamed, with an errno 2 message.
>  
> Wed Jan 26 08:56:23 2005 bbd bb_rename Could not rename 
> /bbvar/logs/.somehost.http to /bbvar/logs/somehost.http - errno: 2
>  
> I've looked through the list archives, and suggestions have been that 
> this is a result of a host appearing multiple times in the BB-HOSTS
file 
> either on the server or the client.  This is most definitley not the 
> case for me.
>  
> The root cause of the problem is high I/O wait times on the disks
where 
> the logs are stored, with the affected disks running at a fairly 
> constant 80% to 90% busy.  This can be alleviated with faster disks, 
> higher I/O bandwidth, and separating the BBVAR directory across
multiple 
> filesystems.

the error occurs because /bbvar/logs/.somehost.http does not exists
anymore.  The only reason this can happen (from my POV) is that
another status for the same somehost.http was received and is being
process concurrently.

Reasons this can happen ?

1) Same server defined as BBDISPLAY multiple times under different
names/IP addresses in bb-hosts
2) Overloaded BB server.  Very slow to process all incoming status
logs in a timely fashion and actually didn't process that particular
status in a BBSLEEP cycle.
3) Inneficient/buggy bbd ?

Now, so far, we can rule out 1) as you've determined.  That leaves
2) and 3).  3) may be improved with locking mecanisms (remedied ?).
So that leaves 2) for now.  There's a couple of things to check.

Make sure BBLOGSSTATUS=DYNAMIC in bbdef-server.sh (if not bbdef.sh)
on the BBDISPLAY, No need to create static HTML pages for all status.
(but I'm pretty sure, DYNAMIC is being used)

Put *just* bbvar/logs in a swap partition.  that should reduce disk I/O.
bbvar/logs do not need to be permanent, a reboot won't affect much
but a reboot shouldn't occur often anyways.  I've tried it for testing
purposes with over 6000 status logs and it did help a lot (but don't
know for that particular error) in the bb.html/bb2.html page generation.

[snipped other comments which I'll look into]

thanks
-- 
Robert-Andre Croteau    BSD,MOTU
Quest Software, Inc.            Big Brother, Product Architect
http://www.quest.com/                          http://bb4.com/
+1 (514) 238-9374
         Si le bonheur ne s'achete pas alors louez le.

--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=
To unsubscribe from this list, or to subscribe to the bb-digest list
send e-mail to mailto:majordomo@bb4.com with unsubscribe bb -and/or-
subscribe bb-digest in the BODY of the message.


--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=
To unsubscribe from this list, or to subscribe to the bb-digest list
send e-mail to mailto:majordomo@bb4.com with unsubscribe bb -and/or-
subscribe bb-digest in the BODY of the message.