Re: {bb} BB spawns multiple processes (bbd)

[Philip Clark <bblist@cannae.uklinux.net>]

On Fri, 2005-09-30 at 22:02, Henrik Storner wrote:
> In <38759BB1FAFDCD4D97E9DC302410AE986B21D9@YBKOPEX01.corp.ybusa.net> "Brookins, Neil (YBUSA-KOP)" <Neil.Brookins@Yellowbook.com> writes:
> 
> >I have also been having the problem where the number of bbd processes
> >grows without bounds. Eventually it hits about 5000 simultaneous
> >processes and I run out of page file space. I tried increasing the
> >number of CPUs in my BB server from 2 to 4 and also increasing RAM from
> >2GB to 4GB but this did not help at all.
> 
> >Then, I changed the polling interval from 300 seconds to 840 seconds on
> >all my non-production BB clients and the problem stopped! It appears to
> >me that the disk I/O to the file system where bbd stores its logs was
> >the bottleneck.
> 
> bbd forks a copy of itself whenever it receives a new message. If
> there's a status in a non-green state, it spawns two processes
> (one for the status message, and one for the potential alert).
> When it runs the web page update, it spawns a lot of new processes
> for all of the stuff that mkbb.sh uses. In other words - BB
> spends a lot of time starting new processes, and writing logs
> to disk. That's ressource intensive, and it simply doesn't scale
> well when you have a large installation (meaning >100 hosts, as
> a rough guesstimate).
> 
> There are some things you can do: Set BBLOGSTATUS=DYNAMIC, so it 
> only saves the raw log file and not the HTML copy; move the logs
> to a RAM disk; install bbgen to replace the shell scripts.
> 
> >I have 646 servers that are being monitored.
> >There are a total of 4009 tests. That's over 6 tests per machine
> >average.
> >If each test needs to run every 5 minutes, then there is only 0.07
> >seconds to processes each test if we assume one at a time sequentially.
> >Right now the number of bbd processes on my BB server is between 120 and
> >433 or so on average depending on how many BB clients are reporting in
> >at that time.
> 
> I have 1957 hosts, 12372 tests, all of them updating every 5 minutes. That's
> 43 status messages every second. Running stock BB, the load was around 10
> (and I was losing status updates). BB+bbgen brought the load to 4, but I
> was still losing some updates. With Hobbit, the load on this box is 0.9
> (and no updates are lost). It's a single-cpu Sun server.

I've never had a really large installation to play with, but are
"run time longer than sleep time" messages in BBOUT not normally
the best indication of an over-loaded BB server? It's what I
would expect to see.

Cheers, Phil.



-- 
Halon's Razor: Never attribute to malice that which is adequately
explained by stupidity.

--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=
To unsubscribe from this list, or to subscribe to the bb-digest list
send e-mail to mailto:majordomo@bb4.com with unsubscribe bb -and/or-
subscribe bb-digest in the BODY of the message.