Re: {bb} BB spawns multiple processes (bbd)

[bb@buglecreek.com]

In my case all the processes are still attached, they do not seem to be
zombies.  All new processes that are generated have the same parent. 
Also, looking at i/o stats and load average the numbers all seem to be
normal.  It seems that everytime a client sends a message to bb with the
security file enabled it spawns a new process and the new process does
not go away.  With the security file not implemented bb spawns the new
process and then the new process goes away once the message is handled.
 For a quick test I issued a telnet command from a client machine
(telnet bbserver 1984).  With the security file active I see the new
process spawned as soon as I hit the return key and as stated above it
does not terminate.  This does not happen with the security file not
active. I do not see anything unusual in BBOUT.


On 30 Sep 2005 23:16:50 +0100, "Philip Clark"
<bblist@cannae.uklinux.net> said:
> On Fri, 2005-09-30 at 22:02, Henrik Storner wrote:
> > In <38759BB1FAFDCD4D97E9DC302410AE986B21D9@YBKOPEX01.corp.ybusa.net> "Brookins, Neil (YBUSA-KOP)" <Neil.Brookins@Yellowbook.com> writes:
> > 
> > >I have also been having the problem where the number of bbd processes
> > >grows without bounds. Eventually it hits about 5000 simultaneous
> > >processes and I run out of page file space. I tried increasing the
> > >number of CPUs in my BB server from 2 to 4 and also increasing RAM from
> > >2GB to 4GB but this did not help at all.
> > 
> > >Then, I changed the polling interval from 300 seconds to 840 seconds on
> > >all my non-production BB clients and the problem stopped! It appears to
> > >me that the disk I/O to the file system where bbd stores its logs was
> > >the bottleneck.
> > 
> > bbd forks a copy of itself whenever it receives a new message. If
> > there's a status in a non-green state, it spawns two processes
> > (one for the status message, and one for the potential alert).
> > When it runs the web page update, it spawns a lot of new processes
> > for all of the stuff that mkbb.sh uses. In other words - BB
> > spends a lot of time starting new processes, and writing logs
> > to disk. That's ressource intensive, and it simply doesn't scale
> > well when you have a large installation (meaning >100 hosts, as
> > a rough guesstimate).
> > 
> > There are some things you can do: Set BBLOGSTATUS=DYNAMIC, so it 
> > only saves the raw log file and not the HTML copy; move the logs
> > to a RAM disk; install bbgen to replace the shell scripts.
> > 
> > >I have 646 servers that are being monitored.
> > >There are a total of 4009 tests. That's over 6 tests per machine
> > >average.
> > >If each test needs to run every 5 minutes, then there is only 0.07
> > >seconds to processes each test if we assume one at a time sequentially.
> > >Right now the number of bbd processes on my BB server is between 120 and
> > >433 or so on average depending on how many BB clients are reporting in
> > >at that time.
> > 
> > I have 1957 hosts, 12372 tests, all of them updating every 5 minutes. That's
> > 43 status messages every second. Running stock BB, the load was around 10
> > (and I was losing status updates). BB+bbgen brought the load to 4, but I
> > was still losing some updates. With Hobbit, the load on this box is 0.9
> > (and no updates are lost). It's a single-cpu Sun server.
> 
> I've never had a really large installation to play with, but are
> "run time longer than sleep time" messages in BBOUT not normally
> the best indication of an over-loaded BB server? It's what I
> would expect to see.
> 
> Cheers, Phil.
> 
> 
> 
> -- 
> Halon's Razor: Never attribute to malice that which is adequately
> explained by stupidity.
> 
> --
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=
> To unsubscribe from this list, or to subscribe to the bb-digest list
> send e-mail to mailto:majordomo@bb4.com with unsubscribe bb -and/or-
> subscribe bb-digest in the BODY of the message.
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=
To unsubscribe from this list, or to subscribe to the bb-digest list
send e-mail to mailto:majordomo@bb4.com with unsubscribe bb -and/or-
subscribe bb-digest in the BODY of the message.