Re: {bb} BB spawns multiple processes (bbd)

[Henrik Storner <henrik-bb@hswn.dk>]

In <38759BB1FAFDCD4D97E9DC302410AE986B21D9@YBKOPEX01.corp.ybusa.net> "Brookins, Neil (YBUSA-KOP)" <Neil.Brookins@Yellowbook.com> writes:

>I have also been having the problem where the number of bbd processes
>grows without bounds. Eventually it hits about 5000 simultaneous
>processes and I run out of page file space. I tried increasing the
>number of CPUs in my BB server from 2 to 4 and also increasing RAM from
>2GB to 4GB but this did not help at all.

>Then, I changed the polling interval from 300 seconds to 840 seconds on
>all my non-production BB clients and the problem stopped! It appears to
>me that the disk I/O to the file system where bbd stores its logs was
>the bottleneck.

bbd forks a copy of itself whenever it receives a new message. If
there's a status in a non-green state, it spawns two processes
(one for the status message, and one for the potential alert).
When it runs the web page update, it spawns a lot of new processes
for all of the stuff that mkbb.sh uses. In other words - BB
spends a lot of time starting new processes, and writing logs
to disk. That's ressource intensive, and it simply doesn't scale
well when you have a large installation (meaning >100 hosts, as
a rough guesstimate).

There are some things you can do: Set BBLOGSTATUS=DYNAMIC, so it 
only saves the raw log file and not the HTML copy; move the logs
to a RAM disk; install bbgen to replace the shell scripts.

>I have 646 servers that are being monitored.
>There are a total of 4009 tests. That's over 6 tests per machine
>average.
>If each test needs to run every 5 minutes, then there is only 0.07
>seconds to processes each test if we assume one at a time sequentially.
>Right now the number of bbd processes on my BB server is between 120 and
>433 or so on average depending on how many BB clients are reporting in
>at that time.

I have 1957 hosts, 12372 tests, all of them updating every 5 minutes. That's
43 status messages every second. Running stock BB, the load was around 10
(and I was losing status updates). BB+bbgen brought the load to 4, but I
was still losing some updates. With Hobbit, the load on this box is 0.9
(and no updates are lost). It's a single-cpu Sun server.


Henrik

--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=
To unsubscribe from this list, or to subscribe to the bb-digest list
send e-mail to mailto:majordomo@bb4.com with unsubscribe bb -and/or-
subscribe bb-digest in the BODY of the message.