Re: {bb} Cluster-aware processes?

[Philip Clark <bblist@cannae.uklinux.net>]

On Thu, 2007-04-12 at 18:18 +0200, Kit Gerrits wrote:
> Hello all!
> 
> I'm currently using BB 1.9 to monitor a fair number of servers.
> Some of these servers happen form a cluster, where processes move around a
> lot.
> If a cluster application switches to another node, 
>   the corresponding processes are no longer running on the default host, 
>   which causes BB to issue a warning.
> 
> 
> I'd like to have BB recognise the fact that the cluster has switched and
> that the processes are no longer required to run on that specific system.
> For instance: systems foo1 and foo2 host package foo and systems bar1 and
> bar2 host package bar.
> (We also have a cluster where hosts foo1, foo2 and foo3 host applications
> foo and bar on separate hosts)
> 
> 
> I have found 2 ways of accomplishing this:
> A/ changing the BB configuration with each cluster switch
> 	This would involve replacing the configuration files on the cluster
> node
> 	and HUP'ing / restarting the BB process
> I would prefer not to do this, because 
>   A) this is a HP-UX system and I'm not very familiar with MC/ServiceGuard
> and 
>   B) I don't trust the software supplier to keep his hands off the cluster
> configuration script, undo-ing my bb hook
>   C) Messing with the cluster configuration may void any warranty on the
> system.
> 
> B/ use the hostname variable in the configuration file.
> 	The cluster package associates with a specific address which can be
> found in the 'hosts' file.
> 	If the cluster package 'foo' is active on that node, the IP
> associated with hostname 'foo' is also active on that host.
> Is is possible to hook processes in the bb-proc file into the 'foo'
> hostname?
> Would BB notice the fact that the IP associated with that hostname has been
> (de)activated 
>   and (de)activate monitoring of the related process?
> 

A couple of other options occur to me, but they might very well
be less workable than those you have already considered. They are
not based on experience, just "blue sky" musing.

C/  Modify the client "procs" test to return "clear" status rather
    than red if the cluster process(es) are not running. Then craft
    an ext script on the server to monitor the aggregate status and
    generate the appropriate status.

D/  If there is a floating IP or hostname associated with each
    process and the BB server can refer to it, you might be able
    to monitor the process with something like bb-central. That
    way, the test should automatically follow the process around.

>From what you have described, however, I agree with your sentiments.
It is preferable to try and keep the "am I active" logic external to
the application. If there is no alternative, I normally try to
de-couple things by getting the app to create and remove a simple
"trigger" file somewhere, and then modify the client to change
behaviour based on that. Instead of swapping configuration files
around, I would much prefer to build the logic directly into the
modified client.

If possible, I would try to use something like ifconfig to check
that the floating IP is configured on the machine. Then include the
cluster process tests only when true.

Cheers, Phil.



--
Dawn, n. The time when men of reason go to bed. (Ambrose Bierce)


--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-==-=-=-=-=-=-=-=-=-=-=-=
To unsubscribe from this list, or to subscribe to the bb-digest list
send e-mail to mailto:majordomo@bb4.com with unsubscribe bb -and/or-
subscribe bb-digest in the BODY of the message.