Monitoring LINUX server's [Archive]

me1960

26-05-09, 14:29

CCcam Monitoring / Firewalling - Protect your cccam server

'' .....Features :
- block peers based on a specific number of bad commands or slow "cards" commands, for any number of minutes.
- block peers doing "illegal logins" (no account or bad password) for any number of minutes.
- protect cccam accounts or ips from being blocked or give them a different error count.
- filter main log and keep only relevant information (reduce log size) and split logs (monitoring, ecm, logins).
- modify monitoring behaviour, turn on/off firewall blocking or logs without having to restart cccam.
- iptables rules use user defined chains in order to keep your iptables firewall fast, clean and safe.

Prerequires :
- cccam running in debug mode with timestamps (-d -t)
- iptables (optional - can be disabled to use only logs)
- sudo to access iptables for your cccam user, with or without password (optional - sudo can be omitted or use of ssh to manage an external firewall)
- cron to perform iptables flush rules (unban) (optional if iptables is not used)
- bash/sh/ash(/busybox) POSIX shell script (read the dreambox section for more info on the scripting problems with ash/busybox shells) and some basic shell commands cat, echo, cut, grep..

Install :
There's not much to do if you know that your system matches the prerequires above.

Open a command line window and make sure you are running your cccam user.
Put the 4 scripts, config.sh fork.sh monitor.sh iptables.sh, in your cccam directory along with the two "protected_*" files if you need them.

Make sure the scripts can run on your shell, to do that first find out which shell you are running, there's a great change your linux system has links to simulate the precense of sh or bash but you can check your installed shells with the "whereis" command ("#whereis sh", "#whereis ash"...).
The scripts should be POSIX compatible to run on most shells and are configured to run on ash which is supposed to be faster than sh/bash, with the exception of "fork.sh" which MUST run on sh/bash or any shell that will release terminated child scripts and won't generate zombie processes (defunct).
If you don't want to install or use "ash" you can edit each script and change their top line :
Code:
#!/bin/ash

with something matching your shell, like :
Code:
#!/bin/bash

Create two subdirectories to put logs and work files, default directories are "logs" and "monitor", path are defined in config.sh and can be set to whatever you'd like :
Code:
cd /path/to/cccam
mkdir monitor
mkdir logs
chmod 0775 logs monitor

Make sure the scripts are executable :
Code:
chmod +x config.sh fork.sh monitor.sh iptables.sh

You need to edit the config.sh file to match your setup, each config line has comments.
Make sure you get the iptables parameters right if you intend to use it.

You can test your iptables setup by running the init command :
Code:
./iptables.sh init

iptables init is always run when the fork.sh script starts so you don't actually need to run the init command manually but if your get some errors you will have to fix them.
If you run the init multiple times you will have two errors "iptables: Chain already exists", this is not a problem.

Errors are redirected to you main log file (default "logs/monitor.log") so read it to see if errors appear.

Make sure cccam output looks like this :
Code:
12:15:21.499 CCcam: deleting client xx.xx.xxx.xxx(userX), bad command
12:15:21.501 CCcam: deleting client yy.yyy.yyy.yy(userY), read result -1
12:15:27.896 CCcam: 1498 cards --> client yy.yyy.yyy.yy(userY) (took 0.2730 seconds)

you can test that by running cccam in the foreground with :
Code:
./CCcam.x86 -d -t -C ./CCcam.cfg

according to cccam --help, the only thing that could be missing is the timestamp. Although i think that -d activates debug mode, not only foreground, and then outputs everything including timpestamps, also overriding cccam config options like DEBUG..

Finally when you're all set you can run cccam with the following command from the cccam directory :
Code:
./CCcam.x86 -d -t -C ./CCcam.cfg | ./fork.sh >> /path/to/cccam/logs/monitor.log &

It will put the cccam process in background. You can use full paths, the script know where it's located.

You can then check out the main log with :
Code:
tail -n30 -f logs/monitor.log

if everything works fine and you are using the firewall, you have to add a cron command to flush rules when they expire.
Code:
*/1 * * * * /path/to/cccam/iptables.sh flush >> /path/to/cccam/logs/monitor.log 2>&1

It will run the iptables flush script every minute and delete rules that have expired (exceeded the ban duration)

Usage :
- BADCOUNT/BADCOUNT_IG options : Knowing that any peer can generate a bad command it is probably not a good idea to ban them at the first one. A good peer will most likely succeed to connect on its second attempt.

- BADDELAY/BAN options : bad commands being mostly caused by networking problems (probably very slow lines) it can be something temporary. That's why a 120min ban is usually enough to let the peer try again at a later time when the network conditions have changed. Using a short ban duration like 30min or less will simply let the bad peer send more commands and cripple your cccam server. If you let a peer send more than one bad command then the error count spreads accross the BADDELAY duration.

- BANLOGIN option : banning illegal logins is optional but after exchanging Clines with many peers you will find out that a bunch of them won't remove your Cline, getting your cccam server to handle a large amount of login attempts that are all doomed to failure. I just thought that filtering that traffic wouldn't hurt (read bugs and limitations below). It should also protect your server from brute force login attacks although i doubt anyone would try to do such a visible hack.

- CARDSDELAY/CARDSBAD options : that one is a bit more tricky, it tells the monitoring script to interpret "xxxx cards --> client" commands as a bad command based on its duration. I have indeed noticed that peers taking very long time (a few seconds) to receive the cards info will disconnect active peers in the exact same way a bad command does ! Unfortunately i wasn't really able to identify a very precise duration above which diconnections happen so i used a two levels trigger to manage those commands.
CARDSDELAY is the max durations in seconds above which the peer will get a "bad command" added to its log, but it will never trigger a ban action, the trigger will only log the event and when a "real" bad command occurs at a later time then it will count as a bad command. The use of this option is quite restrictive, the default is 4 seconds.
CARDSBAD is the max number of seconds allowed for a "xxxx cards --> client" command, above which a ban action is triggered, as if a real bad command was seen. CARDSBAD is probably more important than CARDSDELAY because it will actively try to ban peers with a very slow network, it will act in the exact same way as a bad command action does, respecting the same ban trigger options (BADCOUNT/BADDELAY). CARDSBAD should always be greater than CARDSDELAY.

- Understand that when a peer is banned, he will see your server as offline but you will still be able to connect to his, you know what that means.

Files :
config.sh : config.sh (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Fpastebin.php%3Fdl%3Df1d9f02db) - preview (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Ff1d9f02db)
fork.sh : fork.sh (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Fpastebin.php%3Fdl%3Df24ea2950) - preview (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Ff24ea2950)
iptables.sh : iptables.sh (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Fpastebin.php%3Fdl%3Df35dc81d7) - preview (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Ff35dc81d7)
monitor.sh : monitor.sh (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Fpastebin.php%3Fdl%3Df571e4013) - preview (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Ff571e4013)
protected_account : protected_account (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Fpastebin.php%3Fdl%3Df174babc2) - preview (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Ff174babc2)
protected_ip : protected_ip (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Fpastebin.php%3Fdl%3Df71d2ec71) - preview (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fpastebin.com%2Ff71d2ec71)

Advanced :
- you can rotate logs, all redirects use "append" (>>).

- no sudo password
you can allow access to iptables for your cccam user (if not root) without sudo password by adding the following line to your "/etc/sudoers" file.
Code:
cccamuser ALL=NOPASSWD:/sbin/iptables

That way you don't have to give a large sudo access to your cccam user and you don't have to put a sudo password in config.sh SUPASS option. The cccam user only requires sudo (root) access to iptables, nothing else.

- using a remote firewall
To use a remote iptables firewall on your local network you will have to replace the sudo command with a ssh one. ssh will try to connect to a remote linux system and execute the iptables command there, then return the command output to the script making the iptables command run exactly as if it was executed localy.
I will guide you quickly through a basic ssh setup but if you don't know what you are doing i'd suggest googling some SSH HOWTOs (http://www.eurocardsharing.com/redirect-to/?redirect=http%3A%2F%2Fwww.puddingonline.com%2F%7Edave%2Fpublications%2FSSH-with-Keys-HOWTO%2Fdocument%2Fhtml%2FSSH-with-Keys-HOWTO-4.html).

A proper SSH setup would be as follows :
As root, generate a key pair using your "cccam" system's (where cccam monitoring script runs) ssh key generator (ssh-keygen on debian), if any, with the following command :
Code:
ssh-keygen -t dsa

When asked for a "passphrase", we won't enter one. Just press enter twice.
That should produce two files in your root home directory : "~/.ssh/id_dsa.pub" and "~/.ssh/id_dsa"

Copy the one line content of the new "~/.ssh/id_dsa.pub" file to the following file on your remote server root user directory : /root/.ssh/authorized_keys2
Depending on your remote server ssh configuration, the file could also be : /root/.ssh/authorized_keys

Now from the cccam system you should be able to ssh your remote server with the command :
Code:
ssh root@remoteip

If it asks you to keep a footprint of the new remote system, say yes and it shouldn't bother you with that later, you can "exit" ssh now.
If it worked you can also try accessing the remote iptables firewall with that command, from the cccam system :
Code:
ssh root@remoteip /usr/bin/iptables -L -n

It should display your firewall configuration for INPUT OUTPUT FORWARD chains.

Then set the SUDO option in config.sh, like this :
Code:
SUDO="/usr/bin/ssh -x root@remoteip"

you can also empty the SUPASS option.

in order to make sure you have a working remote iptables setup you can try running the iptables.sh init command from the cccam system, as follows :
Code:
./iptables.sh init

If you run it twice it should display two errors : "iptables: Chain already exists"
That means the cccam chains have been created on your firewall and your setup works.

You can now run cccam as described in the main install guide above.

Known bugs and limitations

- log file output lines may be missplaced, some commands take longer to execute than others and the command output then gets into the log file a bit late, but this is usually something very localized and every line having a timestamp you can always understand what happened.

- some iptables rules may be insterted more than once when a event is not processed fast enough, but it doesn't matter because iptables won't mind and the flush script makes sure duplicate rules are removed.

- reloading your firewall should delete cccam init. you will have to init cccam iptables manually to fix that, with "./iptables.sh init". The good thing is that it should not get the scripts or cccam to crash, so cccam will not really care but you will get lots of errors in your cccam monitoring log..

- banning illegal logins will most certainly ban the new peers you have exchanged Clines with if you haven't activated their accounts before sending them the Clines. You will then have to remove the ban manually or wait for the iptables flush to purge it for you if you haven't banned illegal logins permanently, otherwise new peers may never be able to connect !

Dreambox

Although the scripts should run (with ssh and remote firewall) on dreambox busybox default shell (ash), as they do on my debian stable busybox (1.1.3) binary, the problem i have with that shell, or with ash (dash on debian), is that a parent script will not "release" child scripts that have ended and will then generate tons of zombie processes also known as defunct processes.

If you know a way around that busybox ash shell problem, please contact me, it will most certainly be very helpfull to dreambox users.

I must stress out the fact that running the scripts as they are on the dreambox will work for a short period of time and then stop working when the large number of defunct processes will have filled the system's proc table. It will also most certainly make the dreambox system pretty unstable or even make it crash.. ...... ''