Thursday, March 19, 2009

A (long) day in the life of a sysadmin....

Now sit right back and you'll hear a tale...

[digression]
Isn't it criminal that Baywatch is the first link in Google on that phrase?
Goes to demonstrate that they just aren't that great at search, just better
than the alternatives!
[end digression]


I was looking at my daily logs. You do that too right? It's very helpful to keep you ahead of the game and knowing what's going on rather than playing catch up all the time. On the other hand it can be pretty darned tedious. I use logwatch of course to make it bearable.

Lately I've been rolling out Ubuntu Hardy desktop boxes. I remarked to myself that since I'm monitoring the drive space using Nagios, the disk space report on every log file was filler I really didn't need to be looking at. And hey, I like fortune as much as the next guy, but once you've seen a dozen or so you really don't need anymore for the day. Why not turn that stuff off?

Now I knew that logwatch configuration seems to be a bit, well, baroque,

but hey, I can handle it right?
[that's called foreshadowing]

Heck - I'd even seen an article about managing your log files that mentioned logwatch on my RSS reader. Let's do this!

[The road to hell...]


==> man logwatch
==> view /usr/share/doc/logwatch/README
==> view /usr/share/doc/logwatch/HOWTO-Customize-LogWatch

OK, the default files are in /usr/share/logwatch, and the system local ones go in /etc/logwatch.
Let's go look.
==> cd /etc/logwatch
poke around
It's empty. Five folders, no conf files. Folders with .conf in their name though! Weird, but true. Lets go check out /usr/share/logwatch
[poke around]
Ok, the stuff is all here - and reading the docs at /usr/share/doc/logwatch tells me that these are the defaults, and stuff put in /etc overrides them. Since logwatch is used on more than just Linux, it seems a bit different, but obviously flexible. Let's peruse the logwatch.conf file - it's well commented.
# You can also disable certain services (when specifying all)
Service = "-zz-network" # Prevents execution of zz-network service, which
# prints useful network configuration info.
Service = "-zz-sys" # Prevents execution of zz-sys service, which
# prints useful system configuration info.

Look - they've disabled two services 'zz-network' and 'zz-sys' by default. I wonder what they do?
[Cue the music - the abyss opens]

Well, it says they are useful - lets use them!
A nifty feature of logwatch is you can just keep running it over and over again from the commandline with different options and see what the output will look like. Once you get it tuned up, you adjust the conf file to match and cron takes it from there.
==> sudo logwatch --print
#### Logwatch 7.3.6 (05/19/07) ##
Processing Initiated: Thu Mar 19 14:52:42 2009

[snip 172 lines of detail I see waay too often as it is....]

==> sudo logwatch --print --service 'zz-network'

[snip interesting network info]

OK, so that one's interesting. What about zz-sys?
==> sudo logwatch --print --service 'zz-sys'
---- System Configuration Begin ---
No Sys::CPU module installed.
To install, execute the command:
perl -MCPAN -e 'install Sys::CPU'
No Sys::MemInfo module installed.
To install, execute the command:
perl -MCPAN -e 'install Sys::MemInfo'

Huh, missing modules. Perhaps that's why it's disabled. Let's just make sure
==> cd /usr/share/logwatch/scripts/services
==> ./zz-sys
No Sys::CPU module installed.
No Sys::MemInfo module installed.

OK. Let's install those missing modules. Now, I don't have anything against CPAN, but since these are a bunch of pretty much identical machines that need to be kept that way, keeping to the repositories, or at least .deb files is definately the right way to go.
==> sudo aptitude search meminfo
p python-meminfo-total

Close, but no cigar. Try again.
==> sudo aptitude search perl

[snip snip snip]

No! - don't do that! 1500 lines of stuff...
==> sudo aptitude search perl | grep mem

[snip 8 lines - nothing we want]

OK. So we need to make our own debs. This Debian Admin article is perfect.

Step one - Head over to CPAN and download the source. Their seach box makes it easy to find the links.
==> wget http://search.cpan.org/CPAN/authors/id/B/BU/BURAK/Sys-Info-0.69_07.tar.gz
==> wget http://search.cpan.org/CPAN/authors/id/S/SC/SCRESTO/Sys-MemInfo-0.91.tar.gz

Step two - install dh-make-perl
==> sudo aptitude install dh-make-perl
Need to get 5813kB of archives. After unpacking 21.3MB will be used.
Do you want to continue? [Y/n/?] y

Wow - 32 packages of dependencies! Well, that's what apt is for right? Go!
[hundreds of lines scroll by while I go grab coffee. That's a mistake...]

Building tag database... Done

OK - next step
==> tar xvzf Sys-Info-0.69_07.tar.gz
==> tar xvzf Sys-MemInfo-0.91.tar.gz
==> dh-make-perl Sys-Info-0.69_07
Searching for Sys::Info::Driver::OSID package using apt-file.
E: The cache directory is empty. You need to run 'apt-file update' first.
Searching for Sys::Info::Base package using apt-file.
E: The cache directory is empty. You need to run 'apt-file update' first.
Needs the following modules for which there are no debian packages available
- Sys::Info::Driver::OSID
- Sys::Info::Base

Now I was just plain dumb. I missed the point of the apt-file update message altogether by reading it as apt-get update.

OK, so I need a couple of dependencies too. Man, would be nice if these were in the repositories.
[Have you hugged a packager today? If not, did you buy one beer? or at least say thanks? Just wondering.....]

==> wget http://search.cpan.org/CPAN/authors/id/B/BU/BURAK/Sys-Info-Base-0.69_06.tar.gz
==> tar xvzf Sys-Info-Base-0.69_06.tar.gz
==> dh-make-perl Sys-Info-Base-0.69_06
Done

OK, now we're cooking.
==> wget http://search.cpan.org/CPAN/authors/id/B/BU/BURAK/Sys-Info-Driver-Linux-0.69_06.tar.gz
==> tar xvzf Sys-Info-Driver-Linux-0.69_06.tar.gz
==> dh-make-perl Sys-Info-Driver-Linux-0.69_06

[snip - still ignoring the message]

Needs the following modules for which there are no debian packages available
- Unix::Processors
- Sys::Info::Base
- Linux::Distribution

Grrr..... Sigh.
==> wget http://search.cpan.org/CPAN/authors/id/W/WS/WSNYDER/Unix-Processors-2.040.tgz
==> tar xvzf Unix-Processors-2.040.tgz
==> dh-make-perl Unix-Processors-2.040

[snip including copyright warning. Glad I'm not a packager that has to worry about these things for everybody else]

Done
==> wget http://search.cpan.org/CPAN/authors/id/K/KE/KERBERUS/Linux-Distribution-0.14.tar.gz
==> tar xvzf Linux-Distribution-0.14.tar.gz
==> dh-make-perl Linux-Distribution-0.14
Done

OK, now to make a deb!
==> cd Linux-Distribution-0.14
==> debuild
The program 'debuild' is currently not installed. You can install it by typing:
sudo apt-get install devscripts
-bash: debuild: command not found
==> sudo aptitude install debuild

[Yep - stupid again. Dunno what the heck I was thinking....]

Couldn't find package "debuild". However, the following packages contain "debuild" in their name:
pdebuild
==> sudo aptitude install pdebuild
Need to get 150MB of archives. After unpacking 312MB will be used.

Yikes - hundreds of depencies and many megabytes of Java looking stuff! Wha?
==> debuild
The program 'debuild' is currently not installed. You can install it by typing:
sudo apt-get install devscripts
-bash: debuild: command not found

[face-palm]

==> sudo aptitude install devscripts
Building tag database... Done
==> debuild
This package has a Debian revision number but there does not seem to be an appropriate original tar file or .orig directory in the parent directory;
(expected liblinux-distribution-perl_0.14.orig.tar.gz or Linux-Distribution-0.14.orig)
continue anyway? (y/n) y

[snippage]
gpg: [stdin]: clearsign failed: secret key not available
debsign: gpg error occurred! Aborting....
debuild: fatal error at line 1174:
running debsign failed

I don't need it signed. How do I disable that?
==> man debuild

Hey - right here in the examples it talks about binary only. Bet it needs the key to sign the code right?
==> debuild -i -us -uc -b
dpkg-deb: building package `liblinux-distribution-perl' in `../liblinux-distribution-perl_0.14-1_all.deb'.

Yea! A deb! Finally. Now to do the rest of 'em!
==> cd Unix-Processors-2.040/
==> debuild -i -us -uc -b
dpkg-deb: building package `libunix-processors-perl' in `../libunix-processors-perl_2.040-1_i386.deb'.
==> cd Sys-Info-Driver-Linux-0.69_06/
==> debuild
- ERROR: Test::Sys::Info is not installed
- ERROR: Unix::Processors is not installed
- ERROR: Linux::Distribution is not installed
- ERROR: Sys::Info::Base is not installed

OK, gotta install these in the right order too!
==> sudo dpkg --install lib*deb
Setting up liblinux-distribution-perl (0.14-1) ...
Setting up libunix-processors-perl (2.040-1) ...
==> dh-make-perl Sys-Info-Driver-Linux-0.69_06
E: The cache directory is empty. You need to run 'apt-file update' first.
The directory Sys-Info-Driver-Linux-0.69_06/debian is already present and I won't overwrite it: remove it yourself.

Now I read the darned message....
==> rm -rf Sys-Info-Driver-Linux-0.69_06/debian/
==> sudo apt-file update
Can't get ftp://ftp.mondorescue.org/ubuntu/dists/8.04/Contents-i386.gz

Huh? Oh, ya, that's an extra repository, won't matter for this.
==> dh-make-perl Sys-Info-Driver-Linux-0.69_06
Needs the following modules for which there are no debian packages available
- Unix::Processors
- Sys::Info::Base
- Linux::Distribution
==> cd Sys-Info-Driver-Linux-0.69_06/
==> debuild -i -us -uc -b
- ERROR: Test::Sys::Info is not installed
==> wget http://search.cpan.org/CPAN/authors/id/B/BU/BURAK/Test-Sys-Info-0.13.tar.gz
==> tar xvzf Test-Sys-Info-0.13.tar.gz
==> dh-make-perl Test-Sys-Info-0.13
Done
==> cd Test-Sys-Info-0.13/
==> debuild
gpg: [stdin]: clearsign failed: secret key not available

Bah - idiot!
==> debuild -i -us -uc -b
dpkg-deb: building package `libtest-sys-info-perl' in `../libtest-sys-info-perl_0.13-1_all.deb'.
==> sudo dpkg --install libtest-sys-info-perl_0.13-1_all.deb
==> cd Sys-Info-Driver-Linux-0.69_06/
==> debuild -i -us -uc -b
pkg-deb: building package `libsys-info-driver-linux-perl' in `../libsys-info-driver-linux-perl_0.69-06-1_all.deb'.
==> sudo dpkg --install libsys-info-driver-linux-perl_0.69-06-1_all.deb
==> dh-make-perl Sys-MemInfo
Cannot find a description for the package: use the --desc switch

Hey, new errors. What fun! At least it's a helpful errror. Let's try...
==> dh-make-perl --desc Sys-MemInfo Sys-MemInfo
Done
==> cd Sys-MemInfo/
==> debuild -i -us -uc -b
dpkg-deb: building package `libsys-meminfo-perl' in `../libsys-meminfo-perl_0.91-1_i386.deb'.
==> sudo dpkg --install libsys-meminfo-perl_0.91-1_i386.deb
Setting up libsys-meminfo-perl (0.91-1) ...

OK - this is IT. The moment of truth! Drumroll please!
==> ./zz-sys
No Sys::CPU module installed. To install, execute the command:
perl -MCPAN -e 'install Sys::CPU'
Memory: 495 MB
Machine: i686
Release: Linux 2.6.24-22-generic

Long pause. Quiet wimpering....
==> sudo aptitude search perl | grep cpu
p libsys-cpu-perl - Sys::CPU Perl module for getting CPU infor

Yes Virginia, there is a Santa Claus.

==> sudo aptitude install libsys-cpu-perl
Building tag database... Done


==> ./zz-sys
CPU: 1 Intel(R) Pentium(R) 4 CPU 2.40GHz at 2392MHz
Memory: 495 MB
Machine: i686
Release: Linux 2.6.24-22-generic

Four lines of output.
[And I still haven't gotten it into the logwatch.conf]

Somedays you're the windshield............