System Admin

From Eugene Maker Space
Jump to: navigation, search

System Admin Notes

Basic Packages:

Here are some basic packages that we'll probably need.

apt-get install debian-archive-keyring less tcsh bzip2 zip unzip rsync rdate dnsutils whois pciutils aptitude<br />apt-get install sudo vim ssh telnet ftp

Apache:

Mostly we have to copy over the config, and temporary update it with the test domain name. The test domain name shouldn't appear anywhere else in any config file except where it's used as the actual domain name. If you have any filenames with this string in them, check that references to them don't appear in any of the config files because we'd rather not have to rename them too.

apt-get -t testing install apache2 php5 libapache2-mod-fcgid<br />apt-get install javascript-common<br />rsync -Pav "eugenemakerspace.com:/etc/apache2" /etc

Here is where we update the domain names in the config files to work with the test domain. The pearl script has some extra possibly useful commented out lines in it and looks like this:

#! /usr/bin/perl -sni<br />#      This Perl shell script replaces one or several patterns in a file with others.<br />#      The patterns have to be hard coded.<br />#      Called with a list of files to clean, it replaces them with clean contents.<br /><br />#   The only other thing it does is strip trailing white space<br /><br />s/[ \t]+$//g;            # Strip off trailing white space.<br /><br />#s/\s*=\d+\s*$//g;<br />#s/^.*:webserver\s=\swww.//g;<br /># ... s/www.eugeneweb.com/amusher.org\s=\swww.//g;<br />s/eugenemakerspace\.com/newems.eugeneweb.com/g;<br />#s/ide_/ide-/g;<br />#s/(img\s*src=\s*\")([^\/\"]*\")/$1images\/$2/ig;<br />#s/(height=\"\d+\")>/$1 border="2">/ig;<br />#s/^\d+\s*$/\n/;<br />#s/^\d+ //;<br />#s/%(\d\d)?TWIKIWEB%(\d\d)?//g;<br />print;

Now we give it a list of files to fix up:

cd /etc/apache2<br />~clif/.bin/fixit sites-available/000-default.conf sites-available/eugenemakerspace.com sites-available/default-ssl.conf<br />/etc/init.d/apache2 stop<br />/etc/init.d/apache2 start

Postfix:

Usually you can just install the debian package but Posfix's home is: http://www.postfix.org/

apt-get install postfix-pcre postfix-mysql procmail

Can postfix deliver to maildirs on it's own or does it need something like procmail to help? eg:

mailbox_command = /usr/bin/procmail -a "$EXTENSION" DEFAULT=$HOME/Maildir/ MAILDIR=$HOME/Maildir

BTW it ignores postfixes idea of where the maildirs are. You can either have local mail go into a users home or into the mail spool using one of these options:

home_mailbox = Maildir/  mail_spool_directory = /var/mail/efn.org/

You will probably want to copy the most recent postfix config dir over.

rsync -Pav eugenemakerspace.com:/etc/postfix /etc

Sometimes your Imap client and server can get out of sync on what folders or subfolders exist, which can generate a lot of errors. It's best to restart your client and re-read the folder list when this happens.

If you have a brand new account then courier package util maildirmake is good for creating an empty Maildir. eg,.

maildirmake ~shane/Maildir
chown -R shane.shane ~shane/Maildir/

Some (or most) imap client/servers still use ~/Maildir as their default (instead of ~/.maildir).
In Courier the default location and name of the maildir is set in it's configs residing in /etc/courier, but check for an overide to that in /etc/default/courier (not in debian?).

To set up imap and pop? remove the current deamons ipopd and uw-imapd which are using /etc/cram-md5.pwd for the password file, (and the mbox format mail spool), and install:

apt-get remove ipopd uw-imapd
apt-get install gamin courier-authlib courier-imap courier-imap-ssl courier-pop courier-pop-ssl
sh -c 'for script in /etc/init.d/*courier*; do sudo $script restart;done'

Make sure that the default domain is not set, the unix pam module on accepts bare usernames:

vi /etc/courier/imapd

imapd:# DEFDOMAIN="@example.com"

Things to do:
Check on the realy domain mapings so that it works with postfixadmin.

Here is some info on stats for postfix:
http://www.postfix.org/addon.html#logfile
http://jimsun.linxnet.com/postfix_contrib.html
http://taz.net.au/postfix/mrtg/

Caveats:
Be careful about adding values to new variables in Postfix because they often inherit from others before they're defined even from deprecated ones. for example virtual_alias_maps inherits from virtual_maps and if you don't copy it into virtual_alias_maps before you add the mailman stuff you will break your MTA.
When in doubt check a default value with something like:

postconf [-d] | egrep virtual_alias_maps

For the actuall or default value of a variable

Configuring notes and philosophies:
http://pantz.org/email/postfix/postfixconfigfiles.shtml
Check out Header_Checks also.
To find the current version do:

postconf mail_version

Mailman

Home: http://www.list.org/
Another: http://www.gnu.org/software/mailman/mailman.html
FAQ: http://www.python.org/cgi-bin/faqw-mm.py

First install the debian Mailman package. If you have an old version that you have modified you might need to remove it first to avoid errors.

 apt-get install mailman

Mailman needs a so-called "site list", which is the list from which password reminders and such are sent out from. This list needs to be created before mailman will start. To create the list, run "newlist mailman" and follow the instructions on-screen. Note that you also need to start mailman after that, using /etc/init.d/mailman start. Now you can get to the main page from: http://eugeneweb.com/cgi-bin/mailman/listinfo

In case you want to access mailman thru a shorter URL you should add
something like the following to your webserver configuration (this
line is for apache):
In this case you need to set the DEFAULT_URL_PATTERN in /etc/mailman/mm_cfg.py to http://%s/mailman/ for the cookie authentication code to work. Note that you need to change the base URL for all the already-created lists as well.

In the past we had to keep the second line as well for some links that seem to have incorrect URLs, or at least from virtual domains, maybe we didn't catch all the previously created lists.

The public archives are located in /var/lib/mailman/archives/public,
you can create a symlink from /var/www if you like, or create an alias like
the following:

Alias /pipermail/ /var/lib/mailman/archives/public/

You can also add these two to a virtual host config which enables access from that virtual host. Be sure to put them at the start of the list so that /cgi-bin/mailman/ is found before
/cgi-bin/.

Read /usr/share/doc/mailman/README.Debian.gz for more info. If you add these aliases you need to change DEFAULT_URL_PATTERN, and PRIVATE_ARCHIVE_URL here:
Also uncomment # MTA='Postfix' near the bottom. Read:

zless /usr/share/doc/mailman/README.POSTFIX.gz

Make some changes to /etc/postfix/main.cf, Here the lines to update/add:

Be carefull about adding values to new variables in Postfix because they often inherit from others before they're defined even from deprecated ones. for example virtual_alias_maps inherits from virtual_maps and if you don't copy it into virtual_alias_maps before you add the mailman stuff you will break your MTA.

Add a patch to help with the list name collision problem:
See this post: http://mail.python.org/pipermail/mailman-developers/2005-January/017673.html
Get the modified Postfix.py:
and carefully edit the current Postfix.py here: /usr/lib/mailman/Mailman/MTA/Postfix.py

<strong>/usr/lib/mailman/Mailman/MTA/</strong><strong>/usr/lib/mailman/Mailman/MTA/</strong><strong>PostfixVirtual.py </strong><strong>/usr/lib/mailman/Mailman/MTA/</strong><strong>Postfix.py</strong>

It seems like it is still necessary to add each virtual domain to:

add_virtualhost(), and POSTFIX_STYLE_VIRTUAL_DOMAINS

Also remove the restriction on changing the list name, find real_name near the end of the file
and change it like so:
Initialize the Aliases and Database, set the admin password
To add a new list do:

newlist com.oregonlocations.mopantalk@oregonlocations.com talk@mopan.org

When you change the default list name you will have to add it to privacy --> recipient filters too.

To remove one do:

rmlist -a com.oregonlocations.mopantalk

To remove a message from the archive or the whole archive see this:
[[1]]

vi /var/lib/mailman/archives/private/com.eugenemakerspace.board.mbox/com.eugenemakerspace.board.mbox
mv /var/lib/mailman/archives/private/com.eugenemakerspace.board/ /var/lib/mailman/archives/private/com.eugenemakerspace.board.old/
/var/lib/mailman/bin/arch com.eugenemakerspace.board

Importing lists from another server:

Copy list db in .../lists
Change the name to include the virtdomain prefix
Use check_db to check it.
Use config_list to update the old host_name(s) to the current one, otherwise the default is displayed.
also update the web_page_url var found with dumpdb but not listed with check_db, (though it can be
set with it).
Values to check: owner (listmgr@efn.org) host_name, web_page_url
Use mmmove.pl to read the old /etc/aliases file and generate the new aliases and vitual files.
edit /etc/mailman/mm_cfg.py to add the virtual domain(s)
Copy the list archives
mv archives/private/listname to reverse virtualdomain.listname
mv archives/private/listname.mbox and .../listname.mbox to same.
Symlink from public to private.
Use bin/arch --wipe listname to rebuild the archives
Check / Set permissions

Removing email address from archives:
Based on: http://jstsch.com/post/properly_hide_e-mail_addresses_in_mailman_archives

A quick patch-up solves things! Find Hyper Arch.py in your Mailman binaries dir and find the section which contains ARCHIVER_OBSCURES_EMAILADDRS. Change the try block into something like this:

vi /usr/lib/mailman/Mailman/Archiver/HyperArch.py

               if self.author == self.email:
                   self.author = self.email = "EMAIL HIDDEN" # CCMOD re.sub('@', _(' at '), self.email)
               else:
                   self.email = "EMAIL HIDDEN" # CCMOD re.sub('@', _(' at '), self.email)
           finally:

and much later...

                   if mm_cfg.ARCHIVER_OBSCURES_EMAILADDRS:
                       text = "EMAIL HIDDEN" # CCMOD re.sub('@', atmark, text)
                       URL = self.maillist.GetScriptURL(

py_compilefiles [[Hyper Arch]].py
rm -f /var/lib/mailman/archives/private/com.eugenemakerspace.discuss/*.txt*
/usr/lib/mailman/bin/arch com.eugenemakerspace.discuss
/usr/lib/mailman/cron/nightly_gzip
/usr/lib/mailman/bin/arch com.eugenemakerspace.discuss

Links:
http://homex.subnet.at/~max/mail/mailman.php
http://mail.python.org/pipermail/mailman-users/2003-July/030737.html
http://mail.python.org/pipermail/mailman-developers/2005-January/017673.html
http://mail.python.org/pipermail/mailman-developers/2004-December/017504.html
http://mail.python.org/pipermail/mailman-users/2004-June/037439.html
http://www.megalinux.net/archives/000014.html
http://www.gurulabs.com/files/postfix-to-mailman-2.1.py
Not sure about this but...
http://wiki.osuosl.org/display/LNX/Mailman+with+Postfix+on+Debian+Stable
http://blogs.pure-chaos.com/andrew/archives/2004/12/setting_up_post.html

FUDforum:

Installing FUDforum:

FUDforum dosen't have a deb package but you can probably set it up so it serves multiple sites pretty eaisally. See:
[[2]] Note that the corpus of messages for a forum resides in FUDforum/messages, not in fudforum.

apt-get install php5-pspell
/etc/init.d/apache2 reload

cd /var/www/
wget "http://downloads.sourceforge.net/project/fudforum/FUDforum_3.0.3.zip?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Ffudforum%2Ffiles%2F&ts=1330985295&use_mirror=voxel"    -o FUDforum_3.0.3.zip
unzip FUDforum_3.0.3.zip
mkdir ../www-other/FUDforum
chown -R www-data.www-data fudforum ../www-other/FUDforum
chmod 775 FUDforum
chmod 755  FUDforum/scripts/maillist.php

You might have to move the FUDforum dir contents to it's correct location after the install. You might have other permission problems in messages and errors. When importing check ownership of messages files afterwards.

Create a database and user then goto:
http://www.eugeneweb.com/fudforum/install.php

rm -f install.php uninstall.php upgrade.php

After that if you want to make any changes you can also edit /var/www-other/FUDforum/include/GLOBALS.php.

For each forum that you want to connect to a mailing list you need to add some rules in:
Categories & Forums --> Mailing List Manager, and add lines like these to the
/var/lib/mailman/data/aliases, and /var/lib/mailman/data/virtual-mailman files:

vi /var/lib/mailman/data/aliases
vi /var/lib/mailman/data/virtual-mailman

com.eugenemakerspace.test-forum: "|/usr/bin/php /home/httpd/FUDforum/scripts/maillist.php 1"
...and...
com.eugenemakerspace.test-forum@eugenemakerspace.com  com.eugenemakerspace.test-forum

postalias /var/lib/mailman/data/aliases
postmap /var/lib/mailman/data/virtual-mailman
postfix reload

Make a test list:

newlist com.eugenemakerspace.test@eugenemakerspace.com webmaster@eugeneweb.com

and subscribe the forum to it.

Forum Bridge <com.eugenemakerspace.test-forum@eugenemakerspace.com>

If you are almost done testing and this is a real list such as Announce, you can set the moderation bit so you can cancel the test messages before they get into the list. You probably want to always set the hidden bit for these special addresses.

Testing the setup:

Tailing syslog is a good start, you can also try pipeing just to "|logger" to see that your messages are coming through. Making a little test script like ~clif/.bin/mailpipe is useful:

#!/bin/sh

echo "Running mailpipe"
# /home/httpd/FUDforum/scripts/maillist.php 1
cat - > /home/clif/tmp/foofile
# exit 255

Then set permissions:

mkdir /home/clif/tmp; chmod 777 /home/clif/tmp

Use it with an alias like: test-foo: "|/home/clif/.bin/mailpipe", and capture a test message in foofile. Check the ownership of foofile to see if it's correct, eg www-data, see below... Now try sending that message, (and only keep one in foofile at a time) to a forum like so:

cat /home/clif/tmp/foofile | /home/httpd/FUDforum/scripts/maillist.php 1 

Check that it shows up. Sometimes if there is a permission or db problem it can take forever to process. If this happens your MTA (postfix) will time out and defer the message for later. In that case you won't see the (delivered to command: ... in your mail log. Now uncomment the /home/httpd... line in mailpipe and try sending to test-foo. Sometimes there is a problem in the maillist script that you don't see in the fudforum error log under "General Management", in that case set the mailpipe exit value to 255 and watch the syslog and FF error log for more errors, (also the bounced messages). You can also add a line like: fud_logerror('Maillist Starting.', 'mlist_errors'); to maillist.php and it will show up in the FF error log.

Now try sending a message to com.eugenemakerspace.test-forum@eugenemakerspace.com, and check that it generates a line like this:

"Mar 7 08:31:42 redwall postfix/local[25769]: 57EAB3F4962: to=<com.eugenemakerspace.discuss-forum@eugeneweb.com>,
orig_to=<com.eugenemakerspace.discuss-forum@eugenemakerspace.com>, relay=local, delay=0.78, delays=0.06/0.08/0/0.64,
dsn=2.0.0, status=sent (delivered to command: /home/httpd/FUDforum/scripts/maillist.php 3)"

Oh, you might play with the db settings to see the enhancements are as reliable. Also in the mail rules set up, there is the "Slow Reply Match:" setting which might help if you manually set the subject line to act like a reply to a thread, otherwise it will blindly start a new one with the same subject. Conversely sometimes a message with a different subject will show up in the thread that it was a reply to.

Before importing the archives or going live you might want to set up Subject and Body Mangling. Here are some sample filters:

Subject Mangling:
\^\*(ems|discuss|announce|board)[^\]]*\]\s* i

Body Mangling:
(\n)[>\|\s]*?([>\| \t]*)(________[_]+)[_*\s]*^(([>\| \t]*(________|discuss[ \t]+mailing|discuss\@eug|http:.*mailman).*\n)+([>\|\s]*(\[\d+\])*)*)+   imD
$1$2$3$1$2Discuss mailing list$1

Then add a Forum Signature like:

Posted from the Discuss forum

You should also do something to obfuscate the email addresses contained within message bodies. This may not be that effective but at least it's an attempt. Edit ...scripts/maillist.php and add the bold lines:

vi FUDforum/scripts/maillist.php
      /* Handle inlined attachments. */
      [... a few lines of code ...]
               }
               /* Obfuscate email addresses in the message body */
               // $emsg->body = preg_replace('/((mailto:|<)\s*)?([\w._-]{3,})\@([\w._-]{3,})>?/i', "$3 at $4", $emsg->body);
               $emsg->body = preg_replace('/([\w._-]{3,})\@([\w._-]{3,})/i', "EMAIL HIDDEN", $emsg->body);

               $msg_post->body = $emsg->body;

To import mailman archives do something like this:

cd /usr/src
wget http://download.pear.php.net/package/Mail_Mbox-0.6.3.tgz
tar xzf Mail_Mbox-0.6.3.tgz
cp Mail_Mbox-0.6.3/Mail/Mbox.php /usr/local/lib/php/PEAR

Then get this script: http://cvs.prohost.org/index.php/FudBOX.php
Set these variables; MBOX_FILE, SCRIPT_LOCATION, MBOXPHP_LOCATION, and 'RULE_NUM like so:

vi [[Fud BOX]].php

       define('MBOX_FILE', '/var/lib/mailman/archives/private/announce.mbox/announce.mbox');   // The MBOX file you wish to load
       define('SCRIPT_LOCATION', '/var/www-other/FUDforum/scripts/maillist.php');
       define('MBOXPHP_LOCATION', '/usr/local/lib/php/PEAR/Mbox.php');            // Path to the MBOX PEAR Module
       define('RULE_NUM', 2);                  // Rule number assigned in the Mailing List Manager

chmod +x [[Fud BOX]].php
./FudBOX.php

You can also reload the whole forum from the MM archives this way if you need to do a mass filtering of all the post bodies. However it's tricky, and I'm not sure if all the merged topics will be split up again, or not.

The way I've done it is to delete all the topics in a forum first except maybe a few sticky ones, then you edit /var/www-other/FUDforum/scripts/maillist.php and comment out the part that starts with: /* Handler for our own messages, which do not need to be imported. /. Otherwise you won't get any posts that came from the forum. However you will get the "Posted from the Discuss forum" tag lines with these posts. These could probably be filtered out by adding some code next to the Obfuscate code above.

I do know that the FUD mysql db remembers all the messages in FUDforum/messages/msg_
files so it is a very bad idea to delete those unliess you nuke the db as well. The db tracks character offsets into those files so I made a filter that replaced one string with another of the same length and it mostly worked but there was a problem and I quit before I found out what it was.

Note that postfix delivers with the rights of either the owner of the alias file or nobody.nogroup which can cause problems if it's not set to www-group or the same owner as FUDforum. Try something like this:

chown www-data.list /var/lib/mailman/data/aliases*
chmod 660 /var/lib/mailman/data/aliases*

If for some reason that doesn't work you can also create a fudforum account and put everything there. Then use procmail delivery which will have that user's permissions. See: http://cvs.prohost.org/index.php/Mailing_List_Manager#Import_messages for more info. To wit:

:0:
 * ^TO_.*php-general@lists.php.net
 | /home/FUDdata/scripts/maillist.php 1

Foswiki:

It seems to be missing from the main repos, so add this to your sources.list:

deb http://fosiki.com/Foswiki_debian/ stable main contrib<br />deb-src http://fosiki.com/Foswiki_debian/ stable main contrib

Then get their public key:

apt-key adv --keyserver the.earth.li --recv-keys 379393E0AAEE96F6

Get some extra packages:

apt-get install libapache2-mod-fcgid

Now Foswiki itself, and the old config files:

apt-get install foswiki<br />rsync -Pav eugenemakerspace.com:/etc/foswiki /etc

Then go to:

 http://newems.eugeneweb.com/foswiki/bin/configure
 Also see: http://foswiki.org/System/InstallationGuidePart1 
 and: http://foswiki.org/System/InstallationGuidePart2 
 BTW, the dataset is in /var/lib/foswiki

Copy over all the data:

First the custom webs we added:

rsync -Pav eugenemakerspace.com:/var/lib/foswiki/data/{Ism,Sandbox,Sites} /var/lib/foswiki/data
rsync -Pav  eugenemakerspace.com:/var/lib/foswiki/pub/{Ism,Sandbox,Sites} /var/lib/foswiki/pub

Then the users:

rsync -Pav --ignore-existing eugenemakerspace.com:/var/lib/foswiki/data/Main /var/lib/foswiki/data<br />rsync -Pav --ignore-existing eugenemakerspace.com:/var/lib/foswiki/pub/Main /var/lib/foswiki/pub

Copy over the password, and wikiusers files:

rsync -Pav "eugenemakerspace.com:/var/lib/foswiki/data/.htpasswd*" /var/lib/foswiki/data<br />rsync -Pav eugenemakerspace.com:/var/lib/foswiki/data/Main/WikiUsers.txt /var/lib/foswiki/data/Main

Mailing list <-> Forum notes:

WP Plugins:

wp-mailman
wp-mail-man
http://wordpress.org/extend/plugins/ggis-subscribe/
http://subscribe2.wordpress.com/
http://core.svn.wordpress.org/trunk/wp-mail.php
http://wordpress.org/extend/plugins/mailman-widget/
http://www.plugpress.com/plugin/mailman-widget
http://wordpress.org/extend/plugins/mailman/

RSS to Email: http://tech.ipstenu.org/2011/make-an-rss-powered-email-list/
phpBB:

http://wordpress.org/extend/plugins/phpbb-single-sign-on/
http://wordpress.org/extend/plugins/wordpress-phpbb-last-topics-plugin/
http://www.wpthemesplugin.com/intergrate-phpbb-and-wordpress-wp-united/

Forums:

http://bbpress.org/
http://cartpauj.com/projects/mingle-forum-plugin
http://wordpress.org/extend/plugins/mingle-forum.
http://wordpress.org/extend/plugins/wp-symposium/

Fudforum:

http://fudforum.org/forum/
http://albertfama.com/tutorial-integrating-fudforum-part1/

Drupal:

http://drupal.org/project/mailman_manager
http://drupal.org/project/mailhandler
http://drupal.org/project/listhandler
http://drupal.org/project/user_mailman_register
http://drupal.org/project/mailman_groups
http://drupal.org/project/og
http://groups.drupal.org/node/13220
http://drupalmodules.com/module/mailman-mailing-list-admin
Importing Mailman:

http://www.theworxco.net/blog/importing-mailman-archives-drupal
https://github.com/worxco/drupal_mailman_import

phpBB:

http://mail2forum.com/
[[3]]

Simplemachines:

[[4]]

Forum reviews:

http://www.quickonlinetips.com/archives/2010/05/wordpress-forum-plugins/
http://wordpress.org/extend/plugins/forum-server/

Forums with integrated mailing lists:

[[5]]
http://www.phorum.org/

Notes:

http://codex.wordpress.org/Post_to_your_blog_using_email
Shareing userbase with MM: http://mail.python.org/pipermail/mailman-users/2011-June/071791.html
http://andrewsinlondon.wordpress.com/

Shop IT Notes:

The Shop now has a pfSense router, and a Dell Powerconnect 3448 switch.

Port layout:

table goes here.

Security Cams:

There are two security cams, though only one is active at the moment.

-- Main.ClifCox - 06 Apr 2012