Favoretti's Braindumps

Backing up Zimbra using LVM

favoretti — Sat, 17 Mar 2012 10:34:43 +0000

Now, since we moved to Zimbra I wanted to actually have it backed up.
Small script to do it using LVM:

#!/bin/bash
DATE=`date -u +%d%m%Y`
 
echo "`date +%k:%M:%S` Backup started."
# Stop zimbra, create a snapshot volume, start zimbra
echo -n "`date +%k:%M:%S` Stopping zimbra... "
/etc/init.d/zimbra stop 2&> /dev/null
echo " done."
echo -n "`date +%k:%M:%S` Creating snapshot volume... "
/usr/sbin/lvcreate -L 5G -s -n vol_zimbrabackup /dev/vg0/vol_zimbrarh 2> /dev/null 1> /dev/null
echo " done."
echo -n "`date +%k:%M:%S` Starting zimbra... "
/etc/init.d/zimbra start 2&> /dev/null
echo " done."
 
# Mount the snapshot, create a backup tarball, unmount snapshot, remove the volume
echo -n "`date +%k:%M:%S` Mounting snapshot... "
/bin/mount /dev/vg0/vol_zimbrabackup /zimbrabackup 2> /dev/null 1> /dev/null
echo " done."
echo -n "`date +%k:%M:%S` Packing up stuff... "
/bin/tar jcf /var/backup/zimbra-$DATE.tar.bz2 /zimbrabackup/* 2&> /dev/null
echo " done."
echo -n "`date +%k:%M:%S` Un-mounting snapshot... "
/bin/umount /zimbrabackup
echo " done."
echo -n "`date +%k:%M:%S` Removing the snapshot... "
/usr/sbin/lvremove -f /dev/vg0/vol_zimbrabackup 2> /dev/null 1> /dev/null
echo " done."
 
# Cleanup logs older than a week
echo -n "`date +%k:%M:%S` Cleaning up old stuff... "
/usr/bin/find /var/backup/zimbra*.bz2 -type f -mtime +7 | xargs rm -f 2&> /dev/null
echo " done."
echo "`date +%k:%M:%S` Backup done."

Migrating from Scalix to Zimbra

favoretti — Sat, 17 Mar 2012 10:32:44 +0000

So, this week I again got a feeling that Scalix is not coming back from the dead as a company. After waiting for more than a year for new versions – I gave up.

Search for a new solution ended up on choosing Zimbra Opensource Edition.

Considering that I wasn’t able to find any real guide to migration, I decided to lay down my experiences over here for my own use and for whoever might find this useful.

Setup was smooth. We’re running CentOS 5 on our mailserver, so I just picked the RH5 rpms, did

# ./install.sh --platform-override

and it installed without problems.

Granted that we ran Scalix on the same box, I moved all Zimbra’s standard ports to +60000, so that IMAP was running on 60143, POP on 60110, web on 60080, etc.

After installing I went looking for a way to bulk-create my users.
Provided ‘zmprov’ command helped out in that:

zimbra$ zmprov createAccount user@vlfn.nl displayName 'Firstname Lastname' givenName Firstname sn Lastname

To later migrate the mail, I just set all the passwords to a default one for now:

zimbra$ zmprov ma user@vlfn.nl userPassword foobar

Now I wanted Zimbra to actually start receiving mail for the users.
Stop sendmail and SMTP relay of Scalix:

# /etc/init.d/sendmail stop
# omoff -d0 "SMTP Relay"

Move Zimbra SMTP port into place:

zimbra$ zmprov ms groupserve.vlfn.nl zimbraSmtpPort 25

Restart Zimbra. Voila, Zimbra now receives your incoming mail. Whatever Scalix received in the meantime will be migrated in a few.

After that I needed to migrate users’ mail of course.
A few googling minutes and imapsync found itself.

I’ve created all users with standard password for now, to be able to use imapsync. More on restoring original passwords below.

You can access user’s mailbox as sxadmin in Scalix without knowing user password, so that is the functionality that I abused to achieve what I wanted.
Scalix virtually raped IMAP protocol, by adding custom flags that Zimbra doesn’t understand, so I ended up stripping all the ‘X-*’ flags from mails. Final command to migrate user’s mail looked like:

# /usr/bin/imapsync --nofoldersizes --skipsize --allowsizemismatch --regexflag s/\\X-*//g --noauthmd5 --buffersize 8192000 --nosyncacls --subscribe --syncinternaldates --host1 vlfn.nl --user1 mboxadmin:sxadmin:user --password1 sxadminpass --host2 vlfn.nl --port2 60143 --user2 user@vlfn.nl --password2=foobar

A few words on the flags:
–nofoldersizes and –skipsize will just make process faster as it won’t calculate the size of the folders and will not use message size comparison (in essence, there’s nothing to compare to in the new installation, so it saves an extra call to the new server)
–buffersize increased to 8M helps zimbra to process stuff faster.
–user1 format of “mboxadmin:sxadmin:user” allows you to access user’s mailbox with sxadmin password

Another handy tweak is increasing Zimbra’s indexing batch size to 20, so that it reruns its mail indexing not after each mail, but after every 20:

zimbra$ zmprov mc vlfn.nl zimbraBatchedIndexingSize 20

did that trick for me.

Note that this is where alternative ports for Zimbra come into place. In essence I’m running 2 servers next to each other on the same machine.

This process took a while. My mailbox is about 12GB in size, with around 80000 mails in it, it took somewhat 3,5 hours to migrate.

After mail has been migrated I needed to restore users’s original passwords.

First step was to extract current user passwords in crypted form from existing scalix installation.

# omsearch -d userlist -t h -e s=*

extracted all the needed information and with a few awk tricks I managed to extract UL-AUTH-ID and UL-PWD out of the “/”-delimited fields.

After that I replaced all the escaped slashes in the passwords and created a command to update passwords for the users:

zimbra$ zmprov ma user@vlfn.nl userPassword '{crypt}$1$izF4C3VL$Mw.xYuDwa21UL4dB5.CO.1'

At this point we’re almost good to go. Now we just need to kill Scalix, move Zimbra IMAP to normal port, configure Apache to proxy Zimbra’s virtual host and go have our evening coffee.

Killing scalix is easy:

# /etc/init.d/scalix stop
# /etc/init.d/scalix-postgres stop
# /etc/init.d/scalix-tomcat stop
# chkconfig --level 345 scalix off
# chkconfig --level 345 scalix-postgres off
# chkconfig --level 345 scalix-tomcat off

Now, let’s move Zimbra services into place:

zimbra$ zmprov ms groupserve.vlfn.nl zimbraImapBindPort 143
zimbra$ zmprov ms groupserve.vlfn.nl zimbraPop3BindPort 110

And restart Zimbra:

# /etc/init.d/zimbra restart

At this point, you can happily inform your users that they can access stuff via IMAP/SMTP.

Now for the webmail. I run Apache, so I didn’t want Zimbra to be the only thing hoggering the HTTP(s) ports. Therefore, as I already mentioned above, I told it to bind itself on a +60000 range.

Create a separate virtualhost in apache config:


   ServerName webmail.vlfn.nl
   ServerAdmin webmailadmin@vlfn.nl
   ProxyPass / http://webmail.vlfn.nl:60080/
   ProxyPassReverse / http://webmail.vlfn.nl:60080/
   ErrorLog logs/zimbra-error.log
   CustomLog logs/zimbra-access.log common

And if you please a /webmail redirect from your normal website:

Redirect /webmail http://webmail.vlfn.nl

Restart apache.

Now go grab that coffee. You surely deserve one.

How to install Adobe Photoshop CS5 on a case-sensitive MacOS volume

favoretti — Sat, 21 Jan 2012 11:52:56 +0000

https://bitbucket.org/lokkju/adobe_case_sensitive_volumes

This is just incredible. I’m sooooo thankful to this guy. Millions of kudos.

+1, really +1

favoretti — Fri, 06 Jan 2012 14:46:28 +0000

Oh so true.

And in 2 smooth moves I got myself a new desktop.

favoretti — Sat, 26 Nov 2011 10:37:52 +0000

Having been totally annoyed by constant windows quirks, being a unix admin and being used to easily tunnel SSH connections and whatnot on a mac, one day I got totally pissed off by not being able to do that easily on windows machine, so I rushed into a mac reseller demanding a mac pro… So, there it went

Box was a bit dirty, but who cares, it’s all about the contents!

Your quick geek-a-lion “unboxing”

There, happy now!

F6 :(

favoretti — Wed, 02 Nov 2011 12:13:35 +0000

Python script to clean up binlogs based on number

favoretti — Tue, 25 Oct 2011 11:06:13 +0000

I stumbled on a problem in MySQL that you can only rotate binlogs based on age. In certain cases it wasn’t sufficient. Scripts that I’ve found on the net (mysql doc, etc) were all in PHP/perl… So I took a bit of liberty and stole one idea from mysql’s doc comments and turned it into a piece of Python code. (uyes, ai luv python… hisssssssss).

#!/usr/bin/env python
#
# Since MySQL is unable to purge logs based on number of logs and does it only based on days
# I stole this idea from mysql.com. It was in PHP, I rewrote it in Python <3
#
#         vllazarenko@ebay.com
#
# Questions? Do mail or pass by.
#
#
 
import sys
import MySQLdb
 
""" If you want to smurf, smurf over here """
 
slaveHost="localhost"
slaveUser="root"
slavePass="rootpass"
slaveDb=""
 
masterHost="masterhost"
masterUser="root"
masterPass="rootpass"
masterDb=""
 
keepLogs = 5
 
 
""" Please don't smurf after this line, unless you're an experienced smurf """
 
try:
     connSlave = MySQLdb.connect(slaveHost,
        	                 slaveUser,
                	         slavePass,
                        	 slaveDb)
except MySQLdb.Error, e:
     print "SLAVE: Error %d: %s" % (e.args[0], e.args[1])
     sys.exit (1)
 
try:
	connMaster = MySQLdb.connect(masterHost,
        	                     masterUser,
                	             masterPass,
                        	     masterDb)
except MySQLdb.Error, e:
     print "MASTER: Error %d: %s" % (e.args[0], e.args[1])
     sys.exit (1)
 
cursSlave = connSlave.cursor(MySQLdb.cursors.DictCursor)
cursMaster = connMaster.cursor(MySQLdb.cursors.DictCursor)
 
cursSlave.execute("SHOW SLAVE STATUS")
row = cursSlave.fetchone()
 
if row["Slave_IO_Running"] != "Yes":
	print "Alert. Slave IO not running."
if row["Slave_SQL_Running"] != "Yes":
	print "Alert. Slave SQL not running." 
if row["Last_Error"] != "":
	print "Alert. Error on slave: %s" % row["Last_Error"]
 
masterLog = row["Master_Log_File"]
print "Current Master file on which Slave is acting: %s" % masterLog
 
cursMaster.execute("SHOW MASTER LOGS")
rows = cursMaster.fetchall()
 
if cursMaster.rowcount == 0:
	print "No master logs found."
	sys.exit(1)
 
lognames = list()
 
for row in rows:
	lognames.append(row["Log_name"])
 
marker = lognames.index(masterLog)
delMarker = marker - keepLogs;
 
if delMarker < 2:
	print "Not purging below 2 logs."
	sys.exit(1)
 
print "Going to purge logs up to %s." % lognames[delMarker]
 
delQuery = "PURGE MASTER LOGS TO '%s'" % lognames[delMarker]
# cursMaster.execute(delQuery)
 
print "Master binlogs were purged. The oldest log now available on the master is %s." % lognames[delMarker]
 
cursSlave.close()
connSlave.close()
cursMaster.close()
connMaster.close()

Just me…

favoretti — Mon, 17 Oct 2011 11:58:54 +0000

Music…

favoretti — Mon, 17 Oct 2011 11:31:14 +0000

Want an iPhone 4G upgraded to 4S? No problem.

favoretti — Wed, 05 Oct 2011 15:29:56 +0000