Entries from April 2006 ↓

When came to office on Saturday, I wanted to spend some time on OpenCerti and complete the pending ActionScript work. But there was a problem waiting! The emails were stuck, and we were getting duplicate emails.

Let me give you some background. We have a set of Linux servers in the office. The magnet-i.com site is hosted in a NOC in USA. We have a few POP accounts on the server and a catchall. The catchall captures emails for most of the users. POP accounts are primarily used by people who travel and need roaming access etc. We download emails to local server using Fetchmail. The internet connection is via cable modem and DSL.

Now this kind of problems have happened before. I remember in the dialup days, if we had a large number of emails, and if the internet connection drops while fetchmail is running, it would start downloading all the emails again the next time you connect. Resulting in duplicate emails.

This time around, we have a major recruitment drive going on. The catchall account was more than 85MB in size. And fetchmail was giving up on it. It was not only the size, but also the number of messages that was big. And the mails were not only for the HR, but also for other people in the organization.

I and Vishal looked at the problem and first tried to delete unwanted mails via webmail. We got it down to 55MB, but this was still too big for fetchmail to handle on our internet connection.

The next step we generally take is to take the mbox file, bzip2 it, download it over web, append to the local mbox, and let the mails flood into the email client. This time, we couldn’t do it because it was not only one user account that the emails were destined too. But this is the track that we take up.

The mbox file contains all the mails in a single file. So we SSH’ed to the server, located the catchall account (something like /home/user/mail/domainname/emailaccount/ on cpanel servers) mailbox. Doing a bzip2 on it, got the file size down to 11MB. (OT: I wonder why they didn’t add compression in POP/SMTP. That would have saved a lot of traffic).

Downloading this via web (so that we can resume the download if it gets broken) did not work. Somehow our server did not allow direct file downloads. (Guess it was me only who disabled hot links like this..) We didn’t have too much time to go reconfigure the server, so we simply FTP’ed the file to one of our servers that we use for exchanging files with clients. Downloaded the 11MB file to local machine and posted it on the mail server.

Now what?

Idea! What if we configure fetchmail to connect to the local POP server? We could create a new user and push the downloaded mails into the new user’s mbox file. When fetchmail connects to the local server to fetch emails, it will fetch emails from this new account. And it can deliver them to the local users as per the original configuration!

The idea was right, and we tested it with one or two messages in the mbox file. First it bounced back, saying there’s a “mail forwarding loop”. We removed the “no dns, aka magnet-i.com” part from the fetchmail config, and it started pushing the emails to the postmaster. At least it did not bounce back! A few trials later - and inspecting the logs and the postmaster emails - we figured it out. We need to have the “aka magnet-i.com” line in, and have the mbox file of the server.

So set this up, ran fetchmail, and it went chopping the mbox like crazy and delivering emails to the local users. If we used some other method (like downloading only new messages with UIDL etc), it would have taken 5 times more time to download the emails, and we would have to monitor the process.

This gets us the best practices for handling large emails that are stuck on the server.

• Bzip the mbox file. Download it via web.
• Unzip the mbox file on local server. And process it there.
• If it’s a single email account, simply append the mbox file to local mbox file. And let the user dowload emails via her email client.
• If there are multiple email accounts in the mbox file (mbox of a catchall), create a new account on local server and append the mbox to it. Add a rule in .fetchmailrc to use local server as POP3 and fetch emails for the new account created. And then distribute it to “* here”
• You can use “fetchmail -v” for verbose output.
• Monitor /var/log/maillog and /var/log/fetchmail.log for info. Also check the postmaster account (we run Postfix) for error reports.

Good troubleshooting for the day! I am off to something bigger now!

Basho’s haiku:

If I could bundle
Fuji’s breezes
Back to town…
What a souvenir!

If only I could!

I have been watching more movies lately. Especially after getting the movie club membership. I am not much of a TV guy and I am wondering if I am becoming one! I like movies. I like well made movies. I can watch any movie and still enjoy it. Now I have the freedom to fast forward the parts that are boring. I also like watching movies home better than the cinema hall. It may be my new home theatre system, but it could also be the freedom I get at home.

In the last ten days, I have seen Hanumaan, Zinda, Jawani Diwani, Scary Movie series, Rang De Basanti and a few odd ones on TV.

And I am wondering if I am wasting my time watching so many movies / TV. It’s a passtime activity these days as I am alone at home. Nobody to talk to, so turn on the TV or music. I believe most of the programs on the TV are crap. But I am enjoying the movies. I feel I can watch one good movie a week!

I am also now understanding why people love going to movies. It’s a good break from the routine and good entertainment also.

May be it’s the vacation season! I will continue watching some more!

For doing file uploads with Flash, you have to use the FileReference class. Now it’s a really nice feature - provides the basic features for uploading files and tracking their progress. There are two things I want to note about it on this post.

1. When the file is uploaded from Mac, the “type” property is not available.
The FileReference class has a property “type”, that gives you the extension (with the dot) of the file. This is very handy to check file types and reject the ones you don’t want to allow. But when you upload a file from the Mac, the “type” property is blank. I am not sure if this is a Mac issue or a Flash issue. But as of now, the best way for you to find the extension of the file is to substr it.

var fName = f.name; // f is the FileReference Object
var ext:String = fName.substr(fName.lastIndexOf(”.”), fName.length);
ext = ext.toLowerCase();

2. You can’t return data from the server to Flash when upload completes
When you provide file upload feature in your application, many of the times, you want to return an ID of the uploaded file or its URL. The FileReference class does not support this. You have to make another call to the server to get any additional information. The FileReference will simply give you “onComplete” event when the server returns HTTP status code 200. It discards any data you send from the server in the response.

It would be great if we can pass a url encoded string back and the values were available to the onComplete event.

We had a colleague who was not performing upto my expectations. He broke his promises and did not seem to put in enough efforts. Seemed like he was taking things for granted. We had talked to him a number of times earlier, warned him too, but things did not improve.

One day I had had enough. One simple task I had assigned was not completed even near the end of the week. I called in the HR, we had a meeting with the guy and we asked him to leave. I also asked HR to take further action and inform other people related.

The guy broke down. It was a shock for him to get this all of a sudden. Guess he did not take our earlier warnings any seriously.

After the meeting, he came to me. He told me he accepts his mistakes and requested to cut down on some of the consequences. I had a long conversation with him. I was not sure if he will give a word and break it again in the future. I asked him that he talks to his Dad about this whole incident and that his Dad calls me up about it. I also asked that he joins another organization and the HR of that company should give me a call within six months mentioning that the guy has transformed.

My intention in the exercise was to give him a jolt and get him on track.

I am yet to see actual results, but I can see that he has indeed transformed. He got my point and took up responsibility and integrity as values for his life. I suggested that anytime he is stopped, he should ask himself “What would a responsible person do in this situation?” and take action accordingly.

Seeing at the change in him, I felt a tremendous level of satisfaction. The joy of making a difference. Of impacting a life that would take the person to newer heights.

I guess you get happier by making others happy!