wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

GMail2Notes in less than 300 lines


I admit, the headline is an attention grabber
The total number of lines is, of cause, longer. For one, I used previous code, as well e.printStackTrace() isn't error handling.
Moving data between two places is a favorite IT pasttime (believe me, there are worse ways to kill time). Since I made friends with MIME Messages before, I was wondering how that would fit together with gMail. Every eMail system has its little "specialties". To connect to gMail I used the IMAP with username and password route (which is the only one I know of for IMAP).
Most of the Googles point to " reading the inbox" which is a distinct different task from: " pulling out all your eMail". When connecting to gMail using the enhanced IMAP classes you can pull all "Folders" using store.getDefaultFolder().list(). There a particular folder is interesting: [Gmail]. In Notes speak I would say: it contains the system views, most notably "All documents", "Sent" and "Spam".
To capture all documents going after that one view is sufficient. Google abandoned the idea of "folders" and replaced them with "labels". The idea there was: a document can be in one folder only, but can have many different labels. For other email systems that might hold true, but the way folders a implemented in Notes nicely fits with the idea of labels.
Each message comes with a getLabels() function that directly maps (with a little attention) to a NotesDocument.putInFolder(label) call. The code isn't perfect, but can be a good starting point for your own adventure.
/** ========================================================================= *
 * Copyright (C) 2014 Stephan H. Wissel                                       *
 *                                                                            *
 *  @author     Stephan H. Wissel <stephan@wissel.net>                        *   
 *                                                                            *
 * @version		0.1                                                           *
 * ========================================================================== *
 *                                                                            *
 * Licensed under the  Apache License, Version 2.0  (the "License").  You may *
 * not use this file except in compliance with the License.  You may obtain a *
 * copy of the License at <http://www.apache.org/licenses/LICENSE-2.0>.       *
 *                                                                            *
 * Unless  required  by applicable  law or  agreed  to  in writing,  software *
 * distributed under the License is distributed on an  "AS IS" BASIS, WITHOUT *
 * WARRANTIES OR  CONDITIONS OF ANY KIND, either express or implied.  See the *
 * License for the  specific language  governing permissions  and limitations *
 * under the License.                                                         *
 *                                                                            *
 * ========================================================================== */
package com.notessensei.gimap;

import java.io.InputStream;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Properties;

import javax.mail.Address;
import javax.mail.FetchProfile;
import javax.mail.Folder;
import javax.mail.Message;
import javax.mail.MessagingException;
import javax.mail.Session;

import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.Item;
import lotus.domino.NotesException;
import lotus.domino.NotesFactory;
import lotus.domino.NotesThread;

import com.sun.mail.gimap.GmailFolder;
import com.sun.mail.gimap.GmailMessage;
import com.sun.mail.gimap.GmailSSLStore;

public class GMail2Notes {

	public static int	MAX_TEST_COUNT	= 50;

	public static void main(String[] args) throws NotesException, MessagingException {

		if (args.length < 3) {
			System.out.println("GMail2Notes username@gmail.com password nsfFileName");
			System.exit(1);
		}

		System.out.println("Starting for " + args[0] + " to " + args[2]);
		NotesThread.sinitThread();
		lotus.domino.Session s = NotesFactory.createSession();
		Database db = s.getDatabase("", args[2]);
		GMail2Notes g2n = new GMail2Notes(args[0], args[1]);
		g2n.addExcludedLabel("Spam");
		g2n.addExcludedLabel("Trash");
		int resultCount = g2n.importMessages(s, db, "All Mail");
		System.out.print("Messages imported:");
		System.out.println(resultCount);
		NotesThread.stermThread();
		System.out.println("Done");
	}

	private final String		userName;
	private final String		passWord;
	private GmailSSLStore		store			= null;
	// set it to false for complete import
	private boolean				isTestMode		= true;

	// Labels we don not want to import
	private final List<String>	excludedLabels	= new ArrayList<String>();

	public GMail2Notes(String usr, String pwd) {
		this.userName = usr;
		this.passWord = pwd;
	}

	/**
	 * Add a folder name to the list we are not interested in
	 * 
	 * @param label
	 */
	public void addExcludedLabel(String label) {
		this.excludedLabels.add(label);
	}

	public List<String> getSystemFolderNames() throws MessagingException {
		List<String> result = new ArrayList<String>();
		GmailSSLStore store = this.getStore();
		Folder[] folders = store.getFolder("[Gmail]").list();
		for (Folder f : folders) {
			result.add(f.getName());
		}
		return result;
	}

	public int importMessages(lotus.domino.Session s, Database db, String systemFolderName) throws MessagingException {
		int result = 0;

		// The object to move message to
		Mime2Doc md = new Mime2Doc();

		// Getting gMail ready
		GmailFolder f = this.getSystemFolder(systemFolderName);
		f.open(Folder.READ_ONLY);
		Message[] messages = f.getMessages();
		FetchProfile profile = new FetchProfile();
		profile.add(GmailFolder.FetchProfileItem.CONTENT_INFO);
		profile.add(GmailFolder.FetchProfileItem.LABELS);
		profile.add(GmailFolder.FetchProfileItem.MSGID);
		profile.add(GmailFolder.FetchProfileItem.SIZE);

		f.fetch(messages, profile);
		int count = 0;
		for (Message message : messages) {
			result += this.importOneMessage(s, db, md, message);

			// For testing we don't run through all of them
			count++;
			if (this.isTestMode && count >= MAX_TEST_COUNT) {
				break;
			}
		}
		if (f.isOpen()) {
			f.close(false);
		}
		this.cleanup();
		System.out.println("Done");

		return result;
	}

	/**
	 * We need a delivered date so documents don't show up in SEND
	 * 
	 * @param doc
	 * @param sender
	 */
	private void adjustDeliveredDate(Document doc, Address sender) {
		String senderName = sender.toString();
		if (!senderName.equalsIgnoreCase((this.userName))) {
			try {
				Item PostedDate = doc.getFirstItem("PostedDate");
				doc.copyItem(PostedDate, "DeliveredDate");
				doc.save();
				PostedDate.recycle();
			} catch (NotesException e) {
				e.printStackTrace();
			}
		}

	}

	/**
	 * Strips leading \ from messages
	 * 
	 * @param rawLabel
	 * @return label Stripped from Backslash
	 */
	private String cleanLabel(String rawLabel) {
		return (rawLabel.startsWith("\\") ? rawLabel.substring(1) : rawLabel).trim();
	}

	private void cleanup() {
		if (this.store != null && this.store.isConnected()) {
			try {
				this.store.close();
			} catch (MessagingException e) {
				e.printStackTrace();
			}
		}
		this.store = null;
	}

	private GmailSSLStore getStore() throws MessagingException {
		if (this.store != null) {
			return this.store;
		}
		Properties props = System.getProperties();
		props.setProperty("mail.imaps.connectiontimeout", "5000");
		props.setProperty("mail.imaps.host", "imap.gmail.com");
		props.setProperty("mail.imaps.partialfetch", "false");
		props.setProperty("mail.imaps.port", "993");
		props.setProperty("mail.imaps.timeout", "5000");
		props.setProperty("mail.mime.base64.ignoreerrors", "true");
		props.setProperty("mail.store.protocol", "gimaps");

		Session session = Session.getDefaultInstance(props, null);
		this.store = (GmailSSLStore) session.getStore("gimaps");
		this.store.connect(this.userName, this.passWord);
		// Ready for connection
		return this.store;
	}

	/**
	* can be: All Mail, Drafts, Important, Sent Mail, Spam, Starred, Trash
	**/
	private GmailFolder getSystemFolder(String folderName) throws MessagingException {
		GmailSSLStore store = this.getStore();
		Folder folder = store.getFolder("[Gmail]").getFolder(folderName);
		return (GmailFolder) folder;

	}

	private int importOneMessage(lotus.domino.Session s, Database db, Mime2Doc md, Message message) {
		int result = 0;
		try {
			GmailMessage g = (GmailMessage) message;
			Address sender = g.getSender();
			String[] labels = g.getLabels();
			System.out.print(g.getMessageID());
			if (labels != null) {
				System.out.print(", ");
				System.out.print(Arrays.toString(labels));
			}

			if (this.processThisMessage(labels)) {
				result = 1;
				Document doc = db.createDocument();
				InputStream in = g.getMimeStream();
				md.importMail(s, in, doc);
				this.moveDocToFolders(doc, labels);
				this.adjustDeliveredDate(doc, sender);
				System.out.println(" - processed");
			} else {
				System.out.println(" - skipped");
			}
		} catch (Exception e) {
			// TODO: record the message for follow-up
			e.printStackTrace();
		}
		return result;
	}

	/**
	 * Moves doc to folders as needed
	 * 
	 * @param doc
	 * @param labels
	 */
	private void moveDocToFolders(Document doc, String[] labels) {
		if (labels != null) {
			for (String label : labels) {
				this.movetoMatchingFolder(doc, label);
			}
		}
	}

	private void movetoMatchingFolder(Document doc, String folderCandidate) {
		// TODO handle the SENT folder, Draft folder
		if (folderCandidate != null && !folderCandidate.trim().equals("")) {
			try {
				String realFolder = this.cleanLabel(folderCandidate);
				if (realFolder.equalsIgnoreCase("inbox")) {
					doc.putInFolder("($Inbox)");
				} else if (realFolder.equalsIgnoreCase("drafts")) {
					// TODO handle Drafts
				} else if (realFolder.equalsIgnoreCase("sent")) {
					// TODO handle SENT
				} else {
					doc.putInFolder(realFolder, true);
				}
			} catch (NotesException e) {
				e.printStackTrace();
			}
		}
	}

	private boolean processThisMessage(String[] messageLabels) {
		boolean result = true;

		// If the message has no labels we do process it
		if (messageLabels != null && messageLabels.length < 1) {
			for (String rawLabel : messageLabels) {
				String cleanLabel = this.cleanLabel(rawLabel);
				if (this.excludedLabels.contains(cleanLabel)) {
					result = false;
					break;
				}
			}
		}
		return result;
	}

}

As usual YMMV

Posted by on 2014-01-15 12:37 | Comments (1) | categories: IBM Notes

Comments

  1. posted by francesco marzolo on Monday 28 July 2014 AD - 18:01 Singapore Time:
    Dear Stephan, you made my happy Emoticon rolleyes.gif Emoticon rolleyes.gif. For over 3 years i'm working on Electronic Certified Mail in italy, without finding any other way to convert a mimemessage in notes document than reforward via smtp to Domino Emoticon angry.gif. With many problems about rules and message size.

    Thank you very much.