Rethinking the MimeDocument data source

Tim (we miss you) and Jesse had the idea to store beans in Mime documents, which became an OpenNTF project.
I love that idea and was musing how to make it more "domino like". In its binary format, a serialized bean can't be used for showing view data, nor can one be sure that it can be transported or deserialized other than through the same class version as the creator (this is why Serialized wants to have a serialid).
With a little extra work, that becomes actually quite easy: Enter JAXB. Serializing a bean to XML (I hear howling from the JSON camp) allows for a number of interesting options:

• The MIME data generated in the document becomes human readable
• If the class changes a litte de-serialization will still work, if it changes a lot it can be deserialized to an XML Document
• Values can be extracted using XPath to write them into the MIME header and/or regular Notes items - making it accessible for use in views
• Since XML is text, full text search will capture the content
• Using a stylesheet a fully human readable version can be stored with the original MIME (good to eMail)

I haven't sorted out the details, but lets look at some of the building blocks. Who ever has seen me demo XPages will recognize the fruit class. The difference here: I added the XML annotations for a successful serialization:

package test;

import javax.xml.bind.annotation.XmlAccessType;
import javax.xml.bind.annotation.XmlAccessorType;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement(name = "Fruit", namespace = "http://www.notessensei.com/fruits")
@XmlAccessorType(XmlAccessType.NONE)
public class Fruit {
@XmlAttribute(name = "name")
private String name;
@XmlElement(name = "color")
private String color;
@XmlElement(name = "taste")
private String taste;
@XmlAttribute(name = "smell")
private String smell;

    public String getSmell() {
        return this.smell;
    }

    public void setSmell(String smell) {
        this.smell = smell;
    }

    public Fruit() {
        // Default constructor
    }

    public Fruit(final String name, final String color, final String taste, final String smell) {
        this.name = name;
        this.color = color;
        this.taste = taste;
        this.smell = smell;
    }

    public final String getColor() {
        return this.color;
    }

    public final String getName() {
        return this.name;
    }

    public final String getTaste() {
        return this.taste;
    }

    public final void setColor(String color) {
        this.color = color;
    }

    public final void setName(String name) {
        this.name = name;
    }

    public final void setTaste(String taste) {
        this.taste = taste;
    }

}

The function (probably in a manager class or instance) to turn that into a Document is quite short. The serialization of JAXB would allow to directly serialize it into a Stream or String, but we need the XML Document step to be able to apply the XPath.

public org.w3c.dom.Document getDocument(Fruit fruit) throws ParserConfigurationException, JAXBException {
  DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
  DocumentBuilder db = dbf.newDocumentBuilder();
  org.w3c.dom.Document doc = db.newDocument();
  JAXBContext context = JAXBContext.newInstance(fruit.getClass());
  Marshaller m = context.createMarshaller();
  m.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE);
  m.marshal(fruit, doc);
  return doc;
 }

The little catch here: There is a Document in the lotus.domino package as well as in the org.w3c.dom package. You need to take care not to confuse the two. Saving it into a document including the style sheet (to make it pretty) is short too. The function provides the stylesheet as a w3c Document and the list of fields to extract as a key (the field) - value (the XPath) map. Something like this:

       public boolean saveFruitXmlToDocument(lotus.domino.Session session s, lotus.domino.Document d, org.w3c.dom.Document xDoc, org.w3c.dom.Document style) {
  public static final MIME_FIELD_NAME = "Body";
  boolean success = true;
        Map<String,String> defs = new HashMap<String,String>();
        //These are specific to the fruit class, should be read from a source
        defs.add("Subject", "/fruit:Fruit/@name");
        defs.add("Taste", "/fruit:Fruit/taste");
        defs.add("Color", "/fruit:Fruit/color");
        defs.add("Attributes", "/fruit:Fruit/@*");
        defs.add("Summary", "/fruit:Fruit");

        try {

boolean oldMime = session.isConvertMime();
session.setConvertMime(false);

// We must remove the mime body - otherwise it will not work
if (d.hasItem(MIME_FIELD_NAME)) {
d.removeItem(MIME_FIELD_NAME);
}

// Create the top mime entry
MIMEEntity root = d.createMIMEEntity(MIME_FIELD_NAME);
MIMEHeader header = root.createHeader("Content-Type");
header.setHeaderVal("multipart/mixed");

MIMEEntity body = root.createChildEntity();
MIMEHeader bheader = body.createHeader("Content-Type");
bheader.setHeaderVal("multipart/alternative");

MIMEEntity textMime = body.createChildEntity();
MIMEHeader textHeader = textMime.createHeader("Content-Type");
String cType = (this.style == null) ? "text/plain" : "text/html";
textHeader.setHeaderVal(cType);
Stream stream2 = session.createStream();
stream2.write(this.dom2Byte(xDoc, style));
textMime.setContentFromBytes(stream2, cType + "; charset=\"UTF-8\"",
MIMEEntity.ENC_NONE);

          MIMEEntity xmlMime = body.createChildEntity();
             MIMEHeader xmlHeader = xmlMime.createHeader("Content-Type");
             xmlHeader.setHeaderVal("application/xml");
             Stream stream = session.createStream();
             stream.write(this.dom2Byte(xDoc,null));
             xmlMime.setContentFromBytes(stream, "application/xml; charset=\"UTF-8\"", MIMEEntity.ENC_NONE);

// Now extract the fields
for (Map.Entry<String,String> e : this.definitions) {
this.extractOneField(xDoc, d, body, e.getKey(), e.getValue());
}
session.setConvertMime(oldMime);
} catch (Exception e) {
success = false;
e.printStackTrace();
}

return success;
}

private void extractOneField(org.w3c.dom.Document xDoc, lotus.domino.Document d,
MIMEEntity body, String fName, String xPath) throws NotesException {

if (fName.equalsIgnoreCase(MIME_FIELD_NAME)) {
return; // We can't allow to overwrite the body
}

Vector<String> v = this.xpath2Vector(xDoc, xPath);
Item curItem = d.replaceItemValue(fName, v);
}

private Vector<String> xpath2Vector(org.w3c.dom.Document xDoc, String xPathString) {

        Vector<String> result = new Vector<String>();
        Object exprResult = null;
        XPath xpath = XPathFactory.newInstance().newXPath();

MagicNamespaceContext nsc = new MagicNamespaceContext();
xpath.setNamespaceContext(nsc);

        try {
            exprResult = xpath.evaluate(xPathString, doc, XPathConstants.NODESET);
            for (int i = 0; i < nodes.getLength(); i++) {
               Node curNode = nodes.item(i);
               result.add(curNode.getNodeValue());

}
} catch (XPathExpressionException e) {
System.err.println("XPATH failed for " + xPathString);
}  
 return result;
}

private byte[] dom2Byte(Node dom, org.w3c.dom.Document stylesheet) {
StreamResult xResult = null;
DOMSource source = null;
ByteArrayOutputStream out = new ByteArrayOutputStream();
Transformer transformer = null;

        try {
            TransformerFactory tFactory = TransformerFactory.newInstance();
            xResult = new StreamResult(out);
            source = new DOMSource(dom);

            if (stylesheet != null) {
                DOMSource style = new DOMSource(stylesheet);
                transformer = tFactory.newTransformer(style);
            } else {
                transformer = tFactory.newTransformer();
            }

            // We don't want the XML declaration in front
            transformer.setOutputProperty(OutputKeys.OMIT_XML_DECLARATION, "yes");

            transformer.transform(source, xResult);

        } catch (Exception e) {
            e.printStackTrace();
        }

        return out.toByteArray();
    }

The final missing ingredient is the MagicNamespace:

package test;

import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;

import javax.xml.XMLConstants;
import javax.xml.namespace.NamespaceContext;

public class MagicNamespaceContext implements NamespaceContext {

    private final Map<String, String>      nameSpacesByPrefix = new HashMap<String, String>();
    private final Map<String, Set<String>> nameSpacesByURI    = new HashMap<String, Set<String>>();

    public MagicNamespaceContext() {
        // We add the default namespaces for XML and attributes
        this.addNamespace(XMLConstants.XML_NS_PREFIX, XMLConstants.XML_NS_URI);
        this.addNamespace(XMLConstants.XMLNS_ATTRIBUTE, XMLConstants.XMLNS_ATTRIBUTE_NS_URI);

        // We make d / dxl namespaces for Domino/DXL
        this.addNamespace("", "http://www.lotus.com/dxl");
        this.addNamespace("d", "http://www.lotus.com/dxl");
        this.addNamespace("dxl", "http://www.lotus.com/dxl");
        this.addNamespace("fruit", "http://www.notessensei.com/fruits");
    }

    /**
     * Adds a namespace to the context. If the context doesn't exist
     *
     * @param prefix
     * @param uri
     */
    public synchronized void addNamespace(String prefix, String uri) {
        this.nameSpacesByPrefix.put(prefix, uri);
        if (this.nameSpacesByURI.containsKey(uri)) {
            Set<String> set = this.nameSpacesByURI.get(uri);
            if (!set.contains(prefix)) {
                set.add(prefix);
            }
        } else {
            Set<String> set = new HashSet<String>();
            set.add(prefix);
            this.nameSpacesByURI.put(uri, set);
        }

    }

    public synchronized void addNamespaces(Map<String, String> newNamespaces) {

        for (String prefix : newNamespaces.keySet()) {
            this.addNamespace(prefix, newNamespaces.get(prefix));
        }

    }

    public String getNamespaceURI(String prefix) {
        if (this.nameSpacesByPrefix.containsKey(prefix)) {
            return this.nameSpacesByPrefix.get(prefix);
        }
        return XMLConstants.NULL_NS_URI;
    }

    public String getPrefix(String namespaceURI) {
        return this.getPrefixes(namespaceURI).next();
    }

    public Iterator<String> getPrefixes(String namespaceURI) {
        Set<String> set = this.nameSpacesByURI.get(namespaceURI);
        if (set == null) {
            return null;
        }
        return set.iterator();
    }
}

and of course a test:

package test;

import java.util.Collection;
import java.util.HashSet;
import lotus.domino.Database;
import lotus.domino.Document;
import lotus.domino.NotesException;
import lotus.domino.NotesFactory;
import lotus.domino.NotesThread;
import lotus.domino.Session;

public class InitialTest {

    public static void main(String[] args) throws NotesException {
        NotesThread.sinitThread();
        Session s = NotesFactory.createSession();
        System.out.println("\n\n ************* Test starting ******************\n\n");
        InitialTest t = new InitialTest(s);
        t.createDoc();
        System.out.println("\n\n ****************** Test done ********************\n\n");
        NotesThread.stermThread();
    }

    private static final String dbName = "XMLDocuments.nsf";
    private final Session s;

    public InitialTest(Session s) {
        this.s = s;
    }

    public void createDoc() throws NotesException {
        org.w3c.dom.Document style = DomHelper.INSTANCE.file2Dom("fruit.xsl");
        Database db = s.getDatabase(null, dbName);
        Fruit f = new Fruit("Apple","Green","Sweet","Really nice");
        Document d = db.createDocument();
        org.w3c.dom.Document xDoc = getDocument(f);
        saveFruitXmlToDocument(s, d, xDoc, style);
        d.save();
        d.recycle();
    }
}

Still work in progress, but you get the idea.