Usability - Productivity - Business - The web - Singapore & Twins

Application Migration vs. XPage enablement

In a recent customer discussion a claim was made: " If Notes client application don't automagically can be converted into XPages applications, then we very well can migrate to another platform, it is just the same". An interesting claim, I'll subject it to a reality check. In any case it is a good idea to revisit your investment in your existing applications first.

XPages Enablement

Gradual availability of new functions
  • A reasonable estimate for the required effort is up to 30% of the initial effort. It very much depends on the quality of the existing applications and familiarity and skills of your team
  • There is no data migration, so existing users using a Notes client can continue to use the application until you retire the Notes clients
  • You can look at your application from a functionality and user perspective. A typical first XPages application is an approval module that spans across all your existing applications and provides a single view into anything that needs approval. A next level is forms and tools for mobile personell
  • Domino's security model works in XPages too, so no changes are required
  • If a function in an XPage shows a bug, it doesn't stop executing the function, since a user can fall back to a Notes client until the bug is fixed
  • Mail notifications just work, it's Domino. Also you can build Attention Central
So in a nutshell: low risk, incremental availability, no data migration, freedom to innovate

Read more

Posted by on 2014-12-01 03:32 | Comments (0) | categories: XPages

Poking around the iNotes HTTP API (Part 3) - retrieve message meta data

So far we learned about the URLs, how to retrieve the Outline that contains folders and views and how to utilize JavaScript to deal with an API tha hadn't been designed for language neutral access. This part will deal with the retrieval of messages. There are quite some formats, strategies and tweaks to consider:
  • A message consists of headers (fields), and message parts that can be plain text, HTML, embedded images, attachments or other stuff (like the JSON used for embedded experiences). When to retrieve what part is an excellent question
  • Depending on your use case, your final client might or might not have access to the original iNotes implementation (Firewall, Offline, VPN), so just showing the HTML and let it fetch missing pieces might not do the trick
  • The HTML retrieved might not be well formed or contain links you don't want to execute (tracker images, tracker CSS, JavaScript stuff)
  • Automatically retrieving the whole message can be network heavy if there are large images or attachments in the MIME parts, this needs to be planned carefully
  • iNotes allows to retrieve the message fields as (kind-of) JSON using l_JSVars or retrieve the MIME headers using l_MailMessageHeader
  • The content of the message is accessible via s_MailMemoReadBodyContent or UNID/Body?OpenField or l_MailMessageHeader&PresetFields=FullMessage;1 (as raw Mime)
  • The MIME parts retrieved come infested with <br /> tags, since iNotes renders them in a display window (I would have used <pre>, but that's a different story. So they need cleanup
To begin I'll use the result of l_JSVars to learn about the document. Since, again, it is JavaScript, not JSON, I'll take the Rhino approach to extract a Java object.
The parts I'm interested in is in the variable named DXX which contains item that is a representation of NotesItems. True to the nature of an item, the JSON is quite elaborate, which you can see on the example of the form item:
{ "@name": "Form",
  "textlist": {
    "text": [
        {"0": "Memo"}

For our purposes this will be flattened into a Map that has the item name as key. The Java code is quite similar to the retrieval of the Outline, the magic is in the JavaScript:
public MessageObject(String code, String rawMsg) {
    long start = System.currentTimeMillis();
    ScriptEngine engine = new ScriptEngineManager().getEngineByExtension("js");
    Invocable jsFunc = (Invocable) engine;
    MessageObject msg = (MessageObject) jsFunc.invokeFunction("getMessageObject", rawMsg);
    long end = System.currentTimeMillis();
    System.out.print("Time in ms:");
    return msg;

A little detail to stumble over is the item structure. It can be textlist/text or only text (feels like MIME Headers) and datetimelist/datetimepair/datetime or datetimelist/datetime.

Read more

Posted by on 2014-12-01 11:04 | Comments (0) | categories: IBM Notes

Poking around the iNotes HTTP API (Part 2) - Fun with Rhino

The iNotes HTTP API wasn't designed for consumption outside the iNotes web client. This becomes apparent when looking at Form=l_GetOutline_JSON or Form=l_JSVars that return JavaScript and not JSON! The difference? JSON contains data only, while JavaScript contains function definitions too.
To make use of data included in the JavaScript there are 2 possibilities:
  • do some elaborate String operation
  • actually run JavaScript
Since Java 6, the JDK contains a JavaScript engine, that originally was developed as Mozilla Rhino. The most recent incarnation in Java 8 is Nashorn (which is German for Rhino). When you use an older JDK, think "older JS", nice functions like JSON.stringify or JSON.parse are missing and need to be added from elsewhere.
Turning the outline information into something Java useful we need to parse the returned JSON, lets see how that works. The Outline is a little more complex that it might seem. The JSON delivers 6 values (in an array) about the outline entries:
  1. The hierarchical sequence number. Something like 1 or 3.1 or 3.2.1 - depending on the level. The numbers have gaps where entries in the Outline are hidden from the web
  2. The hierarchy level starting with 0. So Asia\South-East\Singapore would be level 2
  3. The Label of the entry. Can be different from the actual name of the element
  4. The full path to the image for that entry. We'll strip that to get the name without a path, so we can use our own icons
  5. The full URL to that entry. It has the View Label (Name) and its UNID or system name in the string. We need the UNID for the unread count, so we'll extract that
  6. Can something be dragged there. 1 = Yes, it is a real folder, 0 = probably not, it is a view or a label only. "Probably" since e.g. you should be able to drag into the "Delete" view

Read more

Posted by on 2014-11-27 06:07 | Comments (0) | categories: IBM Notes

Poking around the iNotes HTTP API (Part 1)

With the rise of web applications, something interesting happened: applications gained an observable API. Delivered over HTTP(s) with a few commands (GET, POST etc.) it became easier to find out " what is going on". Tools like Wireshark, Fiddler, Charlesproxy or TCPMon sit between your HTTP consuming client and the HTTP generating server. In browser applications the build in developer tools (Ctrl-Shift-I) provide the same level of access.
So anything flowing over HTTP is effectively an API. This led to the idea of the API economy (but that's a story for another time). Intercepting HTTP(s) traffic can be easier and more fun, than trying to customise customise a backend you might not have access to. (Of course there is evil interception too). Point is case, iNotes is quite a stack:
3d view of the iNotes client

So I poked around the iNotes API, this is what I found:

Read more

Posted by on 2014-11-24 10:14 | Comments (0) | categories: IBM Notes

Flashcards - German Version: Die Lernkartei

The offspring and SWMBO had an argument about the need for Chinese tuition when attending Singapore's oldest school. You can guess who had what position. Disecting the positions and looking at interest (you remember your basic negotiation skills , do you?) it boiled down to the wish of self directed learning vs. fear of failure.
Having gone through both (I remember my Latin tuition teacher well. I loathed him and in return he pushed me within 6 month from F to A), I concluded: self direction needs skills. So one task to complete is Learning how to learn. To be very clear: no technique can replace the I want to learn, but once the want is established, being skilful helps (and reinforces the want). Cousera even has a Learn how to learn class online.
One tool, well established for learning facts (like vocabulary), are flash cards (there are others). Interestingly the Germans put some rigor into the general idea of sets of flash cards using a Lernkartei. The Lernkartei uses a spaced repetition approach proposed as early as 1885 and made popular by Sebastian Leitner.
The Learnbox as proposed by Sebasitan Leitner
In a nutshell:
At stage 1 the learner adds new learning topics (vocables, phrases, facts) in bundles of 6-10 with up to 30 cards a day. These cards are reviewed and learned until they are memorised for the session. They then go into stage 2. When reviewing stage 2 cards, successful memorised cards go to stack 3 and failed cards back to stack 1. A successful review on stack 3 promotes the cards to stack 4, while a failure demotes it to stack 1. Stack 5 cards are considered retired and are archived after one successful review.
The key of the system is the ever increasing interval between the reviews: stage 1 is reviewed daily, stage 2 every 3 days, stage 3 every 5, stage 4 every 7 and stage 5 just occasionally. There are other interval suggestions available too.
You might have guessed: that system has been put into software, more than once. One popular application is Anki (which uses the SuperMemo algorithm). Others add a social component.
All application I glanced at made one basic assumption: there is one question with one answer (which might be reversible). However reality for vocabulary is a little more complex:
A word has more than one dimension
* only limited suitability as answer
So what to do? Write a multi-facetted flash card app? A flashy diamond perhaps?

Read more

Posted by on 2014-11-23 06:24 | Comments (1) | categories: Learningbox Learning

Last usage of a mail file - for all users

My admin is getting a little rusty. When I was asked: " How can I indentify a dormant mailbox?" I couldn't name a place in admin where to look. Of course, there is the NotesDatabase.LastModified property, but that would get updated on a design refresh too. So I asked innocently: " dormant, how?" After an amusing 2 hour discussion between the stake holders (only the popcorn was missing) we concluded that it is either
  • No document has been created for a while (including incoming)
  • No email has been sent for a while
Calendar entries or todo were not part of the consideration, but that's just a question in what view you look. For "any" document it is ($All) and for sent documents ($Sent). The LotusScript code is just a few lines. I needs to be run, so it can access all mailboxes.

Read more

Posted by on 2014-11-21 10:05 | Comments (3) | categories: IBM Notes

Providing user information in JSON

In the MUSE project we encountered the need to retrieve user information in JSON format. Easy done one would think. The trouble starts, when you have multiple directories and you need reasonable speed. Sometimes falling back to @Fomulas gives you what you need, fast and easy. @NameLookup knows where to look and you don't need any extra configuration. A simple call to an form will give you all you need: https://yourserver/somedb.nsf/namelookup?Readform for yourself or add &User=John Doe for any other user. This will return:
"QueryName": "John Doe",
"NotesName": "CN=John Doe/OU=ThePitt/O=GIJoe",
"AllNames": [
"CN=John Doe/OU=ThePitt/O=GIJoe",
"John Doe/ThePitt/GIJoe",
"John Doe",
"eMail": "john.doe@thepitt.com",
"MailDomain": "SACMEA",
"MailServer": "CN=PittServer42/OU=ThePitt/O=GIJoe",
"MailFile": "mail/jdoe.NSF",
"Empnum": "0815",
"Empcc": "4711">

The form makes extensive use of @NameLookup and looks in DXL quite simple.

Read more

Posted by on 2014-11-20 04:42 | Comments (0) | categories: IBM Notes IBM Connections XPages

Building a shared approval frontend in XPages

The saying goes: " God would not have been able to create the world in 7 days, if there would have been an installed base to take care of". As much as we wish to have divine powers, we need to make with less and look after an installed base. Point in case: You have a set of approval applications in classic Notes client code, like: Travel, Leave, Expenses, Gifts, Training, BoM changes etc. You are challenged to provide a web and/or mobile interface for them.
Considering your options you have to decide between options:
  1. Dump the applications and rebuild them on a new platform. Hope that it will take long enough, so nobody will ask you to migrate any of the existing data
  2. Enable them one-by-one
  3. Hire Redpill to do an asymmetric modernization
  4. Build a lateral approval screen across your applications and leave the full web enablement for later
While #1 would allow you to silently dump all your technical debt, it will take too long and a potential data migration monster would lurk in the dark. #2 is what non-techies would expect, but again you spend too much time and you will carry forward new technical debt. After all the applications are more similar than different. Hiring Redpill would be a sensible option, but that requires board approval or is against your "we-do-it-inhouse" policy or you simply want to expand your skill horizon.
This leaves you with #4. To do this successful, you need to define your scope clearly. In the rest of the article I will work with the following assumptions, if they don't fit your situation, adjust your plan:
  • The goal is to provide the decision making screen only, not a complete workflow engine (it could evolve to that)
  • Request submission happens in the existing application, providing a new interface for that is outside the current scope
  • The flow logic (how many approvers per level etc.) is determined on submission by the original application
  • Someone from a different department indicated interest to use the approval screen from a complete different system (that is the "scope creep" that happens in any project)
When you look at workflow systems, while planning your application, you will notice, that there are 2 types of data: the flow state/sequence and the specific request data. The flow state is stuff like: what's the current status, what are the potential decisions (usually: Yes, No, More info please), who are the past, current, future approvers. The specific request data is stuff like: duration and type of leave, price of items to purchase, customer involved etc.
In your existing applications the two are stored together (and will stay like that), but you need to extract the flow data to your new approvalCentral application. On the back of a napkin you draw this sequence:
Your approval workflow
(Image created with JS Sequence Diagrams and postprocessed with InkScape)

The green part of the application exists, the first question is: how to design the submission. There are a number of options you could choose from:
  • Have the Approval Central application poll participating applications on schedule. That option is least efficient, but might be your only choice if you can't touch the existing apps at all (happens more often than you think, when admins are paranoid policies are set very cautious
  • Have each application create a flow document directly in the workflow central application. You could use a inherited LotusScript library for that. While that might have been the prefered choice a decade ago, it has two problems: first you limit your participating applications to Notes only (and there are other applications with decision requirements) and you would expose the inner workings of your application prematurely
  • Use a web service interface to implement Contract first development. While REST is the current IT fashion, you pick SOAP. The simple reason: LotusScript can talk SOAP quite well via the Webservice consumer design element, while REST would require your notification code being written in Java. So a SOAP service it is

Read more

Posted by on 2014-11-17 10:13 | Comments (1) | categories: XPages

{{Mustache}} Helper for Domino and XPages

Previously we had a look, how to use Mustache with the CKEditor to have an editing environment for templates. I glossed over the part where to store and how to use these templates. Let me continue there.
I'll store the template in Notes documents and use an application managed bean to transform a document (and later other things) using these templates.
I have 2 use cases in mind: one is to allow the configuration of HTML notification messages in applications and the other to configure the display of workflow details (more on Workflow in a later post). The approach has two parts:
  1. loading (or reloading) the existing templates
  2. using the templates
The first one should need to run only once in the application lifecycle. Since managed beans can't have constructors with parameters, I created a seperate load function, so inside the bean there is nothing that depends on the XPages runtime environment. You therefore can use that bean in an agent, application or a plug-in.
I also opted to load all templates into memory at once, but only compile them when actually needed. If you plan to have lots of them, you want to look for a different solution.
The class that does the work is the MustacheHelper.java. To make it easy to use, I wrap it into a managed bean and a small SSJS Helper function:
<?xml version="1.0" encoding="UTF-8"?>
    <!--  In a production system that should be application -->
  <!--AUTOGEN-START-BUILDER: Automatically generated by IBM Domino Designer. Do not modify.-->
  <!--AUTOGEN-END-BUILDER: End of automatically generated section-->

function applyTemplate(doc, templateName) {
	try {
		// Check for init
		if (!Mustache.isInitialized()) {
			var templateView:NotesView = database.getView("templates");
			Mustache.initFromView(session, templateView);
		return Mustache.renderDocumentToString(templateName,doc);
	} catch (e) {
		return e.message;

With this done transforming a document becomes a one liner in SSJS: return applyTemplate(document1.getDocument(),"Color");. To demo how it works, I created a sample database.

Read more

Posted by on 2014-11-16 05:44 | Comments (0) | categories: XPages

Karma and Wealth

Karma has gotten some attention lately, while I muse it for some time. The common short explanation is " What goes around comes around" or in other words: " Every action (or inaction) you take has consequences, you ultimately will be confronted with". We are reminded by various spiritual traditions , that nothing good can come from a bad deed and that good deeds will yield (in mysterious ways, somewhen) good results.
This is where the trouble starts. Our and the mystic perceptions what is "good" and what is "bad" differ greatly. Pop literature links wealth, affluence and influence to "good Karma". Someone poor or suffering easily gets dismissed as "(s)he has bad Karma".
I think that misses the point completely.
Taking a step back: Buddhist (and others in similar ways) believe we are trapped in a cycle of Samsara that leads us through many lifetimes. Ending Samsara and suffering is the goal of enlightenment (I'm simplifying here). The main force holding us in Samsara is Maya: the illusion of existence. Now adding nice things to our life binds us deeper to Maya, making liberation more remote, so I doubt that this is a good thing per se. And happiness somehow works differently anyway.
If luxury would be the answer, the road to enlightenment would lead through the god realm, which Buddhists believe is a detour. Some scholars argue that our western civilization is god realm like (I'll add a link when I rediscover it) and there's leisure displaying compassion and joyful effort.
Looking at it from a different angle might explain it better: The currency of Karma is compassion. Compassion for all living beings. That includes yourself, so no point moving under the bridge since "it isn't real". Looking after yourself is a requirement, so you can sustainable look after others. IMHO Good Karma is what makes it easier to be compassionate. Your good deeds will make it easier for you in the future to make good deeds. Any ulterior motive might disqualify your actions as good deeds. So if you think improvements in your financial situation are the result of good Karma, you mix cause and effect (which anyway only exist interdependent). An improvement in your personal situation isn't the reward, but the enablement for greater compassion - and it makes you happy for a while, happy people are contagious.
This also reconciles Karma and free will: Contrary to the common perception "It was Karma, that this happened" you end up with "Life offered a situation, I made a decision, now I'm presented with the consequences". Of course all consequences turn into offerings of situations. I think it is a folly to conclude: hardship is automatically an indication of bad Karma (it might).
The best analogy: there is a weight of 100kg you are supposed to lift (quite a hardship for most of us)! So what's the conclusion? Bad Karma? Nope: if you have trained hard, that might be the final test and reward that you mastered your training and you will lift it. Same with life: a difficult situation could be anything: a result of a bad deed or an invitation to show your skilfulness: maintain compassion no matter what.
In the words of beloved teacher: " Life dealt you cards, you make your Karma how you play them"

Posted by on 2014-11-16 12:30 | Comments (0) | categories: After hours Thoughts