wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

Running vert.x with the OpenNTF Domino API


In the first part I got vert.x 3.0 running with my local Notes client. The mastered challenges there were 32 Bit Java for the Notes client and the usual adjustment for the path variables. The adoption of the OpenNTF Domino API required a few steps more:
  1. Set 2 evironment variables:
    DYLD_LIBRARY_PATH=/opt/ibm/notes
    LD_LIBRARY_PATH=/opt/ibm/notes
  2. Add the following parameter to your Java command line:
    -Dnotes.binary=/opt/ibm/notes -Duser.dir=/home/stw/lotus/notes/data -Djava.library.path=/opt/ibm/notes
    (Of course you will adjust the path to your environment, will you?)
  3. Add 4 JAR files to the classpath of your project (or runtime):
    • /opt/ibm/notes/jvm/lib/ext/Notes.jar
    • /opt/ibm/notes/framework/rcp/eclipse/plugins/com.ibm.icu.base_3.8.1.v20080530.jar
    • org.openntf.domino.jar
    • org.openntf.formula.jar
    I used the latest build of the later two jars from Nathan's branch, so make sure you have the latest. The ICU plug-in is based on the International Components for Unicode project and might get compiled into a future version of the Domino API.
Now the real fun begins. The classic Java API is conceptually single threaded with all Domino actions wrapped into NotesThread.sinitThread(); and NotesThread.stermThread();. For external applications (the ones running neither ax XPage/OSGi nor as agent, the OpenNTF API provides the Domino Executor.

Read more

Posted by on 2014-08-07 01:18 | Comments (0) | categories: IBM Notes vert.x

Workflow for beginners, Standards, Concepts and Confusion


The nature of collaboration is the flow of information. So naturally I get asked about Workflows and its incarnation in IT systems a lot. Many of the question point to a fundamental confusion what Worflow is, and what it isn't. This entry will attempt to clarify concepts and terminology
Wikipedia sums it up nicely: "A workflow consists of an orchestrated and repeatable pattern of business activity enabled by the systematic organization of resources into processes that transform materials, provide services, or process information. It can be depicted as a sequence of operations, declared as work of a person or group,[2] an organization of staff, or one or more simple or complex mechanisms".
Notably absent from the definition are: "IT system", "software", "flowchart" or "approval". These are all aspects of the implementation of a specific workflow system, not the whole of it. The Workflow Management Coalition (WfMC) has all the specifications, but they might appear as being written in a mix of Technobabble and Legalese, so I sum them up in my own words here:
  • A workflow has a business outcome as a goal. I personally find that a quite narrow definition, unless you agree: "Spring cleaning is serious business". So I would say: a collection of steps, that have been designed to be repeatable, to make it easier to achieve an outcome. So a workflow is an action pattern, the execution of a process. It helps to save time and resources, when it is well designed and can be a nightmare when mis-fitted
  • A workflow has an (abstract) definition and zero or more actual instances where the workflow is executed. Like: "Spring cleaning requires: vacuuming, wiping, washing" (abstract). "Spring cleaning my apartment on March 21, 2014" (actual). Here lies the first challenge: can - and how much - a workflow instance deviate from the definition. How have cases to be handled when the definition changes in the middle of the flow execution? How to handle workflow instances that do require more or less steps? When is a step mandatory for regulatory compliance?
  • A workflow has one or more actors. In a typical approval workflow the first actor is called requestor, followed by one or more approvers. But actors are not limited to humans and the act of approving or rejecting. A workflow actor can be a piece of software that adds information to a flow based on a set of criteria. A typical architecture for automated actors is SOA
  • Workflow systems have different magnitudes. The flagship products orchestrate flows across multiple independent systems and eventually across corporate boundaries, while I suspect that the actual bulk of (approval) flows runs in self contained applications, that might use coded flow rules, internal or external flow engines
  • On the other end of scale eMail can be found, where the flow and sequence are hidden in the heads of the participants or scribbled into freeform text
  • Workflows can be described in Use Cases, where the quality depends on the completeness of the description, especially the exception handling. A lot of Business Process Reengineering that is supposed to simplify workflows fails due to incomplete exception handling and people start to work "around the system" (eMail flood anyone?)
  • A workflow definition has a business case and describes the various steps. The number of steps can be bound by rules (e.g. "the more expensive, the more approvers are needed" or "if the good transported is HazMat approval by the environmental agency is needed") that get interpreted (yes/no) in the workflow instance
  • Determine the next actor(s) is a task that combines the workflow instance step with a Role resolver. That's the least understood and most critical part of a flow definition. Lets look at a purchase approval flow definition: "The requestor submits the request to approval by the team lead, to be approved by the department head for final confirmation by the controller". There are 4 roles to resolve. This happens in context of the request and the organisational hierarchy. The interesting question: if a resolver returns more than one person, what to do? Pick one randon, round robin or else how?
  • A role resolver can run on submission of a flow or at each step. People change roles, delegate responsibilities or are absent, so results change. Even if a (human) workflow step already has a person assigned, a workflow resolver is needed. That person might have delegated a specific flow, for a period (leave) or permanently (work load distribution). So Jane Doe might have delegated all approvals below sum X to her assistant John Doe (not related), but that doesn't get reflected in the flow definition, only in the role resolution
  • Most workflow systems gloss over the importance of a role resolver. Often the role resolver is represented by a rule engine, that gets confused with the flow engine. Both parts need to work in concert. We also find role resolution coded as tree crawler along an organisational tree. Role resolving warrants a complete post of its own (stay tuned)
  • When Workflow is mentioned to NMBU (normal mortal business users), then two impressions pop up instantly: Approvals (perceived as the bulk of flows) and graphical editors. This is roughly as accurate as "It is only Chinese food when it is made with rice". Of course there are ample examples of graphical editors and visualizations. The challenge: the shiny diagrams distract from role definitions, invite overly complex designs and contribute less to a successful implementation than sound business cases and complete exception awareness
  • A surprisingly novel term inside a flow is SLA. There's a natural resistance, that a superior (approver) might be bound by an action of a subordinate to act in a certain time frame. Quite often making the introduction of SLA part of a workflow initiative, provides an incentive to look very carefully to make processes complete and efficient
  • Good process definitions are notoriously hard to write and document. A lot of implementations suffer from a lack of clear definitions. Even when the what is clear, the why gets lost. Social tools like a wiki can help a lot
  • A good workflow system has a meta flow: a process to define a process. That's the part where you usually get blank stares
  • Read a one one or other good book to learn more
There is more to say about workflow, so stay tuned!

Posted by on 2014-07-24 10:02 | Comments (0) | categories: Software Workflow

Warriors of Light


Inspired by Paulo Coelho's manual for the Warrior of the Light:

Warriors of Light
We were born from the stars
Descended from the heavens
Armed with compassion
Determined to end the suffering
Subjected to the human condition
Battling ignorance with wisdom
Laying our lives for the liberation from illusion

When you look in the mirror - remember!
You are one of us.

Posted by on 2014-07-20 11:57 | Comments (0) | categories: After hours

Adventures with vert.x, 64Bit and the IBM Notes client


The rising star of web servers currently is node.js, not at least due to the cambrian explosion in available packages with a clever package management system and the fact that "Any application that can be written in JavaScript, will eventually be written in JavaScript" (according to Jeff Atwood).
When talking to IBM Domino or IBM Connections node.js allows for very elegant solutions using the REST APIs. However when talking to a IBM Notes client, it can't do much since an external program needs to be Java or COM, the later on Windows only.
I really like node.js event driven programming model, so I looked around. In result I found vert.x, which does to the JVM, what node.js does to Google's v8 JS runtime. Wikipedia decribes vert.x as "a polyglot event-driven application framework that runs on the Java Virtual Machine ". Vert.x is now an Eclipse project.
While node.js is tied to JavaScript, vert.x is polyglot and supports Java, JavaScript, CoffeeScript, Ruby, Python and Groovy with Scala and others under consideration.
Task one I tried to complete: run a verticle that simply displays the current local Notes user name. Of course exploring new stuff comes with its own set of surprises. As time of writing the stable version of vert.x is 2.1.1 with version 3.0 under heavy development.
Following the discussion, version 3.0 would introduce quite some changes in the API, so I decided to be brave and use the 3.0 development branch to explore.
The fun part: there is not much documentation for 3.x yet, while version 2.x is well covered in various books and the online documentation.
vert.x 3.x is at the edge of new and uses Lamda expressions, so just using Notes' Java 6 runtime was not an option. The Java 8 JRE was due to be installed. Luckily that is rather easy.
The class is rather simple, even after including Notes.jar, getting it to run (more below) not so much:
package com.notessensei.vertx.notes;

import io.vertx.core.Handler;
import io.vertx.core.Vertx;
import io.vertx.core.http.HttpServerOptions;
import io.vertx.core.http.HttpServerRequest;
import io.vertx.core.http.HttpServerResponse;

import java.io.IOException;

import lotus.domino.NotesFactory;
import lotus.domino.NotesThread;
import lotus.domino.Session;

public class Demo {
	public static void main(String[] args) throws IOException {
		new Demo();
		int quit = 0;
		while (quit != 113) { // Wait for a keypress
			System.out.println("Press q<Enter> to stop the verticle");
			quit = System.in.read();
		}
		System.out.println("Veticle terminated");
		System.exit(0);
	}

	private static final int listenport = 8111;

	public Demo() {
		Vertx vertx = Vertx.factory.createVertx();
		HttpServerOptions options = new HttpServerOptions();
		options.setPort(listenport);
		vertx.createHttpServer(options)
				.requestHandler(new Handler<HttpServerRequest>() {
					public void handle(HttpServerRequest req) {
						HttpServerResponse resp = req.response();
						resp.headers().set("Content-Type",
								"text/plain; charset=UTF-8");
						StringBuilder b = new StringBuilder();
						try {
							NotesThread.sinitThread();
							Session s = NotesFactory.createSession();
							b.append(s.getUserName());
							NotesThread.stermThread();
						} catch (Exception e) {
							e.printStackTrace();
							b.append(e.getMessage());
						}
						resp.writeStringAndEnd(b.toString());
					}
				}).listen();
	}
}

Starting the verticle looked promising, but once I pointed my browser to http://localhost:8111/ the fun began.

Read more

Posted by on 2014-07-17 06:58 | Comments (1) | categories: IBM Notes Software vert.x

Cycle where?


I like to cycle, I do that often and from time to time I have fun with traffic participants. One of the interesting challenges are multi-lane crossings (note to my overseas readers: Singapore follows the British system of driving on the left, so cyclists are supposed to cycle on the left edge of the road - which makes me edgy in some situations. So for right driving countries, just flip the pictures) where the outer lane allows more than one direction. Like these:

Empty road
Road rules do require the bike to stay on the left (I didn't find the cycle symbol in SmartDraw, so I used a motorbike, just picture me on a bike there).
Staying right
What then happens, I've seen that quite often, is a vehicle closing up next to the cyclist. After all the reason why we stay on the left is not to obstruct other traffic. But that's Roadkill waiting to happen. I've been kicked off a motorbike once (got my shoulder dislocated) because a car felt it is ok to turn right when I was right of it. So I'm sensitive to this problem:
Roadkill waiting
As a result, when there's more than one lane in the direction I need to go and there is ambiguity where traffic in the same lane might be going, I make sure, that it won't happen by occupying the same space as "the big boys". I however pick my trajectory in a way that I end up at the left edge once I cleared the crossing:
Learning new Hokkien
Unsurprisingly some of the motorist aren't happy, after all they loose 2-3 seconds on their journey. So I wonder: is there a better way? Is that behaviour compliant with traffic rules? What do all these rude sounding Hokkien terms mean I do hear?

Posted by on 2014-07-14 01:16 | Comments (3) | categories: After hours

From XML to JSON and back


In the beginning there was csv and the world of application neutral (almost) human readable data formats was good. Then unrest grew and the demand for more structure and contextual information grew. This gave birth to SGML (1986), adopted only by a few initiated.
Only more than a decade later (1998) SGML's offspring XML took centre stage. With broad support for schemas, transformation and tooling the de facto standard for application neutral (almost) human readable data formats was established - and the world was good.
But bracket phobia and a heavy toll, especially on browsers, for processing led to to rise of JSON (starting ca. 2002), which rapidly became the standard for browser-server communication. It it native to JavaScript, very light compared to XML and fast to process. However it lacks (as time of writing) support for an approved schema and transformation language (like XSLT).
This leads to the common scenario that you need both: XML for server to server communication and JSON for in-process and browser to server communication. While they look very similar, XML and JSON are different enough to make transition difficult. XML knows elements and attributes, while JSON only knows key/value pairs. A JSON snippet like this:
{ "name" : "Peter Pan",
  "passport" : "none",
  "girlfriend" : "Tinkerbell",
  "followers" : [{"name" : "Frank"},{"name" : "Paul"}]
}

can be expressed in XML in various ways:
<?xml version="1.0" encoding="UTF-8"?>
<person name="Peter Pan" passport="none">
    <girfriend name="Tinkerbell" />
    <followers>
        <person name="Frank" />
        <person name="Paul" />
    </followers>
</person>

.or.
<?xml version="1.0" encoding="UTF-8"?>
<person>
    <name>Peter Pan</name>
    <passport>none</passport>
    <girfriend>
        <person>
            <name>Tinkerbell</name>
        </person>
    </girfriend>
    <followers>
        <person>
            <name>Frank</name>
        </person>
        <person>
            <name>Paul</name>
        </person>
    </followers>
</person>

.or.
<?xml version="1.0" encoding="UTF-8"?>
<person name="Peter Pan" passport="none">
    <person name="Tinkerbell" role="girfriend" />
    <person name="Frank" role="follower" />
    <person name="Paul" role="follower" />
</person>

(and many others) The JSON object doesn't need a "root name", the XML does. The other way around is easier: each attribute simply becomes a key/value pair. Some XML purist see attributes as evil, I think they do have their place to make relations clearer (is-a vs. has-a) and XML less verbose. So transforming back and forth between XML and JSON needs a "neutral" format. In my XML Session at Entwicklercamp 2014 I demoed how to use a Java class as this neutral format. With the help of the right libraries, that's flexible and efficient.

Read more

Posted by on 2014-07-12 09:09 | Comments (1) | categories: Software

The folly of root cause analysis


IT support's dealing with management is a funny business. Whenever something goes wrong, support teams engage in "defensive blaming" and the elusive quest for a root cause.
I've seen this quest (and blaming god and country along the way if it doesn't appear) taking priority over problem resolution and prevention. The twisted thought is: "If I'm not sure about the (single) root cause, I can't neither fix nor prevent it from happening again".

Why is that a folly?
  • It paralyses: If a person bleeding enters an ER, the first call to action is to stop the bleeding. Asking: Have all the suppliers of the manufacturer of the blade that caused it have ISO 9000 certifications? Where was the ore mined to make the blade? Root cause analysis is like that. IT support however is an ER room
  • It is not real: There is never a single reason. Example: Two guys walk along the street in opposite directions. One is distracted, because he is freshly in love. The other is distracted because he just was dumped. They run into each other and bump their heads. What is the root cause for it?
    You remove a single factor: route chosen, fallen in love, being dumped, time of leaving the house, speed of walking, lack of discipline to put attention ahead of emotion etc. and the incident never would have happened.
    If I apply an IT style root cause analysis it will turn out: it was the second guy's grandfather who is the root cause: He was such a fierce person, so his son never developed trust, which later in the marriage led to a breakup while guy two was young, traumatising every breakup for him - thus being distracted
  • There is more than one: as the example shows: removing one factor could prevent the incident to happen, but might leave a instable situation. Once a root cause has been announced, the fiction spreads: "everything else is fine". Example: A database crashes. The root cause gets determined: only 500 users can be handled, so a limit is introduced. However the real reason is a faulty hardware controller that malfunctions when heating up, which happens on prolonged high I/O (or when the data center aircon gets maintained).
    The business changes and the application gets used heavier by the 500 users, crossing the temperature threshold and the database crashes again.
  • Cause and effect are not linear: Latest since Heisenberg it is clear that the world is interdependent. Even the ancient Chinese knew that. So it is not, as Newton discovered, a simple action/reaction pair (which suffers from the assumption of "ceteris paribus" which is invalid in dynamic systems), but a system of feedback loops subject to ever changing constraints.
Time is better spend understanding a problem using system thinking: Assess and classify risk factors and problem contributors. Map them with impact, probability, ability to improve and effort to action. Then go and move the factors until the problem vanishes. Don't stop there, build a safety margin.
As usual YMMV

Posted by on 2014-07-09 01:45 | Comments (2) | categories: Software

Documents vs eMails


With a public sector customer I had an interesting discussion on non-repudiation, messaging and regulatory control. We were discussing how to ensure awareness of information that has behavioural or legal consequences. While "I didn't know" is hardly a viable defence, relying on the other party to keep themselves updated is just asking for trouble. In a collaborative environment, where a regulator sees itself primarily as the facilitator of orderly conduct and only as policing the conduct as secondary mission, this is inefficient.
An efficient way is a closed loop system of information dissemination and acknowledgement. The closed loop requirement isn't just for regulators, but anybody that shares information resulting in specific behaviour. Just look at the communication pattern of a pilot with air traffic control (paraphrased): Tower: "Flight ABC23 turn to runway 270, descend to 12 thousand feet" - Pilot: "Roger that, turning to 270, descent to 12 thousand"

When we look at eMail, the standard mechanism that seems to get close to this pattern are Return Receipts:
Standard eMail flow
Using RFC3798 - Message Disposition Notification - MDN (commonly referred to as Return Receipt) to capture the "state of acknowledgement", is a folly.
  1. the RFC is completely optional and a messages system can have it switched off (or delete it from the outbox), so it isn't suitable as guarantee
  2. MDN only indicates that a message has been opened. It does not indicate: was it read, was it understood, were the actions understood, was the content accepted (the later one might not be relevant in a regulatory situation). It also doesn't - which is the biggest flaw - indicate what content was opened. If the transmission was incomplete, damaged or intercepted a return receipt wouldn't care.
So some better mechanism is needed!
Using documents that have a better context a closed loop system can be designed. When I say "document" I don't mean: propriety binary or text format file sitting in a file system, but entity (most likely in a database) that has content and meta information. The interesting part is the meta information:
  • Document hierarchy: where does it fit in. For a car manufacturer recalling cars that could be: model, make, year. For a legislator the act and provisions it belongs in
  • Validity: when does it come into effect (so one can browse by enactment date), when does (or did) it expire
  • History: which document(s) did it supersede, which document(s) superseded it
  • Audience: who needs to acknowledge it and how fast. Level of acknowledgement needed (simple confirmation or some questionnaire)
  • Pointer to discussion, FAQ and comments
  • Tags
An email has no structured way to carry such information forward. So a document repository solution is required. On a high level it can look like this:
Document Flow to acknowledge
Messaging is only used for notification of the intended audience. Acknowledgement is not an automatic, but a conscious act of clicking a link and confirming the content. The confirmation would take a copy of original text and sign it, so it becomes clear who acknowledged what. An ideal candidate would be XML Signature, but there isn't a model how to sign that from a browser. There is an emerging w3C standard for browser based Crypto, that has various level of adoption: Once you have dedicated records who has acknowledged a document, you can start chasing, reliable and automated, the missing participants and, if you are a regulator, take punitive actions when chasing fails. It also opens the possibility to run statistics how fast what type of documents get adopted.
The big BUT usually is: I don't want to deploy additional 2 servers for document storage and web access. The solution for that is, you might have guessed it, is Domino. One server can provide all 3 roles easily:
Document Flow on a Domino server
As usual YMMV

Posted by on 2014-07-04 11:41 | Comments (0) | categories: Software

The taxi loyalty program isn't working and how to fix it


Singapore is a little like New York: train and taxis are a mainstay of the daily commute. So the taxi market is highly regulated and fiercely competitive. As no surprise taxi companies try to bind customers before they loyalty switches to alternative bookings or the disruptors.
So Comfort & CityCab started CabRewards. After all loyalty cards work well for their inventor.
In a smart move, instead of creating a new piece of plastic, Comfort teamed up with ezLink Singapore's leading provider of cash cards. Everyone in Singapore has a ezLink card, since they are used for train access and road tolls.
From there it all went downhill.
In Usability studies one of they key activities is to watch users and refine on their feedback. So I loaded my ezLink card, to see how Taxi drivers will handle it. In (so they told me, after I inquired with Comfort) an attempt (a futile one) to make things "consistent", the designer decided to add a prompt into the touchscreen application "Cabrewards Yes/No" before the driver can process payment. So something the driver has no benefit from stands between him and his livelihood. To no surprise, 95% of the drivers don't bother to ask "Are you a CabRewards member?", even when I announce "I will pay by ezLink". They just click NO and process to payment. If they would answer yes, they had to tap once for the points and another time for the actual payment. They have no benefit, so they skip it (and rightly so, their job is driving, not administration of loyalty programs).
I asked Comfort and they explained, I could ask the Taxi driver to switch back and add the points - but I'm not their program admin either. So how to fix this?
Plan, Reality and Fix
The graphic above, shows the intended workflow, the actual workflow and a possible fix. There are several touchpoints, where an automated IT system can determine if I'm a known passenger and a CabRewards member: I call a cab, I uase a mobile app to get a cab or I pay with a registered method of payment (ezLink, Nets, CreditCard). In all of these cases the taxi driver doesn't need to be bothered with a question. Most likely that also would cover most reward members. The ones paying cash might anyway not have registered for the program - or a simple change of terms (points only with electronic payments) would even completely eliminate the question. So far I haven't seen a change. I wonder if the person in charge of the process is trying to cover up the problem?
I would say: It is OK to have an idea, even if problems arise, just fix them. Despite other opinion there is no such thing as a "honest mistake". There is trial, error and correction (and start over).
Simplicity after all isn't simple
Comfort, you can do better!

Posted by on 2014-06-16 10:43 | Comments (1) | categories: Business Singapore

Let's ditch IBM Notes and Domino


Finally you decided it is time to move on, legacy no longer means "tried and tested" but "we need to move one" to you. After all you never really liked Cher.
Notes data is available via LotusScript, dotNet (through the COM bridge), in Java, Corba, C++, XML, REST, MIME, so how hard can it be? Actually not very hard, you just need to:
  1. Find sutiable replacement application platform(s)
  2. Rewrite the applications (don't dream: there is no such thing as "migrate an app")
  3. Migrate your users
  4. Migrate your data
... and then live happily ever after. Probably the easiest part is migrating the users, after all directories are more or less the same when you look at the LDAP side of them.
Migrating the users' encryption keys and secured documents could be considered "an application function", so you dodge a potential problem here.
Getting the data out, short of that pesky RichText content (yeah, the one with sections and embedded OLE controls), is easy - never mind reader and author access protection.
The transformation into the target format happens in a black box (the same the local magician uses for his tricks), you buy from a service provider, so that's covered, leaves platform and applications. (Bring your own <irony /> tags)
Lets have a look at the moving parts (Listing OpenSource components only, you can research commercial ones on your own):
What Domino does for you
  • Directory : OpenLDAP comes with most Linux distributions and is well understood. It doesn't take you down the propriarty extension hell (and you can remove obsolete schema parts) - it requires an RDBMS to run (not a blue one)
  • PKI : PrimeKey
  • Sync Server : For the database you could get away with the database, but you need one for mobile: Z-Push
  • Document Server : Pick any - or go with: Dropbox, Box, Copy etc.
  • Database Server : Notes is the mother of NoSQL, so you want to pick one of those: CouchDB, OrientDB or you go with DB/2 PureXML
  • Application Server : XPages is full fledged JSF, you you might opt for TomEE, but you will need some extra libraries. Or you ditch Java and adopt the cool kid on the block
  • Mail Server : the venerable SendMail does, as the name implies, send mail. Goes nicely with Mozilla Thunderbird
  • Web Server : That's easy. There's NGinX or Apache
  • Enterprise Integration : Might be quite tricky. Domino's DECS and LEI shuffle data from all sorts of sources. OpenESB might do the trick
  • Task scheduling : You can use OS level CRON (Just find a nice UI) or application level Quartz
Keep in mind, this replaces one server - Domino. Each of the tools needs to be:
  1. configured
  2. hardened
  3. populated with data
The ease of clustering you had with Domino, gets replaced with different methods and capabilities for each of the tools. Today Cloud is the new infrastructure, so you might get lucky with someone else configuring all of the above for you.

Once you got all the moving parts in place, you need to redevelop your apps. Don't be fooled, but run the analysis, to see the full magnitude of that task ahead.
As usual YMMV

Posted by on 2014-05-30 07:05 | Comments (7) | categories: IBM Notes