wissel.net

Usability - Productivity - Business - The web - Singapore & Twins

By Date: May 2017

From Blogsphere to a Static Site (Part 4) - Comment backend


The blog needed a comment function. While there are social options around (Facebook, Disqus etc), I decided I want to roll my own. Partly because I want tighter control and partly, well, because I could. My comment backend would:

  • Provide a REST API to create comments in a JSON structure. The comment body will be Markdown. Reading would provide comments in ready to use HTML (I hear howling from the API crowd). No delete or update functionality
  • Cleanup content considered harmful (code injection) and optional sport Captcha
  • Store all content in a NoSQL database, in my case CouchDB (or Cloudant with its 20G free plan)
  • Cache all queries for comment in an online cache to limit calls to the database
  • Initially run on Domino, later on liberty or the raw JVM
  • Initially also update Domino using a web service - so during transition no comments would get lost

In its initial incarnation the Comment servlet is a OSGi plugin that listens to the /comments URL implemented as Wink servlet. So the class of interest is the one defining the service. We have one method for post, one for get and a helper function


/**
 * Wink implementation of Comment service
 */
@Workspace(workspaceTitle = "Blog Comments", collectionTitle = "Create or display comments")
@Path(value = "/comments")
@Produces(MediaType.TEXT_HTML)
@Consumes(MediaType.APPLICATION_JSON)
public class CommentService extends CommentResponse {

    private final Logger logger = Logger.getLogger(this.getClass().getName());

    @POST
    public Response createComment(@Context HttpServletRequest request) {
        final Monitor mon = MonitorFactory.start("CommentService#createComment");
        String result = "Sorry I can't process your comment at this time";
        ResponseBuilder builder = Response.ok();

        try {
            InputStream in = request.getInputStream();
            BlogComment comment = BlogComment.load(in);
            in.close();

            if (comment != null) {
                this.captureSubmissionDetails(request, comment);
                result = CommentManager.INSTANCE.saveComment(comment, true);
            } else {
                builder.status(Status.NOT_ACCEPTABLE);
            }
            builder.entity(result).type(MediaType.TEXT_HTML_TYPE);

        } catch (Exception e) {
            String errorMessage = e.getMessage();
            builder.entity((((errorMessage == null) || errorMessage.equals("")) ? "Undefined error" : errorMessage)).type(
                    MediaType.TEXT_HTML_TYPE);
            Utils.logError(this.logger, e);
        }

        mon.stop();
        return builder.build();
    }

    @GET
    public Response getComments(@QueryParam("parentid") final String parentid) {
        Response response = null;
        final Monitor mon = MonitorFactory.start("CommentService#getComments");
        final ResponseBuilder builder = Response.ok();
        final Collection<BlogComment> bc = CommentManager.INSTANCE.loadComments(parentid);

        if ((bc == null) || bc.isEmpty()) {
            builder.status(Status.NO_CONTENT);
        } else {
            response = this.renderOutput(bc, "comment.mustache");
        }
        mon.stop();
        return (response == null) ? builder.build() : response;
    }

    private void captureSubmissionDetails(HttpServletRequest request, BlogComment comment) {
        final Monitor mon = MonitorFactory.start("CommentService#captureSubmissionDetails");
        try {
            @SuppressWarnings("rawtypes")
            Enumeration hn = request.getHeaderNames();
            if (hn != null) {
                while (hn.hasMoreElements()) {
                    String key = hn.nextElement().toString();
                    comment.addParameter(key, request.getHeader(key));
                }
            }
            @SuppressWarnings("rawtypes")
            Enumeration pn = request.getParameterNames();
            if (pn != null) {
                while (pn.hasMoreElements()) {
                    String key = pn.nextElement().toString();
                    String[] values = request.getParameterValues(key);
                    comment.addParameters(key, values);

                    if (key.equals("referer")) {
                        comment.setReferer(values[0]);
                    } else if (key.equals("user-agent")) {
                        comment.setUserAgent(values[0]);
                    }

                }
            }
            @SuppressWarnings("rawtypes")
            Enumeration an = request.getAttributeNames();
            if (an != null) {
                while (an.hasMoreElements()) {
                    try {
                        String key = an.nextElement().toString();
                        comment.addAttribute(key, String.valueOf(request.getAttribute(key)));
                    } catch (Exception e) {
                        // No action here
                    }
                }
            }
            comment.addParameter("REMOTE_HOST", request.getRemoteHost());
            comment.addParameter("REMOTE_ADDR", request.getRemoteAddr());
            comment.addParameter("REMOTE_USER", request.getRemoteUser());

            // Needed for Captcha
            comment.setRemoteAddress(request.getRemoteAddr());
        } catch (Exception e) {
            Utils.logError(this.logger, e);
            // But no further action here!
        }
        mon.stop();
    }

}

Read more

Posted by on 04 May 2017 | Comments (2) | categories: Blog

The Decline and Fall of IBM


I wrote this quite a while ago, never finished the article until now. Enjoy.

Yeah right!

With this words Robert X Cringley a.k.a Mark Stephens celebrates himself when he asserts something clever. His book The Decline and Fall of IBM created quite some stir and was a hot topic of discussion especially among IBMers and alumni.
So I got myself a copy and had a look. Clearly he has an axe to grind with IBM and everybody is invited. Many came. Half of the book consists of mostly grieving comments ranging from 2007 until 2014. With all this contributions, the content remains light on substance. After all it costs you less than a late.

Some of the stated observations are spot on, like "IBM is a sales organisation", others while looking like observations, but rather are opinions (e.g. "IBM lost its way") and statements that made me feel: "why didn't the IBM board ask him to run IBM? He seems to know so much better!", exactly like an arbitrary spectator of the world cup in a pub can tell you what a team did wrong.

I'm not saying, that all is well in IBM, that would be a fools view. An organisation with a size and workforce exceeding several countries does live in challenging times. An economic system, that values growth over everything is problematic (In biology an organism that grows indefinitely is called cancer) at least and transiting to a global workforce on this scale is unparalleled. I share his view that the dance around the golden calf known as shareholder value might be praying to a false good.
Here are some comments to Cringley's statements, that crossed my mind (paraphrasing):

  • Cringley: "In IBM management is royalty, the sales force the nobility and technical people are the peasants. No peasant can dream to become royalty".
    Yeah right! IBM fellows, IBM Distinguished Engineers or members of the IBM Academy of Technology will disagree. The irony here: in the "good old IBM (of Watson)" there was no career path for technical experts. In the IBM of today there is
  • I worked a lot with our engineers in India and China. They are hard working, ready to learn and, by now, quite experienced. Asserting that they are less capable that their American counterparts, seems quite arrogant to me. Yes, they were unexperienced a decade ago, but that's a long time in IT years. Also: there are capable and incapable engineers everywhere. Pinning it on a specific country or region is (insert your own statement of backwards here). A real issue however are IBM's processes, that had been designed to cater to get less experienced people on board. They need an overhaul.
  • Cringley: "IBM should not sell the Intel servers to Lenovo".
    Yeah right! Cringley portrays it as the complete exit out of a server growth market. However IBM still has Intel based technology in their PureSystems and acquired skills and know how through the Softlayer acquisition how to build the special segment of Intel based machines that run in cloud size data centres. So instead of exiting a segment, it looks to me like eliminating duplicate product lines
  • Cringley: "IBM should port AIX to Intel".
    Yeah right! AIX runs on one class of IBM machines (System P), while Linux runs on everything from Softlayer Bluemix to Mainframe. Linux outperforms AIX on System P in quite some workloads. So what makes a better investment? Port AIX or infuse the security know how of AIX into Linux? (the file system options are there already)
  • Cringley: "IBM should ditch the Power architecture and switch to Intel"
    Yeah right! IBM has vast know how in building processors that will get lost when stopping to develop them. Asset utilisation isn't ditching assets, but making them more competitive. Intel and ARM aren't the only shops who can design processors, just have a look at OpenPOWER
  • Cringley: Hadoop will make mainframes obsolete.
    Yeah right! The technology is around for a decade, Google, its inventor, already moved on. Asserting a single technology will kill the I/O beasts known as Big Iron, looks like an inflight magazine statement. Besides the fact, that IBM offers a robust Hadoop implementation for a while already, which you can run on zLinux, if you choose so.
  • The book contains quite some areas, like current staff morale, the process culture or the layers of management that highlight pain. However where he is wrong: IBM isn't blind to the issues and there are forces inside working for the betterment (even if there is a case of a Knowing-Doing-Gap)
Of course, YMMY, so read it for yourself

Posted by on 01 May 2017 | Comments (1) | categories: IBM

From Blogsphere to a Static Site (Part 3) - Generating pages


The rendering engine I choose is mustache which describes itself as "logic-less templating engine". My main criteria was the availability on multiple platforms including Java and JavaScript (I might port the rendering part to NodeJS at some time in the future).
The only logic mustache supports is conditional rendering based on the presence or absence of an element. When an element is present and is an array (or a collection in Java) the body of the template gets repeated for each element in the array. A scalar value hence is treated as an array with one value only.

Mustache is simple to use. All you need is a data bean (in Java, a JSON structure for JavaScript) and a text file containing placeholders with the property names. E.g. <h1>{{title}}</h1> will render a headline with the title property of you data object. In Java that would be either a public variable or a call to getTitle according to the bean specification. The blog rendering code therefore is quite simple:


private void renderOneEntry(BlogEntry be, Mustache mustache) throws IOException {

        String location = this.config.destinationDirectory + be.getNewURL();
        String outDirs = location.substring(0, location.lastIndexOf("/"));
        File dirs = new File(outDirs);
        if (!dirs.exists()) {
            dirs.mkdirs();
        }
        // Set the current context
        for (LinkItem cat : be.getCategory()) {
            String c = cat.name;
            this.allCategories.get(c.toLowerCase()).active = true;
        }
        this.allDateCategories.get(be.getDateYear()).active = true;

        if (be.getSeries() != null) {
            String series = be.getSeries();
            if (this.allSeries.containsKey(series)) {
                this.allSeries.get(series).get(be.getNewURL()).active = true;
            }
        }

        // Prepare to write out
        ByteArrayOutputStream out = new ByteArrayOutputStream(102400);
        Writer pw = new PrintWriter(out);

        // This is where the magic happens
        mustache.execute(pw, be);
        pw.flush();
        pw.close();
        this.saveIfChanged(out.toByteArray(), location);

        // Cleanup
        for (LinkItem cat : be.getCategory()) {
            String c = cat.name;
            this.allCategories.get(c.toLowerCase()).active = false;
        }
        this.allDateCategories.get(be.getDateYear()).active = false;

        if (be.getSeries() != null) {
            String series = be.getSeries();
            if (this.allSeries.containsKey(series)) {
                this.allSeries.get(series).get(be.getNewURL()).active = false;
            }
        }
    }

##

The actual rendering is just the line mustache.execute(pw, be); The code around it prepares and resets the collections that might render on a page like categories, series or month and year. Also of interest is this.saveIfChanged(out.toByteArray(), location); which only saves results back to disk if it actually has changed. Don't be mistaken: any change in layout will lead to a newly rendered page, so this is quite important to save as needed and not more (you don't want to have tons of identical files that only differ in their time stamp)


    private void saveIfChanged(byte[] newData, String targetName) {

        boolean saveThis = false;

        File targetFile = new File(targetName);
        if (targetFile.isDirectory()) {
            System.out.println("Directory encountered!" + targetName);
        } else if (targetFile.exists()) {
            try {
                InputStream existing = new FileInputStream(targetFile);
                ByteArrayOutputStream compare = new ByteArrayOutputStream(102400);
                ByteStreams.copy(existing, compare);
                // Save if they are not equal..
                saveThis = !Arrays.equals(newData, compare.toByteArray());
                Closeables.close(existing, true);
            } catch (FileNotFoundException e) {
                // Anything goes wrong -> we save the file
                e.printStackTrace();
                saveThis = true;
            } catch (IOException e) {
                // Anything goes wrong -> we save the file
                e.printStackTrace();
                saveThis = true;
            }

            // Now if it is there, get rid of it.
            if (saveThis) {
                targetFile.delete();
            }

        } else {
            saveThis = true;
        }

        if (saveThis) {
            OutputStream finalOut = null;
            try {
                finalOut = new FileOutputStream(targetFile);
                finalOut.write(newData);
                finalOut.flush();
                Closeables.close(finalOut, true);
            } catch (FileNotFoundException e) {
                e.printStackTrace();
            } catch (IOException e) {
                e.printStackTrace();
            }
            System.out.println("\n+" + targetFile);
        } else {
            System.out.print(".");
        }
    }

The template itself is broken into multiple pieces. Since I have different page styles: start page, article, categories, month and year overviews, I have several parts that repeat: headers, footers, side bar elements. Mustache makes that easy to deal with. The constructions I used most:

  • {{somevalue}}: Render somevalue, escape html
  • {{& body}}: Render body, don't escape html
  • {{> snippet}}: Pull in snippet.mustache from disk and use it at template at this position
  • {{#element}}: render if element exists, repeat once per member of collection. End with {{/element}}
  • {{^element}}: render if element doesn't exist or array is empty

With these the actual template becomes rather short and quite maintainable. Adding or removing a side block is just adding or removing a single statement in the main mustache file. As a result the mustache template for a blog entry is less than 70 lines, the home page less than 50

index.mustache


<!DOCTYPE html>
<html lang="en">
<head>
<title>NotesSensei's Blog</title>
{{> partial_analytics}}
{{> partial_meta}}
{{> partial_style}}
{{> partial_feeds}}
<meta name="Revisit-After" content="2 Days" />
</head>
<body>
    {{> partial_navbar}}
    <div class="container">
    {{> partial_header}}
    {{> partial_browserwarning}}
    <div class="row-fluid">
            <div class="span9">
                {{#topArticles}}
                <article class="well well-raised">
                    <h1><small><a href="/blog/{{newURL}}">{{& title}}</a></small></h1>
                    <hr />
                    <div>{{& mainBody}}</div>
                    <hr />
                    {{#moreBody}} <a href="/blog/{{newURL}}">Read more</a>
                    {{/moreBody}}
                    <p>{{> partial_authorentryinfo}}</p>
                </article>
                {{/topArticles}}
                <ul class="breadcrumb pull-centre">
                    <li><a href="/blog/all" title="That's a loooong list!">Older entries<i class="icon-hand-right"></i></a>
                    </li>
                </ul>
            </div>
            <div class="span3">
                {{> sidebar_reference}}
                {{> sidebar_categoriesArchive}}
                {{> sidebar_upgrade}}
                {{> sidebar_twitter}}
                {{> sidebar_languages}}
                {{> sidebar_visitors}}
                {{> sidebar_usefultools}}
            </div>
        </div>
    </div>
    {{> partial_footer}}
    {{> partial_bottomscripts}}
</body>
</html>

blogentry.mustache


<!DOCTYPE html>
<html lang="en">
<head>
<title>NotesSensei's Blog - {{title}}</title>
{{> partial_analytics}}
{{> partial_meta}}
{{> partial_style}}
{{> partial_feeds}}
<meta name="Revisit-After" content="90 Days" />
</head>
<body>
    {{> partial_navbar}}
    <div class="container">
    {{> partial_headersmall}}
    {{> partial_browserwarning}}
    {{> partial_breadcrumb}}
        <div class="row-fluid">
            <div class="span12">
                {{> partial_series}}
                {{> partial_previousnext}}
                <article class="well well-raised">
                    <h1><small>{{& title}}</small></h1>
                    <hr />
                    <div>{{& allBody}}</div>
                    <hr />
                    {{> partial_authorentryinfo}}
                </article>
                <a name="comments"></a>
                {{^commentsclosed}}
                <div class="well well-raised" style="text-align : center">
                <button class="btn btn-lg btn-info" data-toggle="collapse" data-target="#commentform_{{UNID}}" type="button">
                    Add your comment...  <span class="glyphicon glyphicon-comment"></span>
                </button>
                </div>
                <div id="commentform_{{UNID}}"  class="collapse"></div>
                {{/commentsclosed}}
                <div class="well well-raised">
                    <h4>Comments</h4>
                    <ol id="commentList">
                        {{#comments}}
                        <li>
                           {{#gravatarURL}}<img src="{{.}}" class="gravatarimg" />
                           {{/gravatarURL}}
                            posted by <b>{{author}}</b> on <i>{{createdString}}</i>:<br /> {{& comment}}
                            <hr style="clear : both" />
                        </li> {{/comments}}
                        {{^comments}}
                        <li id="nocomments">
                            <h5>No comments yet, be the first to comment</h5>
                        </li>
                        {{/comments}}
                    </ol>
                </div>
                {{> partial_previousnext}}
            </div>
        </div>
        {{> partial_breadcrumb}}
    </div>
{{> partial_footer}}
    <script type="text/javascript">
        var permaLink = "https://wissel.net/blog/{{newURL}}";
    </script>
    {{> partial_bottomscripts}}
    {{> partial_commentscripts}}
</body>
</html>

Next stop: a new comment engine


Posted by on 01 May 2017 | Comments (0) | categories: Blog