Enterprise architecture - from Silos to Layers

In a recent discussion with a client the approaches and merits of Enterprise Architecture took center stage. IBM for a very long time proposed SOA (service oriented architecture) which today mostly gets implemented in a cloud stack. While it looks easy from high enough above, the devils is in the details. Mostly in the details how to get there. The client had an application landscape that was segmented along a full stack development platform with little or no interaction between the segments or silos:

The challenges they are facing are:

• No consistency in user experience
• Difficult to negotiate interfaces point-to-point
• No development synergies
• Growing backlog of applications

In the discussion I suggested to first have a look at adoption all principles to successfully pass the Spolsky test. Secondly transform their internal infrastructure to be cloud based, so when need arises workloads could easily be shifted to public cloud providers. The biggest change would be to flip the silos and provide a series of layers that are used across the technologies. A very important aspect in the layer architecture is the use of Design by contract principles. The inner workings of the layer are, as much as sensible, hidden by an API contract. So when you e.g. retrieve a customer information, it could come from SAP, Notes, RDBMS or NoSQL, you wouldn't know and you wouldn't care.This is important, so you can swap components in and out without breaking all application. The way more important aspect of clean APIs however is testability. Unit tests require predictable results. So being able to call a test API that has the same signature as the real is essential. When you call the real API, that's already an Integration test (of course the icing on the cake is BDD).
On the back of a napkin this is my recommendation:

As you can see I didn't model out the database part, lunch and napkin were at the end and, while important for operation, the choice of databases doesn't alter the fundamental Ogre architecture principles. Let me elaborate on the layers:

• Frontend

  The frontend is more than just the UI. Is is heavy with design principles. The great tempation is to use an integrated approach that tightly couples front- and backend. While that might save some time for the initial rollout, it costs a lot of time in maintenance and is a key reason for UIs that differ depending on the backend. Fashions in IT change as much as in apparel, so the recommendation of today might be outdated next year. Nevertheless: pick a well understood and widely adopted UI framework. In the IBM world that would be the IBM OneUI, Googles material design or Twitter Bootstrap (the later two can be combined in various ways).
  With the explosion of Bootstrap based templates Bootstrap is my clear favorite.
  There is more to the the frontend layer. You can opt to develop Offline first (I like Hood.ie) or target native mobile apps. In any case you need to pick your JavaScript framework. In an IBM world that would be the Dojo Toolkit. On the other hand the most popular framework is JQuery. However both frameworks don't conver data binding specifically well, so you might want to look at ember.js or - my personal favorite - angular.js (Go and watch some videos). There's a nice sample of Angularjs with material design. Why angularjs? You can define your own UI components, that you can then use anywhere. Something like <customeraddress id="123" /> and it will render a complete customer address the same across all your applications.
  In summary: Twitter Bootstrap & angular.js

• API

  This is the most tricky and most rewarding part of your architecture. First of all: there is no additional component or hardware to buy or to deploy, but clear definitions and documentation is required. A few principles have become common practise: the API is build around REST via HTTPs and uses JSON as data (I hear the howling of the XML proponents). However the JSON schema landscape is very weak and you have an Enterprise Service Bus in a different layer. So designing purely around JSON might not be good enough.
  I would have a good hard look at XML Schema and, first and foremost at existing well defined schemas. Some of the obvious candidated: ebXML, hrXML, RDF, Dublin Core and all the others
  A neat way to design and test your APIs, not travelling down the XML route initially is apiary. It automatically generates documentation and test code. Go check it out.
  While https is the common standard today (SOAP is rather on the backend), you want to watch webRTC and webSockets. Both can take JSON, so your schema work can stay.

  Why is that API layer so important?

  A stable API allows you to replace any of the frontend or backend components without disrupting your applications. You also allow developers to easily reuse components you never anticipated when you did build the applications in the first place. It also decouples responsibilities nicely and allows different talent work in their core competencies. Last not least: with a clear API you can build automated test to make sure that all components work as designed (without an API that has mock objects you can't do unit tests)

• Servers

  I deliberately kept this part lean. Generally: stick with the servers you have, even if it is PHP, until your API and frontend is sorted out. Over time you want to consolidate platforms and databases. One relational database is good enough, if you want declarative security and mail integration, Domino is a good choice, not replacing your RDBMS, but complementing it. I'm not particularly fond of .net since it limits my OS choices. The new kid on the block is the MEAN stack, but I would be careful about the M, rather consider Cloudant or its opensource cousin CouchDB.
  An interesting contestant here actually is IBM Portal. It allows you a great deal of integration and is friend with bootstrap and angularjs. The debate build vs. buy is ongoing and needs careful considerations. Check with your job market: do you get enough Portal or Node.js developers? (For Domino you hire Java and/or JavaScript) developers.
  What is essential on the server front: build your server park on open standards around virtualization and cloud enablement. Three technologies come to mind, your server team needs to have a look at: OpenStack, CloudFoundry and Docker. Interestingly all of them originated from the Linux world and are slowly trickling down to Windows Server.
  Of course... all your servers need to be well monitored. You can choose between OpenSource, cloud based offerings or commercial packages. Choosing not to monitor your servers is not an option, it defeats productivity and DevOps

• Enterprise Service Bus

  You have an installed base of applications, that need to exchange data with each other. While, in the beginning, point to point connections seem easy and fast to implement, they rapidly turn into a maintenance nightmare. Since you already adopted the API thinking, it is logical that you will want to provide an API for applications to "talk to each other". It allows to carry forward process steps outsite the interaction with users. E.g. a order placed in SAP could call a service on the bus to dispatch the user documentation to the document sharing. Or a print job PDF output is send to a helper application, based on the rules for the recipient it gets printed or emailed or put into file sharing.
  The ESB can also be the destination for all the additional data the Internet of Things is bringing to the table. Be aware: having an ESB doesn't do anything, the participating applications need to be adjusted accordingly

• Social

  The first association with "social" are the respective UI applications. However social is rather a stance and a service with the default user interfaces being just one access to people. Having the APIs available to query people, places and things, social can be deeply integrated into any business application: links to the wiki with help, embedded chat to clarify a process, creation of activities to supplement workflows, output of periodical updates into blogs etc. Your possibilities are endless

• DevOps

  There is a lot of chatter, lets cut out all the chase: You have a working version control, with GIT (better get it right) being the undisputed champion. You run continuous integration for all your apps, starting with Jenkins on the backend. The integration server doesn't do anything, you have to provide build and test scripts (For Domino there are unit tests and automatic builds).
  You generate all the documentation using things like JSUnit, LSUnit, JUnit, NUnit and e.g. Visustin and deploy to staging servers. Your automation doesn't stop there. You want to automate your front-end development too. E.g. a good application is aggressive caching for static resources. This is fine unless you are changing them. Usng a suitable front-end workflow that problem can be avoided. So add Yeoman, grunt, gulp, bower and maven to your tool set

• Shared Services

  Last not least: build shared services. Social was already an example of such a shared service. But there is more. A workflow module (you could use Domino for that) and a Business Rules Engine are good candidates too. So is an extended directory with schedule information (e.g. who has an approval role when the original approver is absent depending on time and reason). Notifications are a good candidate too. Shared services can only work with good APIs. Both on the ESB and HTTP level. A report scheduler makes sense too

The list provided shall serve as food for thought, not as ultimate wisdom. About each of the items tons of books have been written, so fill in the details as needed.
As usual YMMV