Your new system went live. Migration of current, active data went well. A decision was made not to move historic data and keep the old system around in "read-only" mode, just in case some information needs to be looked up. Over time your zoo of legacy systems grows. I'll outline a way to put them to rest.

The challenges

All recent systems (that's younger than 30 years) data is stored more or less normalized. A business document, like a contract, is split over multiple tables like customer, address, header, line items, item details, product etc.

Dumping this data as is (csv rules supreme here) only creates a data graveyard instead of the much coveted data lake or data warehouse.

The issue gets aggravated by the prevalence of magic numbers and abbreviations that are only resolved inside the legacy system. So looking at one piece of data tells you squid. Only an old hand would be able to make sense of Status 82 or Flags x7D3z

Access to meaningful information is confined to the user interface of the legacy application. It provides search and assembly of business relevant context

The solution approach

Solving this puzzle requires a three step approach:

• denormalize
• transform
• make accessible

There are two challenges getting into TDD:

• Why should I test upfront when I know it fails (there's this massive aversion of failure in my part of the world)?
• Setting up the whole thing.

I made peace with the first requirement using a very large monitor and a split screen, writing code and test on parallel, deviating from the 'pure teachings' for the comfort of my workflow.

The second part is trickier, There are so many moving parts. This post documents some of the insights.

Testing in the IDE

TDD has the idea that you create your test first and only write code until your test passes. Then you write another failing test and start over writing code.

As a consequence you need to test in your IDE. For JavaScript or Java that's easy (the languages I use most):

• In JavaScript you define a script test in your package.json you can run any time. For a connoisseur there are tools like WallabyJS or VSCode Mocha Sidebar that run your tests as you type and/or save. The tricky part is: what testing libraries (more on that below) to use?
• In Java Maven has a default goal validate and junit is the gold standard for tests. For automated continuous IDE testing there is Infinitest
• For Salesforce you have a combination of JavaScript and Apex (and clicks-not-code), testing is a little trickier. The commercials IDE TheWelkingSuite and Illuminated Cloud make that a lot easier. How easy is in they eye of the beholder. (Honorable mention: JetForcer - I simply haven't tested that one yet)

Testing in your Continuous Integration

Automated testing, after a commit to Github, GitLab or BitBucket happens once you configure a pipeline as a hook into the repository and have tests specified the pipeline can pick up. Luckily your maven and npm scripts will most likely work as a starting point.

The bigger challenge is the orchestration of various services like static testing, dependency management and reporting (and good luck if your infra guys claim, they could setup and run everything inhouse).

Some of the selections available:

• Repository
• Github
• GitLab
• BitBucket
• Pipelines
• Heroku Flow
• BitBucket
• Travis
• Jenkins
• CodeShip
• CircleCI
• and more of them
• Testing and Reporting service
• CodeClimate
• SauceLab
• Codacy
• Coveralls
• Snyk (for vulnerabilities)
• GreenKeeper (for dependency management)
• and many more
  some run extra tests, some report on tests that ran in your pipeline

OAuth in its various versions is the gold standard for Authorization (and usingOpenID Connect for Authentication as well). There are plenty of introductions around explaining OAuth. My favorite HTTP tool Postman makes it really simple to obtain access via OAuth.

Nevertheless all those explanations are quite high level, so I wondered what happens on the wire for the getToken part so I started digging. This is what I found. Nota bene: There is no inherit security in OAuth if you don't use https.

The components

• Authorization server: server to interact with to get an authorization
• Client identifier (ClientID): "userid" of the application
• Client Secret: "password" of the application
• A user

I'm not looking at the Resource Server here - it only comes into play before or after the actual token process.

The Form-Post Flow

There are several flows available to pick from. I'm looking at the Form-Post flow where user credentials are passed to the authentication server to obtain access and refresh tokens.

For this flow we need to post a HTTP form to the authorization server. The post has 2 parts: Header and body. A request looks like this:

POST /yourOAuthEndPoint HTTP/1.1
Host: authserver.acme.com
Accept-Encoding: gzip, deflate
Accept: *.*
Authorization: Basic Y2xpZW50aWQ6Y2xpZW50c2VjcmV0
Content-Type: application/x-www-form-urlencoded
Cache-Control: no-cache

grant_type=password
  &username=user%40email.com
  &password=password
  &scope=openid+email+profile
  &client_id=clientid

Some remarks:

• The Authorization header is just as Base64version of clientid:clientsecret - you have t replace it with your actual info
• Content-Type must be application/x-www-form-urlencoded
• The body is just one line with no spaces, I split it only for readability
• scope is a encoded list the + signs are actually spaces. Keeping that in mind you want to keep the server side scope names simple
• You need to repeat the clientid as header value

As a result you get back a JSON structure with authorization information. It can look like this:

{
    "access_token": "wildStringForAccess",
    "refresh_token": "wildStringForRefreshingAccess",
    "token_type": "Bearer",
    "expires_in": 300
}

The result is easy to understand:

• expires_in: Duration for the access token in seconds
• token_type: Bearer denotes that you call your resource server with a header value of Authorization: Bearer wildStringForAccess

As usual YMMV