Domino Development - Back to Basics - Part 3: Not all Documents are created equally

Continuing from Part 2 this installment will introduce you to the four types of documents you can find inside an NSF:

1. Documents
2. Responses
3. Response to Responses
4. Profiles

Profile Documents

All document types follow the same rules as outlined in Part 2, but differ in the way they are created or accessed. Short of profiles all documents typically get accessed in one or more views, by users or your code. Profiles on the other hand, never show up in a view, they are only accessed using some code. The easiest case is @Commmand([EditProfile]"NameOfAForm") for a System profile and @Commmand([EditProfile]"NameOfAForm";@UserName) for a user profile. Notes keeps an internal index for profiles and they are typically cached in memory, so accessing them is superfast (and quite a headache if you fall into the trap to use them for frequently updating information). Their intended use is storing configuration and selection options, so @GetProfileField(FormName;FieldName) has rapid access to options. In XPages applications the need for profiles is diminished, since you typically would store such configuration options in the Session and Application Scope where you load them once.

Document hierarchies

The different "Document types" are actually "only" form properties (One can see how forms and documents are points of confusion). Inside a document the presence of the $REF field (containing the UNID of the parent) makes a document a response. If the parent document happens to be a response itself, the document then is considered a "Response to Response" All Notes APIs have methods to create or query a Parent - Child hierarchy. A good overview can be found in this Ytria TechLab. To my knowledge a Response-to-response document doesn't contain a direct reference to the root document
The form properties are used when you create a document in the Notes client to automatically establish Parent-Response relationships. A typical mechanism used in response document forms is to enable inheritance (that feature can also be used for standard documents, it isn't limited to responses!): When the document get created in the context of the parent (think: you read an entry in a discussion and press reply) all items from the parent document, that have fields in the response form with the same name as the item get copied into the response document (good fun if item and field don't have the same data type). Additionally all formulas computation in the response get executed against the parent, but stored in the response. This happens once at creation, not thereafter. To keep fields in sync you would use @GetDocField($REF;"SomeFieldName") to obtain the latest value (there are methods in LotusScript, JavaScript and Java too).
Sounds complicated? Go and create a discussion database (the template is on your Notes client) and play with it for a while. All this leads to an important Notes concept:

Inheritance over Normalization

Database normalization is a corner stone concept of RDBMS. If RDBMS is all you ever used and you are stuck in the idea that databases = RDBMS ( really?), this will be hard to swallow.
Normalization is an abstraction that is suitable and needed for an RDBMS, it isn't a real life fact or an absolute necessity. If you blindly apply Normalization to XML Database, document databases, Graph databases you create more problems than solutions. So free your mind and follow:
Let's look at a typical normalization example: an invoice: You have one customer table, one invoice table, one items table and one products table. In your invoice table you point to the customer table, in the item table to your invoice table and your product table. Life seems good.

But then pesky reality kicks in: Customers move, but legal demands that existing invoices don't change (commonly changing a legal document is called forgery). Product prices change, or sales negotiated a different price. Customers might call your articles different from your product table etc. So you adjust your diagram:

The attributes in red are inherited from the respective tables. This inheritance in an RDBMS is entirely done in code, in (classic) Notes it could be done through a form property. So inheritance is also quite common in RDBMS based applications too.
In Notes it is rather the rule than the exception. Retrofitting an RDBMS normalization onto a Notes application might not bring the benefits you might expect, so keep in mind:
Notes favours inheritance over normalization.
Interestingly when you want to export data, e.g. for data interchange you hardly generate one table/file per RDBMS table, but you de-normalize the data for transport. In Notes this tandem of normalization/de-normalization isn't necessary since you deal with a (self contained) document already.

Items have multi-values and can be grouped

A typical way in a Notes application to handle parent-child records is the use of a group of related multi-value fields (in the form), resulting in a set of items in the document with the same number of values. You might have one address field (which can have many lines - much more flexible than a text column in an RDBMS) and a set of multi-value fields like: ProductID, description, Price, Quantity and SumPrice. Since the (classic) UI didn't make it very easy to keep those in sync, a module called Table Walker had been made available on the Sandbox. I later created Tablewalker for XPages as part of the XPages tutorial (with enlightened bug fixes courtesy Matt White). Keeping the line items inside a document makes transporting, processing or securing them much easier, however it makes querying the line items a bit more difficult (more on that when looking at views). You can take a similar (actually more powerful from a modelling point of view) approach when using DB/2 PureXML. DB/2 PureXML and XPages are actually a match made in heaven (or was that the other place starting with h?), which is the subject to another story, another time.
Remember (before you start bitching about parent-child relations): the explanations above are guidelines, there are valid cases where RDBMS like separation of properties and detail rows into separate documents is a better approach. You know how to normalize, so I don't need to explain that part

A Document can save directly to an RDBMS

There was an ill-fated attempt to use DB/2 as storage destination for NSF data called NSFDB2 which unfortunately predated PureXML capabilities (otherwise the story might have been different), and it shall not be spoken about it anymore (support for it has ceased anyway). The tools here, available since R5 are DECS (Domino Enterprise Connection Service, included on every Domino server and LEI ( Lotus IBM Enterprise Integrator, an IBM product you need to buy). Filling in some simple forms in the configuration database can link your Notes forms (and partially views) to backend data from various databases or ERP systems. Once you got a grasp of these tools, some creative usage patterns become possible ( more details here about Domino and RDBMS). Of course you can use code (ODBC, JDBC or LCLSX) your data transfer or use the IBM Swiss Data Knife for moving data around.

Next up: Domino views are different