Insight gathers data from the process client using the publish and subscribe method.
Insight is generally configured using a global subscription, which subscribes the insight connection group to documents and events. The subscription is generally applied to the Home folder, using rules of "*/document+" and "*/event+". This means that insight will get data from each event creation and each document publication.
Events
Events are published automatically and no action is required within a process to pass events to Insight.
Internally, the data for the event publish is:
{
"event_timestamp": ""yyyy-mm-ddThh:mm:ss.ttt",
"user_identifier": "userid",
"user_name": "User Name",
"event_type_reference": "ref",
"event_type_name": "Name",
"status_reference": "ref",
"status_name": "Name",
"description": "Event description",
"attachments": [
{
.. file info object ..
}
]
}The consume process looks up worker and group details, and passes these to the insight module to be stored along with the document or event data.
Documents
A document represents a significant file or piece of data, or a file with associated data.
Insight does not treat all files as documents, only those that are considered the significant products of the process (other files may appear as attachments in events).
Documents must be explicitly published by a process, typically using a topic of "document".
The structure of a document publication is illustrated by the example configuration of the set step type.
{
"name": "publishDocument",
"script": "library.worker.SetStepType",
"config": {
"show": false,
"dataReference": null,
"publish": "document",
"data": {
"document_title":"Document title",
"document_description":"Document description",
"document_type_reference": "ref",
"document_type_name": "Document type",
"document_received_timestamp": "yyyy-mm-ddThh:mm:ss.ttt",
"document_date": "yyyy-mm-dd",
"document_valid_from_date": "yyyy-mm-dd",
"document_valid_to_date": "yyyy-mm-dd",
"document_status_reference": "ref",
"document_status_name": "name",
"document_tags": "tag1 tag2 tag3...",
"document_key": "document/path",
"document_data": {
.. other data ..
},
"document_file": {
.. file info object ..
}
}
},
"next": false
}Tags can either be passed as a comma- or space-delimited string, in which case it will be converted to an array of lower case values. Alternatively, it can be passed as an array, in which case the values of the array will be used as they are without conversion.
The document key represents a logically unique key for the file or data, and can be used to later extract the document from Insight. It should be a path-like structure, with parts separated by slashes, e.g. "supplier/123456/non_disclosure_agreement".
Document validation
Document are produced from workers. The process client security assumption is that individual workers are not fully trusted, but groups (folders) are. Documents with keys may be used for further processing and must be validated by the group that contains the worker before they are accepted into Insight. This reduces the risk that a rogue process (or a misconfiguration such as copying a task between folders) creates an invalid document.
Document validation is achieved by an execute of the worker's owning group, using the "validateDocument" action, passing in the <data> element the document object before the worker and group information is added, and in the <source> element the node reference of the source worker or group. The validateDocument action must return in the <return> element a JSON string containing {"valid": true} for the document to be considered valid. Documents that are not valid are logged and silently dropped. An error message can be returned in the <return> element, using {"message":"Error message..."}. If the validateDocument returns an error, that is logged also.
If documents do not have a document_key, document validation is not considered necessary and validateDocument is not called. Groups that do create documents with keys do not need a "validateDocument" action.
The validateDocument action can be achieved with an evaluate step that uses the group's configuration options to specify valid document key patterns. See Validate Document Script for an example and instructions on how to do this.