AutoStore: Workflow Loop Example
Regardless of the programming language, a looping-statement is a common declaration for iterating through collections of things, such as the items of an array, records selected from a database, elements within an XML file, or the files inside a directory.
A recent AutoStore challenge was presented where a script was used to fetch data in a process that returned either one or many items. This was the easy part. However, each return record needed to be specific to an individual Route based on one of those items returned.
Although the captured document being processed would be common for all in the return, the recipient belonging to one of these unique records should not be aware of nor receive any of the information that would belong to a different recipient.
Because an unknown number of return records would be expected, this presents a challenge due to the fact that the Route component has no logic in terms of knowing that each of the items collected in a custom script should be handled individually, and should not to deal with these in a single route.
AutoStore is known for Capture, Process, Route, and in that order. There is no native mechanism to do something like Capture, (Process, Route) * n until the end of a loop count has been reached.
This scenario could be much more simple if the information returned was more precise, and only returned one record for the Route component to handle. Sounds like we're caught between a rock and hard place, but to quote a famous movie character, "Do... or do not. There is no try."
We'll look at a couple of different techniques to handle this scenario. Although this discussion does contain quite a bit of VBScript, the intent here is to stay as close to supporting the AutoStore process as much as possible.
This would certainly be a custom approach to handle this type of custom data return, and requires some additional scripting knowledge in order to integrate with other back-end systems. Scripting is one thing, but also scripting the route is another. The bottom line here is that it can be done.
As a slightly off-topic side note, in the case of a URL, another factor to consider is that additional considerations might possibly need to be factored in, such as any normalization and encoding for special characters.
To demonstrate this, a complete, zipped-up sample configuration can be downloaded here, which will be used to explain the main concepts from here. There are 4 Tasks in the configuration file, and the following is an general description of what each one is responsible for:
Let's step through the more significant details of each of these tasks.
The cAutoCapture task uses AutoCapture to capture files for this example, but any type of Capture component could be used. AutoCapture is not configured to collect any field input in this example to simply show that any information formulated in this task is truly independent.
From there, the document arrives at the VB/JScript component, where the procedure name given is CollectJobInfo to generally describe the more significant action taking place in the OnLoad and OnUnload events.
During the OnLoad event, an RRT is created to hold some sample data that is similar to what was described at the beginning of this article.
The intExamples variable indicates how many example returns we want to test with, and simulates the number of returned records, thus the number of times the workflow would need to loop. This numeric value is used as the input parameter for the GetJobValuesSampleData() function, which was created is this example for a few reasons:
This is where any complex work could be done to collect any results obtained from some type of lookup. In order for the pNextTaskLogic task to do its part, the formatting of the collected information in this task is important so it can be properly analyzed in an expected way further downstream.
As such, formatting the collected information is key. For the records found, we are delimiting each item with a semicolon, and each record set is separated using a pipe-character.
After the VB/JScript component is finished, the Knowledge Package Builder allows the value of the ~USR::%Collected%~ RRT to be packaged up with the captured file, and the Send to Folder settings of the MultiRouter passed on over to the inbound folder of the next task.
After the Knowledge Package Loader consumes the packaged-up metadata and captured file, the next stop is the VB/JScript component. The first order of business is to determine how many jobs are left to process. All of the collected values from the Capture task are brought into the script.
Looking at the script, from the point of view of the Capture task, the GetJobsRemaining() function takes that collected job information from the p_Collected field, and returns the number of jobs, which are those entire string values positioned in between the pipe character.
So long as there is at least one job, there is something to work with. Beyond the pNextStepLogic task, the other two are the pSendToOutput task, and then the rSendToOutput task, which is the last task for the routing the document.
The job count is used again, this time for the GetNextTaskName() function, which is used to determine what the next task should be. The rSendToOutput task will handle the last job when there is one job left, and the pSendToOutput task sends the file and returns to the pNextStepLogic task.
With the next task determined, the metadata for this specific job needs to be isolated from the collected job information by splitting the collected information, and returning the first index. Both the pSendToOutput and rSendToOutput tasks need this information to route the file.
The actual return is everything between the pipe characters, including the semicolon delimiter, which we can break down further to assign variables we use to create RRTs to send over.
One of the last things left to do is to remove the values that will be used for routing the job into the next task, so the Collected information needs to be updated. Otherwise, the job re-enters this task in an endless loop, and would keep sending the file until the AutoStore service is stopped.
The UpdateCollected() function works by reformatting the string in a similar manner to how the capture task created the collected information. Here, the function is provided both the initial collected string, as well as the job information that would need to be removed.
Aside from creating the RRTs, the script work is complete. Now, the Knowledge Package Builder takes it from here, packages up the captured file, and includes a few new fields that contain the RRTs we created from the script, which will be used for routing by either of the next tasks.
The final stop is the Send to Folder component, and it's responsible for delivering the file to the inbound folder of either the pSendToOutput folder, or the rSendToOutput folder. How it does so is with the ~USR::%NextTaskName%~ RRT value created in the script, and set into the Folder Path.
In terms of logic, this pNextTaskLogic task is responsible for keeping track of the job count, but it needs good information form the Capture task. The file routing can now begin, and it's downhill from this point on. For now, let's just say that there was more than one job to route.
When a new job lands into the inbound folder for this task, the Knowledge Package Loader sends the file and job values over to an eConnector, which is responsible for sending the file. This example uses the Send to Folder eConnector for demonstration, but it could be any other eConnector.
Afterwards, the file and metadata are packaged back up as the last part in this Process task, and sent (you guessed it) right back to the pNextTaskLogic task's inbound folder.
When the job count is down to one left, this task simply takes in the package from the pNextTaskLogic task, and moves the file over to the intended destination. Here, we actually use a routing component instead of using an eConnector in the pSendToOutput task.
Just like with the pSendToOutput task, Send to Folder is in place here for demonstration purposes, but the Route component used can be any them.
A recent AutoStore challenge was presented where a script was used to fetch data in a process that returned either one or many items. This was the easy part. However, each return record needed to be specific to an individual Route based on one of those items returned.
Example
| Example1@email.com | Example1/URL |
| Example2@email.com | Example2/URL |
| Example3@email.com | Example3/URL |
Although the captured document being processed would be common for all in the return, the recipient belonging to one of these unique records should not be aware of nor receive any of the information that would belong to a different recipient.
Because an unknown number of return records would be expected, this presents a challenge due to the fact that the Route component has no logic in terms of knowing that each of the items collected in a custom script should be handled individually, and should not to deal with these in a single route.
AutoStore is known for Capture, Process, Route, and in that order. There is no native mechanism to do something like Capture, (Process, Route) * n until the end of a loop count has been reached.
This scenario could be much more simple if the information returned was more precise, and only returned one record for the Route component to handle. Sounds like we're caught between a rock and hard place, but to quote a famous movie character, "Do... or do not. There is no try."
We'll look at a couple of different techniques to handle this scenario. Although this discussion does contain quite a bit of VBScript, the intent here is to stay as close to supporting the AutoStore process as much as possible.
Looping a Custom Route
One way to handle this situation would be to incorporate the document routing within that looping statement of the custom VBScript. In other words, go through each item, and call a custom procedure to actually send the file using the information for each returned item.
Example
| Set KnowledgeDocument = KnowledgeObject.GetFirstDocument() While Not(KnowledgeDocument Is Nothing) ' build a delimited string of working file paths ... If Len(kFilePath) = 0 Then kFilePath = KnowledgeDocument.FilePath Else kFilePath = kFilePath & ";" & KnowledgeDocument.FilePath End If Set KnowledgeDocument = KnowledgeObject.GetNextDocument Wend For Each Item In ReturnedItems ' assuming Item = "Example1@email.com;Example1/URL" itemParts = Split(Item, ";", -1, vbTextCompare) emailAddr = itemParts(0) urlString = itemParts(1) emailBody = "Your information is " & urlString & "." atchFiles = kFilePath Call SendInDomainCdoMessage(emailAddr, emailBody, atchFiles) Next |
This would certainly be a custom approach to handle this type of custom data return, and requires some additional scripting knowledge in order to integrate with other back-end systems. Scripting is one thing, but also scripting the route is another. The bottom line here is that it can be done.
As a slightly off-topic side note, in the case of a URL, another factor to consider is that additional considerations might possibly need to be factored in, such as any normalization and encoding for special characters.
Looping a Process Task
Another way is to build the AutoStore configuration so that files and metadata can loop back into a Process task. Using a couple of scripts to assist with the looping, and some tasks equipped with the Knowledge Package Builder and Loader, operations can remain more within the AutoStore realm.To demonstrate this, a complete, zipped-up sample configuration can be downloaded here, which will be used to explain the main concepts from here. There are 4 Tasks in the configuration file, and the following is an general description of what each one is responsible for:
- cAutoCapture: Collect details used to determine the next task.
- pNextTaskLogic: Figures out if the job is the last one.
- pSendToOutput: Sends the file, and feeds back to pNextTaskLogic.
- rSendToOutput: Handles routing the last job as the final task.
Let's step through the more significant details of each of these tasks.
cAutoCapture Task
The cAutoCapture task uses AutoCapture to capture files for this example, but any type of Capture component could be used. AutoCapture is not configured to collect any field input in this example to simply show that any information formulated in this task is truly independent.
From there, the document arrives at the VB/JScript component, where the procedure name given is CollectJobInfo to generally describe the more significant action taking place in the OnLoad and OnUnload events.
During the OnLoad event, an RRT is created to hold some sample data that is similar to what was described at the beginning of this article.
Collect the Job Values
| Dim intExamples : intExamples = 3 ' assign the collected pre-formatted data to the "Collected" variable ... Collected = GetJobValuesSampleData(intExamples) EKOManager.StatusMessage("VB/JScript: Collected=" & Collected) ' create a test RouteCount RRT ... Set Topic = KnowledgeContent.GetTopicInterface If Not(Topic Is Nothing) Then Topic.Replace "~USR::%Collected%~", Collected Else ' do nothing End If |
The intExamples variable indicates how many example returns we want to test with, and simulates the number of returned records, thus the number of times the workflow would need to loop. This numeric value is used as the input parameter for the GetJobValuesSampleData() function, which was created is this example for a few reasons:
- A loop count was needed to test the workflow
- Relegate most of the script-complexity to the main topic
- Handles the formatting of the collected information
This is where any complex work could be done to collect any results obtained from some type of lookup. In order for the pNextTaskLogic task to do its part, the formatting of the collected information in this task is important so it can be properly analyzed in an expected way further downstream.
As such, formatting the collected information is key. For the records found, we are delimiting each item with a semicolon, and each record set is separated using a pipe-character.
Collected Values Format Example
| "a1;b1|a2;b2" |
After the VB/JScript component is finished, the Knowledge Package Builder allows the value of the ~USR::%Collected%~ RRT to be packaged up with the captured file, and the Send to Folder settings of the MultiRouter passed on over to the inbound folder of the next task.
pNextTaskLogic Task
After the Knowledge Package Loader consumes the packaged-up metadata and captured file, the next stop is the VB/JScript component. The first order of business is to determine how many jobs are left to process. All of the collected values from the Capture task are brought into the script.
Looking at the script, from the point of view of the Capture task, the GetJobsRemaining() function takes that collected job information from the p_Collected field, and returns the number of jobs, which are those entire string values positioned in between the pipe character.
GetJobsRemaining() Function
| Function GetJobsRemaining(ByVal JobInfo) arrJobInfo = Split(JobInfo, "|", -1, vbTextCompare) intJobInfo = UBound(arrJobInfo) + 1 GetJobsRemaining = intJobInfo End Function |
So long as there is at least one job, there is something to work with. Beyond the pNextStepLogic task, the other two are the pSendToOutput task, and then the rSendToOutput task, which is the last task for the routing the document.
The job count is used again, this time for the GetNextTaskName() function, which is used to determine what the next task should be. The rSendToOutput task will handle the last job when there is one job left, and the pSendToOutput task sends the file and returns to the pNextStepLogic task.
GetNextTaskName() Function
| Function GetNextTaskName(ByVal JobInfo) Dim strReturn : strReturn = "" Dim sNextTask : sNextTask = "" If JobsRemaining = 1 Then ' this would be the last job ... sNextTask = "rSendToOutput" Else ' there are additional jobs to loop through ... sNextTask = "pSendToOutput" End If strReturn = sNextTask GetNextTaskName = strReturn End Function |
With the next task determined, the metadata for this specific job needs to be isolated from the collected job information by splitting the collected information, and returning the first index. Both the pSendToOutput and rSendToOutput tasks need this information to route the file.
GetNextJobValues() Function
| Function GetNextJobValues(ByVal JobInfo) Dim strReturn : strReturn = "" If Len(JobInfo) > 0 Then arrJobInfo = Split(JobInfo, "|", -1, vbTextCompare) strNextJob = arrJobInfo(0) strReturn = strNextJob End If GetNextJobValues = strReturn End Function |
The actual return is everything between the pipe characters, including the semicolon delimiter, which we can break down further to assign variables we use to create RRTs to send over.
Assigning Job Information Variables
| NextJobValues = GetNextJobValues(p_Collected) Dim arrNextJobValues arrNextJobValues = Split(NextJobValues, ";", -1, vbTextCompare) JobValue1 = arrNextJobValues(0) JobValue2 = arrNextJobValues(1) |
One of the last things left to do is to remove the values that will be used for routing the job into the next task, so the Collected information needs to be updated. Otherwise, the job re-enters this task in an endless loop, and would keep sending the file until the AutoStore service is stopped.
Updating the collected job information
| Collected = UpdateCollected(p_Collected, NextJobValues) |
The UpdateCollected() function works by reformatting the string in a similar manner to how the capture task created the collected information. Here, the function is provided both the initial collected string, as well as the job information that would need to be removed.
UpdateCollected() Function
| Function UpdateCollected(ByVal JobInfo, ByVal JobToRemove) Dim strReturn : strReturn = "" Dim Collected : Collected = "" If Len(JobInfo) > 0 Then arrJobInfo = Split(JobInfo, "|", -1, vbTextCompare) For Each strJobInfo in arrJobInfo If Not strJobInfo = JobToRemove Then If Len(Collected) = 0 Then Collected = strJobInfo Else Collected = Collected & "|" & strJobInfo End If End If Next End If strReturn = Collected UpdateCollected = strReturn End Function |
Aside from creating the RRTs, the script work is complete. Now, the Knowledge Package Builder takes it from here, packages up the captured file, and includes a few new fields that contain the RRTs we created from the script, which will be used for routing by either of the next tasks.
The final stop is the Send to Folder component, and it's responsible for delivering the file to the inbound folder of either the pSendToOutput folder, or the rSendToOutput folder. How it does so is with the ~USR::%NextTaskName%~ RRT value created in the script, and set into the Folder Path.
In terms of logic, this pNextTaskLogic task is responsible for keeping track of the job count, but it needs good information form the Capture task. The file routing can now begin, and it's downhill from this point on. For now, let's just say that there was more than one job to route.
pSendToOutput
When a new job lands into the inbound folder for this task, the Knowledge Package Loader sends the file and job values over to an eConnector, which is responsible for sending the file. This example uses the Send to Folder eConnector for demonstration, but it could be any other eConnector.
Afterwards, the file and metadata are packaged back up as the last part in this Process task, and sent (you guessed it) right back to the pNextTaskLogic task's inbound folder.
rSendToOutput
When the job count is down to one left, this task simply takes in the package from the pNextTaskLogic task, and moves the file over to the intended destination. Here, we actually use a routing component instead of using an eConnector in the pSendToOutput task.
Just like with the pSendToOutput task, Send to Folder is in place here for demonstration purposes, but the Route component used can be any them.
This is really cool! Thanks for the detail JimmyB :)
ReplyDelete-Kyle.Hart@nuance.com
Thanks Kyle, and you're welcome!
Delete