Push Connector > Audit

Auditing & Error Handling

Setting up logging and auditing on a connector is imperative for the following two reasons:

  1. Due to the validation which most applications or web services place on data before it is committed it is natural that most errors within an integration occurs during a connector transform.
  2. Connectors provide the touchpoints with the application being integrated, as such they are providing the tangible artefact of the integration.
Therefore it is important the auditing and logging controls on connectors are properly setup as these will provide valuable information to the audit report recipients.

Audit Summary

Setting up the audit summaries provide an instant reconciliation between the data processed by the connector and the source application.

At minimum, an audit summary should be setup with basic number of processed, successfully processed and erroneous records.

These are configured under the Audit tab of the connector.

example

%Record.PROCESSED Records Processed. %Record.INSERTED Created. %Record.UPDATED Updated. %Record.ERRORS Errors.

For transactional imports you may want to also add a list of transactions processed. Typically this would include listing the id of the transaction from the source application with any generated id of the destination application, possibly with a textual success/failure of the transaction:

Transaction %Record.ForeignId - Created Transaction %Record.GeneratedId - %Record.SAGE200IMPSUCCESS

Log Keys

Log keys are another critical component of the audit & logging setup.

As previously described, most errors during an integration originate from the connector/application. Any errors are reported in the detail section of the audit report. Log keys will always ascribe the message to the causative data!

Log Keys Recommended Setup

  1. Never use "writeback" fields as log keys.
    • If a transaction fails the fields value will not be generated, therefore it will be empty and there will be no value printed on the audit report..
  2. Use Natural Foreign Ids
    • Use id fields generated by the source application. These should always be populated, and will allow the end user to identify the erroneous transaction easily. Use of natural keys for log keys will help users identify data using an identifier they recognise.
  3. Avoid use of 'database' keys
    • Conversely, try to avoid use of database keys, where possible as these are often hidden from the user and will provide little value to the user.
  4. Repeat Parent Log Keys on Child Levels
    • Where a field has been specified as a log key on parent, repeat this same field on a child level. The result will print provide consistent error logging where the same data is printed for both header and detail levels. It should also allow you identify data down to the offending record.
  5. Each transaction level should have one field more than its parent
    • This should allow the user to identify the data down to the specific record causing the error or message.

Supported Counters

PROCESSED

Incremented for each record being processed.

INSERTED, UPDATED, DELETED

The counters will show the number of inserted, updated & deleted records.

Action on Transform Error

On Push Connectors you need to think about how you want IMan to handle errors. Careful attention should be given to the way this property is set.

Abort

The transform will stop at the first error. Any records processed prior to any error will typically not be undone.

Reject Record

This will cause the connector to process the valid transactions but reject erroneous ones. Each connector understands the concept of an atomic transaction i.e. an Order, a Despatch, a Customer or an Item, and will not leave half written records! For example, an Order with an invalid item code, the entire order will be rejected not just the lines with invalid items.

Continue

Will be removed in future versions. DO NOT USE.

Example Scenario: Combined Customer & Order Import

In this integration, as opposed to the Customer import that was left to Abort, the Order Import will be set to Reject Record.

The effect as follows:

  • If there is an error during the Customer import stage, the entire integration will stop.

    This way, the Customer import can be run multiple times without risk of duplicated data. The Customer import will ONLY insert new records and update existing ones.
  • If the Action on Transform Error is set to Abort on the Order Import, the process will stop at that point. This will leave the Orders in Sage that were successfully processed up to the point of the error.

    The successfully processed orders would probably need deleting prior to re-running the import, otherwise duplicates could occur.
By setting to Reject Record, any erroneous Orders will be rejected, but processing will continue and complete. Erroneous orders will need remedial action.