the HTTP IO Controller

The HTTP IO Controller provides the means to read from/write over an http/https. For JSON and Xml formats the Http Controller provides advanced webservice capabilities for integration with almost any REST and some (simple) SOAP services.

Read Mode

Readers will typically perform a GET operation to query data from a webservice.

Encoding Method

The character encoding method for the data being queried, typically by UTF-8.

See Character Encoding for more details. 

Webservice Behaviour

The Webservice Behaviour to control the Read request. The webservice behaviour determines the base url, authentication, paging, throttling & tracing.

Paging

When a result set is paged the Read operation will generate multiple requests. Each request to obtain a page of data is transparent to the Read operation.

Http Headers

Http Headers to be sent with each Read request.

Evaluate URL

When checked the URL will be evaluated via a VBScript expression.

URL

The URL of the request. This may be only the query path or the full Url. The Query path is combined with Base Url from the Webservice Behaviour to form the full Url.

The URL may be evaluated:

  • Not evaluated.
    • The URL should be a static string.
  • Evaluated
    • Evaulated URLs give you the ability to parameterise any request. In order to parameterise the reader it must have a parent transform, where the parent transform will provide the input fields (and values).
    • Evaluated URLs should evaluate to a valid URL. The evaluated expression can contain any valid VBScript function.
    • Evaluated URLs can reference any field which is from a mapped parent transform.
    • Evaluation applies only the URL displayed on the Options tab. Evaluation is not performed on URLs for stepped Reader requests.

    Example

    "/orders.json?updated_at_min=" & Format(DateAdd("d", -1, %[SYS.LASTRUNDATE]), "yyyy-MM-ddThh:mm:ssZ")

    Assuming SYS.LASTRUNDATE is 1st April 2020, 6:30 PM the above expression would evaluate to:

    /orders.json?updated_at_min=2020-03-31T18:30.00Z

    • The DateAdd function is subtracting a single day from the last run date to return 31st March 2020, 6:30 PM.
    • The Format function is formatting 31st March 2020, 6:30 PM to an ISO8601 style date: 2020-03-31T18:30.00Z
    • The formatted date is appended to '/orders.json?updated_at_min='

Paging

For paged results the URL is the initial request. Requests for subsequent pages of data are dictated by the paging mode.

HTTP Operation

The default operation for Readers will be a GET request. If the request needs to use another operation (POST, PUT, MERGE, PATCH) it can be chosen from this picklist.

The value is only used when the Post Request (next field) is populated.

Post Request

This contains either a path to a file containing the body of the request OR the actual textual request itself. Only when there is a value in this field will the HTTP Operation be used, otherwise a GET request will be made.

Write Mode

The HTTP IO Controller has the capability to both perform 'insert' and 'update ' requests using different HTTP methods and URL's.

Encoding Method

The character encoding method for the data being exported.

Webservice Behaviour

The Webservice Behaviour to control the Read request. The webservice behaviour determines the base url, authentication, throttling & tracing.

Http Headers

Http Headers to be sent with each Write request.

The headers can be parameterised by specifying a field placeholders. The placeholders must correspond to the transaction selected in ‘Create File Per Transaction’ drop down of the corresponding Write transaction.

Insert Url

This may be only the post path or the full Url. The path is combined with Base Url to form the full Url.

The insert url may be parameterised by specifying a field placeholders. The placeholders must correspond to the transaction selected in ‘Create File Per Transaction’ drop down of the corresponding Write transaction.

Insert Operation

The operation to perform to perform an insert. By default this is set to POST which is typical of almost all webservices.

Modify Field

Use this drop down to specify when modify operations are performed.

When set the selected field is checked for a non-empty value. Where the value is non-empty the modify operation & modify url are used, allowing you to modify an existing resource. If either this property is empty or the selected field’s value is empty the insert operation & url are used.

Modify Operation

The operation to perform to perform an insert. By default this is set to PUT which is typical of most webservices.

Modify Url

This may be only the modify path or the full Url. The path is combined with Base Url to form the full Url.

The modify url may be parameterised by specifying a field placeholders. The placeholders must correspond to the transaction selected in ‘Create File Per Transaction’ drop down of the corresponding Write transaction.