the HTTP IO Controller

The HTTP IO Controller provides the means to read from/write over an http/https. For JSON and Xml formats the Http Controller provides advanced webservice capabilities for integration with almost any REST and some (simple) SOAP services.

Jump To

Read Mode

Readers will typically perform a GET operation to query data from a webservice.

Encoding Method

The character encoding method for the data being queried, typically by UTF-8.

See Character Encoding for more details. 

Webservice Behaviour

The Webservice Behaviour to control the Read request. The webservice behaviour determines the base url, authentication, paging, throttling & tracing.

Paging

When a result set is paged the Read operation will generate multiple requests. Each request to obtain a page of data is transparent to the Read operation.

Http Headers

Http Headers to be sent with each Read request.

Evaluate URL

When checked the URL will be evaluated via a VBScript expression.

URL

The URL of the request. This may be only the query path or the full Url. The Query path is combined with Base Url from the Webservice Behaviour to form the full Url.

The URL may be evaluated:

  • Not evaluated.
    • The URL should be a static string.
  • Evaluated
    • Evaulated URLs give you the ability to parameterise any request. In order to parameterise the reader it must have a parent transform, where the parent transform will provide the input fields (and values).
    • Evaluated URLs should evaluate to a valid URL. The evaluated expression can contain any valid VBScript function.
    • Evaluated URLs can reference any field which is from a mapped parent transform.
    • Evaluation applies only the URL displayed on the Options tab. Evaluation is not performed on URLs for stepped Reader requests.

    Example

    "/orders.json?updated_at_min=" & Format(DateAdd("d", -1, %[SYS.LASTRUNDATE]), "yyyy-MM-ddThh:mm:ssZ")

    Assuming SYS.LASTRUNDATE is 1st April 2020, 6:30 PM the above expression would evaluate to:

    /orders.json?updated_at_min=2020-03-31T18:30.00Z

    • The DateAdd function is subtracting a single day from the last run date to return 31st March 2020, 6:30 PM.
    • The Format function is formatting 31st March 2020, 6:30 PM to an ISO8601 style date: 2020-03-31T18:30.00Z
    • The formatted date is appended to '/orders.json?updated_at_min='

Paging

For paged results the URL is the initial request. Requests for subsequent pages of data are dictated by the paging mode.

Post / Put Request to Read Data

By default a Read operation will issue a GET request. It is possible to have IMan send a POST, PUT, MERGE, PATCH or DELETE request with a corresponding request body.

HTTP Operation

Set the Http Operation to one of values POST, PUT, MERGE, PATCH and DELETE.

The selected value will only be used when the Request Body (next field) is populated.

Request Body

This field contains the request body.

The Request Body can be parametrised with expando values (in the same way as the URL).

Only when there is a value in this field will the HTTP Operation be used, otherwise a GET request will be made.

The request body can be either:

  • The path to the file containing the body. This can be useful when the request body is large or when the request is being created dynamically.
  • The actual request body.

Write Mode

The HTTP IO Controller has the capability to both perform 'insert' and 'update ' requests using different HTTP methods and URL's.

Encoding Method

The character encoding method for the data being exported.

Webservice Behaviour

The Webservice Behaviour to control the Read request. The webservice behaviour determines the base url, authentication, throttling & tracing.

Http Headers

Http Headers to be sent with each Write request.

The headers can be parameterised by specifying a field placeholders. The placeholders must correspond to the transaction selected in ‘Create File Per Transaction’ drop down of the corresponding Write transaction.

Insert Url

This may be only the post path or the full Url. The path is combined with Base Url to form the full Url.

The insert url may be parameterised by specifying a field placeholders. The placeholders must correspond to the transaction selected in ‘Create File Per Transaction’ drop down of the corresponding Write transaction.

Insert (HTTP) Operation

The HTTP operation to perform - the controller supports POST, PUT, PATCH, MERGE and DELETE. By default this is set to POST which is typical of almost all webservices.

DELETE Operation

When the operation is set to DELETE the modify operation (the Field and HTTP operation) properties will be disabled.

DELETE operations can either have a body or none. If the body should be empty, leave the setup for the (JSON and Xml) writer empty.

Modify Field

Use this drop down to specify when modify operations are performed.

When set the selected field is checked for a non-empty value. Where the value is non-empty the modify operation & modify url are used, allowing you to modify an existing resource. If either this property is empty or the selected field’s value is empty the insert operation & url are used.

Modify Operation

The operation to perform to perform an insert. By default this is set to PUT which is typical of most webservices.

Modify Url

This may be only the modify path or the full Url. The path is combined with Base Url to form the full Url.

The modify url may be parameterised by specifying a field placeholders. The placeholders must correspond to the transaction selected in ‘Create File Per Transaction’ drop down of the corresponding Write transaction.