Asynchronous Tracking

Introduction

There are two main purposes to AsyncTracking. One is to allow faster tracking than with the synchronous tracking by making each tracking independent of the others, so that the tracking party can continue its work without waiting for trackings to complete. The other purpose is to allow tracking to continue during service windows; where trackings are stored temporarily in the offline database and transfered to Link’s database when it’s ready.

The first part, supporting faster tracking, changes the way that interchanges and documents are identified. In the synchronous tracking scenario the server generates identifiers and the tracking party has to wait for the identifier before it can generate the next tracking. With asynchronous tracking, the tracking partner assigns a guid to each interchange and document and use these to continue tracking without waiting. When trackings are persisted to the Link database identifiers are still generated as usual, but trackings are resolved using the guids.

The second part, continue tracking during service window, also changes the way things work. All asynchronous trackings are initially written to a new offline database and then subsequently transfered to Links database. This transfer is handled by the async tracking job, which runs every minute (configurable) and ensures that all trackings in the offline database are transfered. When Link is offline, trackings remain in the offline database until Links database becomes online again, at which point they are transfered.

To facilitate offline operations, it is necessary to synchronize (one-way) users, permissions, clients and redirect urls to the offline database. The synchronization jobs runs hourly (configurable) to ensure this. Having a copy of these data allows the identity server to continue to issue tokens when the Link database is not available. Issued access tokens are the same and continues to work when switching between offline and online. The identity server automatically switches between offline and online mode. When it detects that it cannot access Links database it switches to offline mode and start a background task that checks every minute if it can revert to online mode.

During a planned service window the scheduling service should be stopped and thereby both jobs will cease operation. For async tracking to continue working, the web api must continue to run (on at least one web server). It is possible to set a ‘service unavailable’ flag and this will cause all the other api’s to return status code 503 (service unavailable) while async tracking continues to work.

Offline database

The offline database is intended to run on the web server, probably using a SQL Server Express instance, but that is not a requirement. It may also be installed on the same database server as Link or maybe elsewhere, where there is a database server. The requirements are that each web server has its own database and that the database and database server remain online. Obviously, if the offline database is not operational, neither is asynchronous tracking. The offline database connection string must be specified in the web.config file. See also: here.

Automatic switching between online and offline

The asynchronous tracking api always writes trackings to the offline database; from where they are processed and moved to the Link database by the async tracking processor job. The authorization service (IdentityServer3) automatically switches between on-/offline condition. For this to happen without much delay, it must be able to detect when the Link database is not available and it must timeout fairly fast. Therefor the connection string in use (for the Link database) must include the TIMEOUT option and the server name/ip must be prefixed with “tcp:”.

Furthermore, to avoid changing the connection string for all uses (web, Biztalk etc.) use the useConnectionStringsFromConfigFile setting in the web.config file for the Rest Api and change the configConnString to use the tcp prefix and TIMEOUT setting. The timeout should be as small as possible without causing other issues. Suggested value is 5 seconds.

  <appSettings>
    ...
    <add key="useConnectionStringsFromConfigFile" value="True" />
    <add key="configConnString" value="SERVER=tcp:dbserver;TIMEOUT=5;DATABASE=Link;Integrated Security=SSPI" />
    
  </appSettings>

Synchronization (Link → Offline database)

This job is in charge of synchronizing all relevant tables from Link to the offline database.

This job is run as part of the Scheduler Service.

It is important that only one instance of this job runs per offline database.

  <appSettings>
    ...
    <add key="configConnString" value="SERVER=tcp:dbserver;TIMEOUT=5;DATABASE=Link;Integrated Security=SSPI" />
    <add key="configConnOfflineDbString" value="SERVER=.\SqlExpress;DATABASE=Link.API;Integrated Security=SSPI" />
	<add key="JobConfiguration:EnableOfflineDbSyncJob" value="false" />
	<add key="JobConfiguration:OfflineDbSyncJobTriggerIntervalInHours" value="1" />
  </appSettings>

There are four configurations to keep in mind when setting up the OfflineDbSyncJob.

configConnString is the connection string for the Link database. This is required.

configConnOfflineDbString is the connection string for the offline database. This is required.

JobConfiguration:EnableOfflineDbSyncJob is used to either enable or disable the job in the scheduler service. If this is not set, it will default to false.

JobConfiguration:OfflineDbSyncJobTriggerIntervalInHours is used to determine how often the job will run in hours. If this is not set, it will default to 1 hour.

IdentityServer (online/offline mode)

Identity server issues tokens that tracking parties use to authorize against the api’s. This authorization is based on Link’s user and permission model and requires access to the list of users and permissions; furthermore it need access to api clients and associated redirect urls. To facilitate offline mode, where the Link database is not available, the previously mentioned data is synchronized to the offline database. The synchronization is optimized to include only the users that has api access permissions and only the permissions that relate to the apis. The synchronization is performed hourly (configurable).

When an authorization request is received by Identity server, it validates the request using data in the Link database. If this fails because of database access errors, Identity server switches to offline mode and retries the validation against the offline database. The tokens that are issues do not reflect against which database the validation occurred and they remain valid for their lifetime, regardless of online/offline mode switches.

Switching to offline mode also triggers a background job, that continues to check for access to the Link database and when it successfully connects, it switches Identity server back to online mode and the background job ends.

The implementation relies primarily on connection timeout to detect whether Link database is available and for that reason it is recommended to use a low timeout period, preferably around 2-5 seconds. The best value depends on the installation; generally it should be as low as possible, but not so low that it fails when it shouldn’t. To specify the timeout it is necessary to add a timeout property to the connection string and since this only should apply here, it must be changed in the web.config file, which must also include the option to use connection strings from the config file. See also: here.

Async tracking job

This job is in charge of resolving all the async tracking requests that have come in from the Link REST API.

This job is run as part of the Scheduler Service.

It is important that only one instance of this job runs per offline database.

In a load balanced scenario, this job can run on each server. Each server will have their own offline database, but all async tracking jobs will resolve their requests to the same Link database.

The job prioritizes requests in the following order:

1. Create document or interchange

2. Upload document or interchange

3. Everything else

This order minimizes the chance of requests failing.

If failures do occur, the requests are left in the offline database so they can be retried later. An increasing delay is added to the request each time it fails so they do not bloat the next batch.

  <appSettings>
    ...
    <add key="configConnString" value="SERVER=tcp:dbserver;TIMEOUT=5;DATABASE=Link;Integrated Security=SSPI" />
    <add key="configConnOfflineDbString" value="SERVER=.\SqlExpress;DATABASE=Link.API;Integrated Security=SSPI" />
	<add key="JobConfiguration:EnableAsyncTrackingBufferSyncJob" value="true" />
	<add key="AsyncTrackingBufferSyncJobTriggerIntervalInSeconds" value="10" />
	<add key="JobConfiguration:AsyncTrackingBufferSyncJobRequestBatchSize" value="100" />
	<add key="JobConfiguration:AsyncTrackingBufferSyncJobMaxRequestRetentionInHours" value="24" />
  </appSettings>

There are six configurations to keep in mind when setting up the AsyncTrackingBufferSyncJob.

configConnString is the connection string for the Link database. This is required.

configConnOfflineDbString is the connection string for the offline database. This is required.

JobConfiguration:EnableAsyncTrackingBufferSyncJob is used to either enable or disable the job in the scheduler service. If this is not set, it will default to false.

JobConfiguration:AsyncTrackingBufferSyncJobTriggerIntervalInSeconds is used to determine how often the job will run in seconds. If this is not set, it will default to 10 seconds.

JobConfiguration:AsyncTrackingBufferSyncJobRequestBatchSize is used to determine how many requests are retrieved from the database to be processed in a single loop. If this is not set, it will default to 100 requests.

JobConfiguration:AsyncTrackingBufferSyncJobMaxRequestRetentionInHours is used to determine when async tracking requests can safely be deleted from the database. If this is not set, it will default to 24 hours.

Automatic database cleanup (offline database)

The AsyncTrackingBufferSyncJob handles cleaning up the offline database.

During the process of finishing a batch, any request that is deemed to be too old, will be logged and deleted.

This limit can be set by the JobConfiguration:AsyncTrackingBufferSyncJobMaxRequestRetentionInHours configuration and will default to 24 hours if it is not set.

Service unavailable

In Service unavailable mode all api’s except asynchronous tracking return status code 503, which indicates that the service is not currently available. Service unavailable mode is enabled by toggling the app-setting with the same name in the web.config file. This is a manual operation (both ways) and it takes effect immediately, it does not require the web application to restart. It is meant as a way to keep asynchronous tracking online (in offline mode), while all the synchronous api’s becomes unavailable, fx. during a service window. See also: here.