Ingoing Transport Types

If you have a need to create ingoing transport locations in Link, and none of the existing plugins fulfill your needs, you may need to develop a new plugin yourself.

Please consider whether the functionality you need is sufficiently generic that it may be of general use, and decide accordingly whether to make the plugin customer specific or not.

Requirements

Two things are required to make a new plugin: a class implementing a specific interface, and a Link Transport Type defining which class to call at runtime (enclosed in a DLL file) and which configuration values to expose to the GUI.

Interface

Ingoing adapter plugins need to implement the following interface:

C#

namespace Bizbrains.Link.Polling.Adapters
{
    public interface IReceiveAdapter
    {
        void Configure(Dictionary<string, string> config);

        string DisplayAddress();

        IEnumerable<AdapterMessage> Poll(System.Threading.CancellationToken ct, IEnumerable<ILogger> loggers);
    }
}

The interface itself is fairly simple, but the implementation of some of the methods can be quite complicated.

In addition to implementing the interface, the project needs references to Bizbrains.Link.Polling.Adapters (which contains the IReceiveAdapter interface as well as the AdapterMessage class and other basic components) and to Bizbrains.Logging (for the ILogger interface).

Configure()

This method is used to set the runtime configuration of the adapter. The configuration values are delivered as a dictionary with string values for each configured TransportTypeField. The Key of the dictionary is the name of the TransportTypeField.

The Bizbrains.Link.Polling.Adapters namespace exposes a number of extension methods to help with extracting typed configuration values from this dictionary.

A few examples:

C#

_folder = config.ExtractString("Folder", false);
_deleteAfterGet = (bool)config.ExtractBool("DeleteAfterGet", true);
_proxyPort = config.ExtractInt("Proxy_Port", true);

In your implementation of the Configure() method please remember to validate configuration values as necessary, so that configuration values that have to be set in order for the adapter to work will result in an error if they are missing or have a value that is not valid.

DisplayAddress()

This method is used by the polling framework to extract the address a message came from. This is displayed on the document details view in Link and is equivalent to the URI field on a Biztalk receive location. Depending on the type of adapter you are making the precise formatting of the address may vary.

As an example, here is the implementation from the SFTP adapter:

C#

public string DisplayAddress()
{
    if(String.IsNullOrEmpty(displayAddress))
    {
        if(!String.IsNullOrEmpty(_server))
        {
            // Create the display address for this adapter configuration.
            displayAddress = "sftp://" + _server + ":" + _port.ToString() + "/" + (_folder != null && _folder.Length > 0 ? _folder + "/" : "") + _fileMask;
        }
    }
    return displayAddress;
}

This gives an address looking something like this: sftp://sftp.exampleserver.com:22/ToLink/*.xml

Poll()

This is the big one, the method where all the magic happens.

Depending on the type of adapter you are making, there are various considerations to take into account which may affect the implementation.

The most important thing is to avoid data loss. If at all possible, your adapter should ensure that a message has been successfully saved to Link before forgetting about the message, and if it hasn’t been saved to Link then make sure that the message can either be polled again later on (preferable) or is persisted somehow.

In order to better understand how to do this, it is useful to know how the polling framework interacts with this method.

Here is a simplified version of relevant part of the polling framework’s code:

C#

IEnumerable<AdapterMessage> messages = receiveAdapter.Poll(ct, Loggers);
try
{
    foreach (AdapterMessage msg in messages)
    {
        fetchedMessages.Add(msg);
        try
        {
            SendToInbox(msg, transportLogGuid);
            msg.SavedToLink = true;
        }
        catch (Exception ex)
        {
            Loggers.LogException(ex);
        }
    }
}
catch (Exception ex)
{
    Loggers.LogException(ex);
    throw;
}

On the surface, this looks fairly simple. The receive adapter returns a list of messages, the polling framework loops through the list and saves each message to Link.

In reality, how this actually works at runtime can be somewhat more complicated, depending on how the adapter is implemented. Note that the framework sets the SavedToLink flag on the message to true after saving the message. This flag is actually meant for the adapter, letting it know that the message has been saved successfully so it can delete/invalidate/acknowledge the message wherever it got it from, and if the message hasn’t been saved successfully, to do whatever it needs to make sure the data isn’t lost.

But how will the adapter know what the flag is set to when the Poll method has already returned? By ensuring that the Poll method does not simply return. This is done by use of the C# keyword yield. If you are familiar with the use of this keyword, you can probably just skim the next part quickly. Otherwise, read on.

yield return

A method which has an IEnumerable as the result can use yield return instead of just return. This changes how the method behaves at runtime. Anyone calling the method do not need to be aware of this, but it opens up some possibilities for better control of the flow.

In order to better understand what happens, let us replace the foreach call in the framework code with something closer to what the compiler actually converts it to.

Instead of this:

C#

foreach (AdapterMessage msg in messages)
{
    [Code logic here]
}

We will think of it as this:

C#

AdapterMessage msg;
IEnumerator<AdapterMessage> enumerator = messages.GetEnumerator();
msg = enumerator.MoveNext();
while(msg != null)
{
    [Code logic here]
    msg = enumerator.MoveNext();
}

An IEnumerable method using yield return instead of just return essentially “returns” immediately without actually running its code. What it returns is a pointer to an IEnumerable object. When the MoveNext() method is called on the enumerator of the IEnumerable object, the code runs until it encounters a yield keyword, and then hands that object back to the caller. The next time the MoveNext() method is called, the code continues running from right after the yield keyword. Eventually, the code should reach the end of the method without hitting a yield keyword, at which point the result of MoveNext() will be a null value, and the while loop ends.

As an example, here is an extremely simplified version of the Poll method used in the SFTP adapter:

C#

public IEnumerable<AdapterMessage> Poll(System.Threading.CancellationToken ct, IEnumerable<ILogger> loggers)
{
    Queue<SimpleFile> fileQueue = new Queue<SimpleFile>();
    try
    {
        Connect(); // Connect to the SFTP server.
        fileQueue = GetFileList(); // Get the list of files in the configured folder matching the configured filemask.

        while (fileQueue.Any() && !ct.IsCancellationRequested) // Loop through the retrieved list of files.
        {
            SimpleFile file = fileQueue.Dequeue();
                    
            DownloadFile(file); // Download the current file from the SFTP server.

            AdapterMessage tmpMsg = new AdapterMessage(file);
            tmpMsg.Properties.Add(new MessageProperty() { Name = "ReceivedFileName", Namespace = "http://schemas.microsoft.com/BizTalk/2012/Adapter/sftp-properties", Value = file.FileName, Type = typeof(System.String) });

            yield return tmpMsg; // Return this AdapterMessage back to the foreach loop in the polling engine.

            // Execution will continue from here when the polling engine asks for the next message.
            if (tmpMsg.SavedToLink) // At this point, the polling engine has tried saving the file to Link.
            {
                DeleteFileFromServer(tmpMsg); // The file has been saved to Link, so delete it from the server.
                tmpMsg.CompletedByAdapter = true;
                continue;
            }
        }

        if (ct.IsCancellationRequested) // The polling process has been asked to stop, possibly due to the service restarting.
        {
            Loggers.Log(LogLevel.Warning, "Polling halted midway due to cancellation from the service.");
        }
    }
    finally
    {
        Close();
    }
}

If you follow the code above you will note that when MoveNext() is called after the first time, execution continues here while still having the same tmpMsg object that was returned to the polling framework previously. As such, the adapter can now see from the object whether the message was successfully saved to Link or not, and act accordingly. It can also indicate on the message whether it itself did what it needed to successfully. The polling framework can then use this information (mostly for logging purposes) at the end of the poll.

CancellationToken

Another thing you may need to take into account is the cancellation token passed to the Poll method. Essentially, if there is a reasonable chance that a single poll may take a long time to complete, and the polling process can be halted midway with no ill effects, we need to be able to do that. This is so that the polling service can be stopped or restarted in a reasonable amount of time without breaking anything. The way the cancellation token works is that when the service is told to stop, it sets the IsCancellationRequested flag in the cancellation token, to let the various adapters know that it would like them to finish what they are doing ASAP.

In the method from the SFTP adapter above, this is quite simple. Since we are looping through a list of files and each file is an independent message, it is not a problem to simply stop downloading files partway through the list. The rest of the files will simply be picked up on the next poll. At the same time, we don’t stop the process in the middle of dealing with a single file. We finish that up, and then check if we have been asked to stop before continuing with the next file in the list.

Logging

Logging is done via the IEnumerable<ILogger> object given to the Poll method. There are extension methods for it so you can treat it as a single logger object even though it is a collection.

Anything you log in the adapter will be put into the TransportLog of the interchanges created during the poll as well as the polling log of the incoming transport location running the adapter and the event log of the server hosting the polling service. As such, it is fairly important that you log things at the correct log level.

Anything that is normal to the execution of the adapter (e.g. “Polling started” or “Connecting to server sftp.exampleservername.com”) should be logged as LogLevel.Info. This means that it can be left out of the event log under normal circumstances, making the event log easier to deal with.

Anything that is technically an error but will still allow the poll to continue should be logged as LogLevel.Warning, unless it is something that might cause problems later (e.g. not being able to delete a file from an SFTP server, in which case it will be picked up again on the next poll), in which case you may choose to log it as LogLevel.Error.

Anything that prevents the polling process from completing normally should be logged as LogLevel.Error.

Transport Type

Once you have created an adapter plugin, you need to register it in Link as a transport type in order to be able to use it on incoming transport locations. This is done via an SQL script. You can use the administration tool to generate the script, but it is recommended to write it manually, using an existing script as a template.

The easiest script to use is this: https://bizbrains-productteam.visualstudio.com/Link/_versionControl?path=%24/Link/Main/Common/Bizbrains.Link.Polling/Adapters.AsyncSend.SFTP/SQL%20Scripts/CreateOrUpdateTransportType.sql

The one limitation this script has is that it puts all DataFields into a single DataField group. If you really need to have your fields split up into different groups, you will have to either modify the script in order to do this, or use another method.

There are a few thing to be aware of and to take into account when you design your transport type.

Transport type name

If you a making a customer specific adapter designed to fetch messages from a specific REST API, do not name it something as generic as RestAPI or HTTP. While there are currently no generic ingoing RestAPI or HTTP adapters, there may be in the future, and having a very specific transport type with such a generic name is sure to cause confusion and possibly technical problems due to unique constraints in the database.

Transport type field names

Keep in mind that the names of the transport type fields you define are the same ones you need to extract from the dictionary in the Configure method in the adapter. As such, they need to match 100%.

Data field names

The names of the data fields linked to the transport type fields are the ones that will be seen in the GUI. As such, if you have a transport type field called “NumberOfMessagesPerPoll”, it is probably nicer for the user if you call the data field “Number of messages per poll”.

Data field types

Please use appropriate types for the various data fields. Checkboxes for Booleans, dropdowns for enums, etc. Forcing the user to write “True” or “False” in a textbox is not nice. And remember to always use the Password type for passwords.

Data field groups

As the GUI groups the various configuration fields according to the data field groups they belong to, consider whether it is appropriate for your adapter to use multiple data field groups. This could for instance be if your adapter has settings for different distinct features, not all of which are mandatory and/or commonly used. For an example, please look at the ingoing SFTP transport type.

Deployment

Deploying a polling plugin consists of two steps: deploying the assembly itself (along with any referenced assemblies the plugin needs) and running the SQL script that creates the ingoing transport type.

The assembly (and references) should be deployed to the folder the polling service is running from. Depending on the Link system you are working with, this could be on a web server or the Biztalk server itself. If the service is hosted on a server you do not have permission to make changes to yourself, you will need to ask the assistance of the Operations team.