WSO2 Choreo — New Variant of iPaaS and Its Connector Model

Needlessly to say, world is full of data. In earlier days most of the applications built in an organisation dealt with data internal to it, typically gathered from on-premise systems. It is no longer the case now — to build applications they need to integrate data from various external systems which is increasingly challenging and time consuming to do from scratch. Sometimes those external systems operate on different protocols, needs to know their message formats and certain domain knowledge is required to work with them as well. This forces companies to seek for professionals with specific knowledge for each area, which would increase project costs and would also generate a scalability problem in the long run.

This is one of the reasons organisations are now moving on to iPaaS solutions where they can configure and connect to external systems easily and integrate data from various sources. Some popular such platforms are Oracle Integration Cloud, TIBCO Cloud, Zapier, MuleSoft Anypoint Platform and Celigo. However, these platforms are either complex or lack of customisations (template driven). Again, organisations on their digital translation are in trouble.

WSO2 has recently released its new iPaaS solution called “Choreo” which can address both of the above problems. Charles Humble wrote to Thenewstack

In June, a new integration platform entered the field, with features that are designed to help bridge the gap between low code and experienced developers: WSO2 has announced Choreo, currently in public beta, a low-code iPaaS for Kubernetes, built on top of the company’s open source programming language, Ballerina.

Let’s see what above statement means.

Choreo

Today people’s requirements change rapidly. Hence, organisations cannot develop applications and keep them intact. As the business changes these applications needs to be modified. Business people are usually not technical experts. So they need some UX driven high level approach to “configure” the system according to the business change. This is why modern iPaaS solutions have low-code application platforms (LCAPs) in place. Choreo has a low code approach as shown in below screenshot for citizen integrators.

Scenario: When a GitHub issue is assigned send an alert via SMS

However, if requirement change needs more complex customisation, that also needs to be supported by the iPaaS. In this case, it will be a joint effort by business people and technical developers in the organisation to implement the solution. To facilitate this, the iPaaS has to support pro code approach as well. Interestingly, Choreo runs on Ballerina language specially designed for data integration and network communication. Owing to this underlying language support any customisation is possible. It enables professional developers to develop using low-code and pro-code simultaneously as well.

Low Code and Pro Code integrated together

When low code UI is configured, respective Ballerina program code is generated automatically. If you edit the code, UI view is updated automatically.

Above should have given you a basic idea about what Choreo is and its strengths. Now let’s see what it provides to enable communication with different external platforms.

Connectors at Choreo

In Choreo, the unit used to communicate with a particular external system is called a connector. External systems expose their functionalities in a standard manner using styles like REST, gRPC and GraphQL. Using Ballerina language support connector packages are implemented and then onboarded onto Choreo.

• How are they presented in Choreo: as a unit you can select and use in UI (Low Code Editor)
• Where the connectors are hosted: at Ballerina Central
• Where is the source: at GitHub repository Ballerina Platform

At Choreo, a connector comes into the picture in two ways depending on the way it is used to construct application logic.

1. As a Listener — listen to the incoming events from an external system (i.e Google Calendar events). Internally, Webhook or an HTTP listener is used in the connector. This can be the starting point of an integrated message flow in a Choreo application.
2. As a client — call an external system to post or get data (i.e. send a SMS using Twilio). At Choreo, when you select one of the remote functions available, respective inputs needed by that function are loaded as a form to fill.

Connector as a listener and a client

Currently, connectors are displayed as a plain list to choose. In near future, Choreo will have a comprehensive way to search for the connector you need for your integration — via tags and enterprise domains.

Connectors available to choose

Features

Connector operation configuration view

• When a connector operation is selected, the respective API documentation is pulled from Ballerina central and displayed. To know what are the inputs and their meanings user does not need to navigate anywhere else — it is loaded at the low code environment itself.
• Mandatory fields are marked with a (*) and presented by default in the pop up form to fill.
• Auto suggestions for Ballerina expressions — Choreo UI exchange messages with the Language Server to pull the suggestions using the Language Server Protocol, an open, JSON-RPC-based protocol, originally developed by Microsoft.
• Output fields are accessible with a (.) — Connectors operations are designed to return ballerina records as output. What this means is output can be manipulated by language features easily. If you assign the output to a variable and then type “.” output fields are loaded. This way, at implementation time you know what would be the result of this operation.
• Static value with (“”) — to input static values to connector fields use this notation.
• Data mapping features and support for conditional routing, looping and writing statements.

Authentication support

Before you use a connector, you need to authenticate to the remote API or the service. There are various options available depending on the connector and below are some seen in common.

• No authentication : Some connectors do not need any authentication as they use public APIs (i.e COVID-19 connector)
• OAuth 2.0 support : wherever OAuth 2.0 support is there in third party API, you can use SSO to login to your account and use the connector (except for Google connectors ATM). It is easy as you can avoid the process of obtaining tokens from remote applications.
• Manual connector connection config: If you already have an account or your organisation has granted you an account on a third party application, manually you can configure the connection using the tokens when you use the connector for that application at Choreo.
• Automatic token refresh: Access token renewal happens automatically using the refresh token. Application developer who uses the connector do not need to do any additional effort to do it.
• Automatic web hook connection refresh: If you use a listener, usually server will expire the connection after a certain time. However Choreo users do not need to worry on this. Web hook connection re-init upon expiry is also handled internally. You can keep on listening to events.
• Temporary connection support: If you are given a Bearer token to use temporary to implement or test your Choreo application, you can configure it in the connection to connect to the external system. When you go production, you can change to OAuth 2.0 without any logic changes.
• Support for API key and JWT tokens: Some APIs use API key or a session key as a header. In such scenarios you can configure the connection with the API key or the session ID you obtained by registering to the third party service. When JWT support is there, in the connection config form that would get rendered at Choreo, you can define the inputs to generate the JWT token that should be sent to the backend

Connector versioning at Choreo

As you already know now, at Choreo connectors are imported from Ballerina central — which means these are Ballerina modules. They can get updated when eco-system team (this is how we call connector development team at WSO2 :)) release new versions of connectors to Ballerina central. These might be bug fixes, function improvements, new implementation on the next API version, improvements brought forward due to improvements in underlying Ballerina language. Choreo is automatically kept up to date with latest backward compatible version. For major versions, soon it will have a drop down for the Choreo users to select which connector version they prefer to use for their application.

Data mapping and transformation

After invoking an external system, the user who develops a Ballerina program using Choreo will have to either return the result to the caller or transform it and forward appropriately to the next service.

Users can extract data or sections from the response message using Ballerina language constructs. To make this task easier to the user, Choreo is comprised of a data mapping tool.

Extensibility and customisations

Language tools and other features

Of course, Choreo is the low code way of writing a Ballerina program. When you configure the message flow at Choreo, in the back it generates the Ballerina code. Thus, what Choreo does is bringing out the Ballerina language power to the frontend in a simplified way providing the support for different Ballerina language constructs. Following three are basic essentials to construct any programming logic, which is configurable via low code environment as well.

• Variables and statements— write a statement and assign it to a variable so that you can use later in the application logic
• Conditions — if statements
• Loops — Foreach and While for conditional looping

This essentially means that you can configure any programming logic using Choreo constructs.

Other Choreo features

It worths mentioning below Choreo features that would help using the connectors in a more managed manner at Choreo iPaaS. Note that these features are not limited to connectors. Check them out!

Deploy as a periodic task

Once you have configured how the program or service integration should work, when you deploy the Choreo application containing that logic, you can choose it to run periodically. For an example, you can schedule a logic to run every day at 6 A.M before your team commence work which will go through GitHub repos your team work on and update the issues on a Google Sheet.

Observability

When you expose your integration as a service, in the observability tab you can see the throughput and latency of each resource from last 30 days to past 10 minutes. In the diagnostic view it shows memory and CPU usage along with any errors occurred in the timeline.

Troubleshooting

When your program has run the execution history shows which calls were successful and which were not. Also average latency of that call is also displayed. This information is useful for troubleshooting at which point the external invocations get failed.

You will soon get

Choreo is still at its young ages. At WSO2, we are doing continuous innovation and research to make Choreo iPaaS more user friendly, performant, intelligent and observable. Expect more on near future! These are some related to connectors.

• Connection sharing facility — configure a connection to a remote system and share it within your organisation. Good for collaborative testing also.
• Open API connectors — convert your Open API (Swagger) 3.0 file into a connector instantly and use it in your application in a Ballerina native manner.
• CDATA connectors — we will be exposing capability to connect to external systems through CDATA SQL interfaces.