Connecting to Data

The primary goal of Kelp is to help you build data-driven apps. Before you can build your app and analyze your data, you must first connect Kelp to your data. Kelp can connect to a variety of data sources: APIs, databases, streaming sources, and even static files. For example, you can connect to the following:

• Google Analytics using a RESTful API.
• A cloud database, such as Google BigQuery or Amazon Redshift.
• Multiple public datasets available on the web.

APIs

Kelp connects to external data sources using web APIs. Web APIs are APIs that can be accessed using the HTTP protocol. Kelp supports a range of API protocols, for example, REST, GraphQL, SOAP, and streaming (with some limitations).

Some APIs are open, while others are closed or secure.

Open APIs

Open APIs, also known as public APIs, are available to developers and other users without authentication.

Secure APIs

Most APIs you will come across are secure (for example, Google API, Dropbox API, and many others). A secure API accepts only authenticated requests from users.

APIs can use various schemes to authenticate users. Kelp supports the following authentication schemes:

• Basic Auth. Uses a username and password for authentication.
• API Key. Uses unique identifiers called API keys for authentication. Your app will send an API key with each API request.
• OAuth 2.0 / OpenID Connect. Authenticates and authorizes your app to access an API on your behalf. The app receives an access token and uses it in API requests. Many popular services use OAuth 2.0, for example, Twitter, Facebook or Google.

To connect to a secure API:

Step 1. Register your app.

Register your Kelp app with the API you want to connect to. For details on how to do so, refer to the documentation of the API provider. The exact procedure of the app registration differs depending on the API.

As a result, you'll receive the API credentials (for example, client ID and secret). You'll need them later to configure the authentication scheme during step 3.

Step 2. Create a connection.

1. Return to Kelp. Go to Home > Settings > Connections and then click Add Connection.

2. We've prepared a list of preconfigured connections to some popular APIs (for example, Google API, HubSpot API, and more). These connections will save you time and effort in configuring the APIs.

   If the API you want to connect to is on the list, select the API connection, click Next, and then go straight to step 3 "Configure the authentication scheme".

   If the API you want to connect to isn't on the list, create an API connection by clicking Create New.

3. Select the authentication scheme the API supports and click Next. Refer to the API documentation to find out which authentication scheme the API supports.

Note: Step 3 depends on the authentication scheme you've selected.

Step 3. Configure the authentication scheme.

Basic Auth

1. Give your connection a name.
2. Specify a username and password in the corresponding fields and click Next.
3. Verify the data you've specified and then click Submit.

API Key

1. Go to the API provider website and create API keys. The exact procedure differs depending on the API. Despite that, the result will be the same: you'll get a private key. Copy it and keep it safe.

2. Return to Kelp. Give your connection a name.

3. Paste the private API key value in the corresponding field.

4. Specify the following API key settings. To find out which settings to specify, refer to the documentation of the API you want to connect to.

   • Placement. Select how to send the authorization token: in a header or in a query.

   • Param name. Specify the header or the query parameter name, for example, Authorization or X-API-KEY.

   • Param value template. Specify the header or the query parameter value template, for example, Bearer {token}.

     Note: {token} here is a placeholder. Keep it as it is. No need to replace {token} with the actual value of the private API key. The data gateway does it for you during API requests. This way Kelp ensures that tokens are safe: neither visible to users nor stored as a part of the dataflow graph.

5. Click Next, verify the data you've specified, and then click Submit.

OAuth 2.0

1. Go to the API provider website and get OAuth 2.0 client ID and secret. To do so, use the link below as the Redirect URI:

   https://iam.kelp.app/app/oidc-external-auth.html

2. Return to Kelp. Give your connection a name.

3. Specify the following settings:

   • Client ID

   • Client Secret

   • Set the client credentials in the body or the header. By default, Kelp sends the client credentials in the Authorization header. To send them in the request body, turn on "Set client credentials in the body".

   • Authorization URL (required). The URL of the authorization server.

   • Token URL. The token URL can be optional depending on the token grant method.

   • Default scopes. Scopes are similar to permissions. They limit your app access to the API.

   • Advanced settings. To find out which settings to specify, refer to the documentation of the API you want to connect to.

   • Placement. Select how to send the authorization token: in a header or in a query.

   • Param name. Specify the header or the query parameter name, for example, Authorization or X-API-KEY.

   • Param value template. Specify the header or the query parameter value template, for example, Bearer {token}.

     Note: {token} here is a placeholder. Keep it as it is. No need to replace {token} with the actual value of the private API key. The data gateway does it for you during API requests. This way Kelp ensures that tokens are safe: neither visible to users nor stored as a part of the dataflow graph.

   • Authorize request params. Depending on the API requirements, you may need to specify additional authorization request parameters. Click Add parameter and specify the parameter name and value.

4. Click Next, verify the data you've specified, and then click Submit.

Note: Once you've configured the connection, you'll be able to change only the advanced settings and the default scopes.

OpenID Connect

You configure OpenID Connect in almost the same way you configure the OAuth2.0 schema. The only difference is that you specify "OpenID Connect URL" instead of "Authorization URL".

To configure OpenID Connect, refer to the procedure for OAuth 2.0.

Step 4. Create a part of the graph that will interact with the API.

1. Create an app or edit the existing one.

2. Place the following components on the canvas: String, HTTP, HTTP Credentials.

3. Connect the following components' ports with wires:

   • output of String with url of HTTP.
   • credentials of HTTP Credentials with credentials of HTTP.

4. Select the HTTP credentials component and press the space bar. Under "Connection", select the API connection you've configured earlier and click Done.

5. Select the String component, press the space bar, provide API endpoint URL, and then click Done.

Databases

Kelp can use relational, NoSQL, and OLAP database types as data sources. At the moment, you can connect Kelp to the following databases:

• MySQL
• Microsoft SQL
• PostgreSQL
• Oracle
• MongoDB
• Apache Druid

Most databases are secure and accept only authenticated requests from users. The most wide-spread authentication scheme for databases is Basic Auth, which Kelp supports. Basic Auth uses a username and password for authentication.

To connect to a database:

Step 1. Create a connection to a database.

1. Go to Home > Settings > Connections and then click Add Connection.
2. Click Create New to create a database connection.
3. Select the "Basic Auth" authentication scheme and click Next.
4. Give your connection a name.
5. Specify a username and a password in the corresponding fields and click Next.
6. Verify the data you've specified and then click Submit.

Step 2. Create a part of the graph that will interact with the database.

1. Create an app or edit the existing one.

2. Place the following components on the canvas: DB Credentials, SQL DB, and two Strings.

3. Connect the following components' ports with wires:

   • output of String 1 with connection-string of SQL DB.
   • output of String 2 with query of SQL DB.
   • credentials of DB Credentials with credentials of SQL DB.

   Note: To tell one String component from another, you may want to rename String 1 "DB Connection String" and String 2—"SQL Query". To rename a component, double right click the component name and type the name you want.

4. Select the DB Credentials component and press the space bar. Under "Credentials", select the database connection you've configured earlier and click Done.

5. Select String 1, press the space bar, provide the URL of the database you want to connect to, and then click Done.

6. Select String 2, press the space bar, provide the database query you want to perform, and then click Done.

Streaming Data (beta)

Kelp can use streaming data as data sources. At the moment, Kelp supports SSE and Apache Camel standards.

Note: Connecting Kelp to streaming data is still under development. We can't describe all the necessary configuration steps in our documentation yet. If you want to connect your app to streaming data, we'll personally help you in our Kelp community in Slack.

Static Data

In Kelp, you can store data as a part of the dataflow graph. We call such data static. To add static data to the graph, use the Const components: Array, Boolean, JSON, String, and so on.

Static data has the following limitations:

• It can't be added or updated during the app runtime. You can only populate static data when you update the dataflow graph.
• Everyone who has access to the dataflow graph can access the static data contained inside.
• Large static datasets can slow down the initial loading of the app.

Note: We recommend restricting static data usage only to defining static dictionaries, reference data, mappings, and constant parameter values.