(Kotlin) How to Unfurl Links

What will we do

We'll create an application that unfurls Slack links. The preview of the referenced Slack message will appear in Space chats, documents, and other places that support link unfurling. This will be a read-only preview without the ability to answer in Slack directly or navigate to the message thread. If you don't want to pass this tutorial step by step and want just to look at the resulting code, it's totally OK – here's the source code.

Step 1. Create a Ktor project

First, we'll need to create a Ktor application. You can find the detailed instructions about how to do that in step 1 of the (Kotlin) How to create a chatbot tutorial. Follow these steps, paying attention to the following aspects:

• Ktor version. The tutorial assumes using Ktor 2.0.1. You can choose any other version, but some Ktor-related imports and method calls might need to be updated accordingly.

• Space SDK. Be sure to pick the freshest version as described here.

• We're not going to write any tests for this application, so clean up all testImplementation dependencies from the build.gradle file and delete the src/test folder in the project root.

• The application needs the Slack SDK for Java, so make sure you add the following dependency to the dependencies section of build.gradle: implementation "com.slack.api:slack-api-client:1.18.0"

The resulting build.gradle file should look like this:

Step 2. Run a tunneling service

While developing the application, we will need to expose it publicly so that the Space server can reach it. The easiest way to make this possible is to use a tunnelling service. The process of setting up a tunnel by using the ngrok service is described in details in step 3 of the (Kotlin) How to create a chatbot tutorial.

Step 3. Register the application in Space

Our application will talk to both Slack and Space, so we need to register it on both sides. Let's start with Space. We will go the easy way and create a single-org application as we're just learning. This means that the application can't be distributed to other Space organizations through JetBrains Marketplace. Yes, it sounds like a drawback, but on the other hand, this allows us to configure the application manually in the Space UI. In case of a multi-org application, we would need to implement the configuration logic in the code of our application.

1. Open your Space instance.

2. On the main menu, click Extensions, then Installed.

3. Click New application, give your application a unique name, e.g. Slack unfurls. Click Create, then Go to application settings.

4. Open the Endpoint tab.

5. In the Endpoint URL, specify the endpoint of your application where it should receive calls from Space. Construct this URL by appending /api/space to the public URL of your ngrok tunnel. Also make sure that Public key is selected as the authentication method on this tab. Click Save.

6. Open the Authentication tab. Note that the Client Credentials Flow is enabled by default. Our application will use this flow to authorize in Space on behalf of itself.

7. Copy the Client ID and Client secret values from the Application credentials section.

8. In the application source code, create the Credentials.kt file and add the following code to it (replace the URL, the client ID and secret values with the values from the previous steps):

   package com.linkpreviews import space.jetbrains.api.runtime.SpaceAppInstance import space.jetbrains.api.runtime.SpaceAuth import space.jetbrains.api.runtime.SpaceClient import space.jetbrains.api.runtime.ktorClientForSpace val spaceAppInstance = SpaceAppInstance( clientId = "<Space app client id>", clientSecret = "<Space app client secret>", spaceServerUrl = "https://<your-Space-org>.jetbrains.space" ) private val spaceHttpClient = ktorClientForSpace() val spaceClient by lazy { SpaceClient(spaceHttpClient, spaceAppInstance, SpaceAuth.ClientCredentials()) }

That's it for the Space application setup. Now let's move on to the Slack side.

Step 4. Configure the application in Slack

1. Navigate to https://api.slack.com/apps in the browser. Sign in to your workspace.

2. Once you're logged in, click Create New App.

3. In the dialog, choose the From an app manifest option.

4. Choose the required Slack workspace.

5. Choose the YAML format and provide the following manifest, replacing your-ngrok-url with the public address of your ngrok tunnel:

   display_information: name: Slack unfurls in Space oauth_config: redirect_urls: - https://your-ngrok-url/slack/oauth/callback settings: token_rotation_enabled: true

6. Review the application settings and click Create.

7. On the settings page, scroll down to the App Credentials section containing the client id and client secret. Copy these values as our application will need them to communicate with the Slack API.

8. In the application source code, add the following code to Credentials.kt (replace the URL, client id and secret values with the values from the previous steps):

   // ... other imports import com.slack.api.Slack // ... Space credentials and client related code object SlackWorkspace { val clientId = "<Slack app client id>" val clientSecret = "<Slack app client secret>" val domain = "<your Slack domain>.slack.com" } // list of Slack permissions needed to fetch link preview contents val slackPermissionScopes = listOf( "channels:history", "groups:history", "channels:read", "groups:read", "team:read", "users:read", "usergroups:read" ) val slackApiClient = Slack.getInstance()

Step 5. Request permissions in Space

First thing the application should do after it is added to a Space organization is to configure itself and request all necessary permissions. In step 3 we've mentioned that we create a single-org app. This means that we can request the required permissions right in the application settings in Space. Nevertheless, we offer you to take a sneak peek at how multi-org applications are created and request the permissions right from the application code.

Typically, an application must require permissions during the initialization phase, after it receives the InitPayload from Space. But, as our app is not a 100% multi-org application, we will initialize it by sending the init command to the application chat channel.

1. Create the CommandInit.kt file with the following code inside:

   package com.linkpreviews import space.jetbrains.api.runtime.resources.applications import space.jetbrains.api.runtime.resources.chats import space.jetbrains.api.runtime.types.* suspend fun commandInit(spaceUserId: String) { // request global permission to provide link previews spaceClient.applications.authorizations.authorizedRights.requestRights( ApplicationIdentifier.Me, GlobalPermissionContextIdentifier, listOf("Unfurl.App.ProvideAttachment") ) // allow providing previews for the specific domain (our Slack workspace) // you can specify just slack.com here - Space will expand it to subdomains spaceClient.applications.unfurls.domains.updateUnfurledDomains(listOf(SlackWorkspace.domain)) // send “OK” to inform the user that init was successful spaceClient.chats.messages.sendMessage( channel = ChannelIdentifier.Profile(ProfileIdentifier.Id(spaceUserId)), content = message { section { text("ok") } } ) }

   The code above requests the permission for attachment link previews – pieces of content that appear next to chat messages or in tooltips. Learn more about previews.

2. Now we need to connect the commandInit function to the Ktor HTTP endpoint that handles Space commands, and pass a ready-to-use Space client into it. Let's create the Routes.kt file with the following code inside:

   package com.linkpreviews import io.ktor.http.* import io.ktor.server.application.* import io.ktor.server.request.* import io.ktor.server.response.* import io.ktor.server.routing.* import io.ktor.util.* import space.jetbrains.api.runtime.helpers.readPayload import space.jetbrains.api.runtime.helpers.verifyWithPublicKey import space.jetbrains.api.runtime.types.ChatMessage import space.jetbrains.api.runtime.types.MessagePayload fun Routing.api() { // register route listening to POST requests at api/space post("api/space") { // read request body val body = call.receiveText() // verify that the request comes from Space val signature = call.request.header("X-Space-Public-Key-Signature") val timestamp = call.request.header("X-Space-Timestamp")?.toLongOrNull() if (signature.isNullOrBlank() || timestamp == null || !spaceClient.verifyWithPublicKey( body, timestamp, signature ) ) { call.respond(HttpStatusCode.Unauthorized) return@post } // read and process the payload when (val payload = readPayload(body)) { // determine message payload type is MessagePayload -> { // determine the init command if ((payload.message.body as? ChatMessage.Text)?.text == "init") { commandInit(payload.userId) } call.respond(HttpStatusCode.OK, "") } } } }

3. Finally, let's configure the routing feature so that it uses our Routing.api() function. Open the Routing.kt file in the plugins directory and edit it as shown below:

   package com.linkpreviews.plugins import io.ktor.server.application.* import io.ktor.server.routing.* fun Application.configureRouting() { routing { api() } }

Step 6. React to links posted to Space

Our application has to be notified when a new Slack link appears in a chat channel, document or issue description. Each time such link is posted, Space persists it to the special unfurling queue and sends a notification. Let's take a look at how to use the Space API for polling the queue.

The notifications from Space are just another type of payload that Space can send to the same /api/space endpoint of our application. To handle this new payload type, let's expand the when statement in the Routes.kt file.

1. First, add new imports and a global variable for tracking the last etag value:

   import space.jetbrains.api.runtime.resources.applications import space.jetbrains.api.runtime.types.NewUnfurlQueueItemsPayload import space.jetbrains.api.runtime.types.ProfileIdentifier private var lastEtag: Long? = null

   What's etag? Etag is a number that defines the place of an item in the unfurling queue. The number is guaranteed to be strictly increasing. In a real application, we would store it in some persistent storage, but for the sample application it's OK to store it as an in-memory variable.

2. Add handling of the NewUnfurlQueueItemsPayload type to our when statement:

   when (val payload = readPayload(body)) { // ... // handle notifications about new items in the unfurling queue is NewUnfurlQueueItemsPayload -> { val queueApi = spaceClient.applications.unfurls.queue // get queue items in batches of 100 items at a time var queueItems = queueApi.getUnfurlQueueItems(lastEtag, batchSize = 100) while (queueItems.isNotEmpty()) { queueItems.forEach { item -> // 'authorUserId' - is an ID of the user who posted a link (instance of 'ProfileIdentifier.Id') // Checking that a link points to a Slack message and link author is a Space user. // The reason is that we want to query Slack API on behalf of a user, not an application. // Thus, the app has to map each Space user to their personal token in Slack. if (item.authorUserId != null && item.target.startsWith("https://${SlackWorkspace.domain}/archives")) { val spaceUserId = (item.authorUserId as? ProfileIdentifier.Id)?.id ?: error("ProfileIdentifier.Id") provideUnfurlContent(item, spaceUserId) } } // save our current position in the queue lastEtag = queueItems.last().etag queueItems = queueApi.getUnfurlQueueItems(lastEtag, batchSize = 100) } } }

   Note that this code won't compile yet, because we have to create the provideUnfurlContent function. We'll do this in the next step.

Step 7. Request a Space user to authenticate in Slack

Let's now look at how the application can ask a Space user to authenticate in Slack so that it can provide link previews on their behalf.

1. Create the unfurls.kt file and add the following code to it, replacing your-ngrok-hostname in NavigateUrlAction with the hostname of your ngrok tunnel:

   package com.linkpreviews import io.ktor.http.* import space.jetbrains.api.runtime.helpers.unfurl import space.jetbrains.api.runtime.resources.applications import space.jetbrains.api.runtime.types.* import java.util.concurrent.ConcurrentHashMap // Data class that stores a pair of Slack API tokens. // We use [[[Slack token rotation|https://api.slack.com/authentication/rotation]]]. // The first token is used to fetch data from Slack. // The second token is used to refresh the first token. data class SlackUserTokens(val accessToken: String, val refreshToken: String) // Each Space user (link author) has an associated pair of Slack tokens. // In real applications, store this map in a persistent storage. val slackUserTokens = ConcurrentHashMap<String, SlackUserTokens>() suspend fun provideUnfurlContent(item: ApplicationUnfurlQueueItem, spaceUserId: String) { // Check whether the link that we have conforms to the format of the Slack message link val url = Url(item.target) val parts = url.encodedPath.split('/').dropWhile { it != "archives" }.drop(1) val channelId = parts.firstOrNull() val messageId = parts.drop(1).firstOrNull() if (channelId == null || messageId == null) return // If the Slack tokens pair is missing for a given Space user (link author), // request this user to authenticate in Slack. var tokens = slackUserTokens[spaceUserId] ?: run { requestAuthentication(item, spaceUserId) return } // TODO - fetch data from Slack and provide the link preview } // The authentication request is a pseudo link preview that appears in the Space UI // in the same position as a regular link preview but is visible only to the message author. suspend fun requestAuthentication(item: ApplicationUnfurlQueueItem, spaceUserId: String) { spaceClient.applications.unfurls.queue.requestExternalSystemAuthentication( item.id, // use the message constructor to create the message unfurl { section { text("Authenticate in Slack to get link previews in Space") controls { button( "Authenticate", NavigateUrlAction( // replace it with your ngrok tunnel public address "https://<your-ngrok-hostname>/slack/oauth?user=$spaceUserId", withBackUrl = true, openInNewTab = false ) ) } } } ) }

   Note that currently, external system authentication requests are supported only in Space chats. Learn more.

Step 8. Process the authentication flow in Slack

When a user clicks the Authenticate button in the request, the browser gets redirected to the endpoint of our application. This is achieved by the NavigateUrlAction attached to the button. Let's now look at how the application handles this redirect.

1. Add these lines to the end of the Routes.kt file, outside of the Routing.api() call:

   // key - generated nonce, value - Space user id val oAuthSessions = ConcurrentHashMap<String, OAuthSession>() data class OAuthSession(val spaceUserId: String, val backUrl: String)

2. In the Routes.kt file, add one more route into the lambda passed to the api function. This code will provide a redirect URL for Slack:

   // the endpoint expects two query string parameters: 'user' and 'backUrl' get("slack/oauth") { // 'user' is a Space user ID specified in 'NavigateUrlAction' val spaceUserId = call.parameters.get("user") ?: run { call.respond(HttpStatusCode.BadRequest, "user parameter expected") return@get } // 'backUrl' points to a page in Space where the user was before being redirected. // The app will return the user back to this page after authentication. // Space sends 'backUrl' as we set 'withBackUrl=true' in 'NavigateUrlAction'. val backUrl = call.parameters.get("backUrl") ?: run { call.respond(HttpStatusCode.BadRequest, "backUrl parameter expected") return@get } val flowId = generateNonce() oAuthSessions[flowId] = OAuthSession(spaceUserId, backUrl) val authUrl = with(URLBuilder("https://${SlackWorkspace.domain}/oauth/v2/authorize")) { parameters.apply { append("client_id", SlackWorkspace.clientId) append("user_scope", slackPermissionScopes.joinToString(",")) append("state", flowId) // replace it with your ngrok tunnel public address append("redirect_uri", "https://<your-ngrok-hostname>/slack/oauth/callback") } build() } call.respondRedirect(authUrl.toString()) }

   The basic flow of this endpoint is simple. We generate the nonce, remember it in our in-memory map together with the Space user id and the back url, and then redirect the user further to the Slack endpoint responsible for performing the OAuth flow. While redirecting, we provide the client id of our application in Slack, the generated nonce (we will verify it later upon receiving the callback from Slack) and the list of Slack permissions needed for the application to fetch data for link previews. After that, we just need to await for Slack calling back to our application using the redirect url that we've specified.

   Here you can find more details about the OAuth flow in Slack. We're going to call Slack API on behalf of the user, so we obtain a user token and use the user_scope parameter for passing the list of permission scopes.

   The state parameter serves two purposes here:

   • Persisting additional context about the authentication request (a Space user id and a back url) to use it later after the flow is completed.

   • Protecting from the CSRF attacks by verifying that the user request to the application's callback url comes from a legitimate Slack authentication flow and not from some malicious party. You can find more information about the security aspect of the state parameter here and here.

3. To complete the OAuth flow, we need to handle the callback from Slack with the authorization code. Let's add a route for this into the lambda passed to the api function. Add the following code to the Routes.kt file:

   get("slack/oauth/callback") { val flowId = call.parameters.get("state") ?: run { call.respond(HttpStatusCode.BadRequest, "state parameter expected") return@get } val session = oAuthSessions.get(flowId) ?: run { call.respond(HttpStatusCode.Unauthorized, "invalid auth session") return@get } val code = call.parameters.get("code") ?: run { call.respond(HttpStatusCode.BadRequest, "code parameter expected") return@get } val slackUserToken = requestOAuthToken(code) if (slackUserToken == null) { call.respond(HttpStatusCode.Unauthorized, "could not fetch OAuth token from Slack") return@get } slackUserTokens[session.spaceUserId] = slackUserToken spaceClient.applications.unfurls.queue.clearExternalSystemAuthenticationRequests( ProfileIdentifier.Id(session.spaceUserId) ) call.respondRedirect(session.backUrl) }

4. Let's also implement the requestOAuthToken function used in this handler. Add it to the end of the Routes.kt file:

   fun requestOAuthToken(code: String): SlackUserTokens? { val response = slackApiClient.methods() .oauthV2Access { it.clientId(SlackWorkspace.clientId).clientSecret(SlackWorkspace.clientSecret).code(code) } return response?.takeIf { it.isOk } ?.authedUser?.takeIf { it.accessToken != null && it.refreshToken != null } ?.let { SlackUserTokens(it.accessToken, it.refreshToken) } }

   On receiving a callback from Slack, we extract the session id and the authorization code from the request parameters. Then, the requestOAuthToken function uses the Slack API to exchange the authorization code for a pair of Slack tokens. We expect to get both a short-lived and a long-lived access tokens, as we have specified the token_rotation_enabled: true flag in the Slack application settings in step 4.

Step 9. Provide preview content on behalf of a user

Let's use the Slack tokens to fetch preview content. In step 7, we made a TODO comment in the provideUnfurlContent function. Now, the time has come to fill in this gap.

1. First, let's implement several simple helpers for fetching data from Slack. Open the Unfurls.kt file, and add the following imports:

   import com.slack.api.methods.SlackApiException import com.slack.api.model.Message

2. Add the code below to the file. All these methods use the Slack API to fetch data and transform response models. With these methods, we can make the provideUnfurlContent function to actually provide link previews.

   fun fetchMessage(channelId: String, messageId: String, threadTs: String?, accessToken: String): Message? { return if (threadTs != null) { // https://api.slack.com/methods/conversations.replies slackApiClient.methods(accessToken) .conversationsReplies { it.channel(channelId).latest(threadTs).ts(messageIdToTs(messageId)).inclusive(true).limit(1) } ?.messages?.singleOrNull() } else { // https://api.slack.com/methods/conversations.history slackApiClient.methods(accessToken) .conversationsHistory { it.channel(channelId).latest(messageIdToTs(messageId)).inclusive(true).limit(1) } ?.messages?.singleOrNull() } } // https://api.slack.com/methods/users.info fun fetchAuthorName(accessToken: String, slackUserId: String): String { return slackApiClient.methods(accessToken).usersInfo { it.user(slackUserId) }?.user?.profile?.let { it.displayName?.takeUnless { it.isBlank() } ?: it.realName?.takeUnless { it.isBlank() } } ?: slackUserId } // https://api.slack.com/methods/conversations.info fun fetchChannelName(accessToken: String, channelId: String): String { return slackApiClient.methods(accessToken) .conversationsInfo { it.channel(channelId) }?.channel?.name?.let { "[#$it](https://${SlackWorkspace.domain}/archives/$channelId)" } ?: channelId } // Converts Slack message id as it's present in the message url to // the timestamp value for usage in Slack API requests private fun messageIdToTs(messageId: String) = messageId.removePrefix("p").let { it.dropLast(6) + "." + it.drop(it.length - 6) }

3. Add the code below to the provideUnfurlContent function:

   suspend fun provideUnfurlContent(item: ApplicationUnfurlQueueItem, spaceUserId: String) { // Extract a message, channel, and optional thread IDs from the URL. val url = Url(item.target) val parts = url.encodedPath.split('/').dropWhile { it != "archives" }.drop(1) val channelId = parts.firstOrNull() val messageId = parts.drop(1).firstOrNull() if (channelId == null || messageId == null) return var tokens = slackUserTokens[spaceUserId] ?: run { requestAuthentication(item, spaceUserId) return } val threadTs = url.parameters["thread_ts"] // Fetch the message itself, author and channel names from Slack val message = try { fetchMessage(channelId, messageId, threadTs, tokens.accessToken) // Catch Slack API exceptions, specifically token expiration. } catch (ex: SlackApiException) { // If token is expired, refresh the short-lived token // by performing a call to https://api.slack.com/methods/oauth.v2.access if (ex.error.error == "token_expired") { val response = slackApiClient.methods().oauthV2Access { it .clientId(SlackWorkspace.clientId) .clientSecret(SlackWorkspace.clientSecret) .grantType("refresh_token") .refreshToken(tokens.refreshToken) } val accessToken = response.accessToken ?: response.authedUser?.accessToken val newRefreshToken = response.refreshToken ?: tokens.refreshToken if (accessToken != null) { tokens = SlackUserTokens(accessToken, newRefreshToken) // Save new tokens to our in-memory storage. slackUserTokens[spaceUserId] = SlackUserTokens(accessToken, newRefreshToken) // Fetch the message one more time, but this time with updated tokens. fetchMessage(channelId, messageId, threadTs, tokens.accessToken) } else null } else null } if (message == null) return val channelLink = if (threadTs != null) { // Converting message timestamp value to id for the message link // (an operation opposite to `messageIdToTs`) val parentMessageId = "p" + threadTs.filterNot { it == '.' } "https://${SlackWorkspace.domain}/archives/$channelId/$parentMessageId" } else { fetchChannelName(tokens.accessToken, channelId) } val authorName = fetchAuthorName(tokens.accessToken, message.user) // Build link preview with message constructor DSL val content: ApplicationUnfurlContent.Message = unfurl { outline( MessageOutline( ApiIcon("slack"), "*$authorName* in $channelLink" ) ) section { text(message.text) text("[View message](${item.target})") } } // Post the preview to Slack including queue item ID spaceClient.applications.unfurls.queue.postUnfurlsContent( listOf(ApplicationUnfurl(item.id, content)) ) }

Step 10. Run the application

We're almost ready to run our application and see the link previews in action. But don't forget that our application being run for the first time, requires initialization with the init command. During initialization, the application will request all necessary permissions from the Space organization. The organization administrator must approve the request.

1. Start the application by running the Gradle run task: either run ./gradlew run in the command prompt or run the main function in Application.kt using the icon in the gutter.

2. Open Space, click the Search icon in the left sidebar and enter the application name. After Space finds the application, click it to go to the application chat channel.

3. In the chat channel, type init and send this message to the application. The application must respond with ok.

4. On the main menu, click Extensions, then Installed. Click the application to go to its settings page.

5. On the Authorization tab, there's one pending Provide external unfurls as attachments permission. Approve it.

6. Switch to the Unfurls tab. Here, the application is requesting permission to listen to Slack links. Approve it as well.

Step 11. Try the application in action!

1. Go to any public or private channel in Slack and find there any message. Copy the link to this message.

2. Go to Space, open any chat channel, paste the copied link and send the message.

3. After a while, you'll see the authentication request under the chat message. Click Authenticate to go to Slack.

4. Sign in to the Slack workspace. Slack will redirect you back to your last page in Space.

5. Send one more message with a Slack message link to the Space chat. After a while, it will get a preview displaying the contents of the original Slack message.

Note that if you want to remove a link preview from a chat messages, hover over the preview and click the cross icon.

Takeaway

Let's have a bird's eye view of what it takes to implement application-provided unfurling in Space:

• Space application requests the Unfurl.App.ProvideAttachment permission scope. It also requests the list of unfurl domains and patterns that it is going to handle. We haven't talked about unfurl patterns here, but they are basically the same as external links. Permission scopes, domains, and patterns must be approved by the Space organization administrator.

• When Space encounters a link pointing to the unfurling domain of an application or a word that matches the unfurling pattern, it creates an item in the unfurling queue and sends a notification to the application.

• The application can poll this queue using the Space API. On receiving a notification, the application is expected to query items from the queue. The item lives in this queue only 30 minutes.

• Each queue item has etag assigned to it – an unsigned 64-bit monotonically-growing integer. The application is expected to track the last processed etag by storing it in a persistent storage and provide it to the Space API method when querying the next item batch from the queue.

• The application can handle a queue item in one of two possible ways – by either providing the link preview right away or by requesting a user to authenticate in the external system. Tip: we recommend that the application has its own queue for new items and handles them in the background.

• Authenticating in the external system on behalf of the application itself is simpler but less secure compared to authenticating on behalf of the user who posted the link.

• If the application uses user tokens for the external system, it should handle the authentication flow and securely (that's important!) store the obtained user tokens. We recommend using the flows with refresh tokens.

• To provide link preview content, the application must call the Space API method that takes the list of items and pairs the original item id with the unfurled content (so you can do batch processing). There are several possible formats for providing unfurled content.