When you use , you can only save your last 10K messages. For social Slack instances, it may be impractical to upgrade to a paid plan to retain these messages. For example, for an open-source project like Airbyte where we interact with our community through a public Slack instance, the cost of paying for a seat for every Slack member is prohibitive. Slack's free licensing tier However, searching through old messages can be really helpful. Losing that history feels like some advanced form of memory loss. What was that joke about Java 8 Streams? This contributor question sounds familiar—haven't we seen it before? But you just can't remember! This tutorial will show you how you can, for free, use Airbyte to save these messages (even after Slack removes access to them). It will also provide you with a convenient way to search through them. Specifically, we will export messages from your Slack instance into an open-source search engine called . We will be focusing on getting this setup running from your local workstation. We also cover how to set up a version of this pipeline that's more "productionized". MeiliSearch We want to make this process easy, so while we will link to some external documentation for further exploration, we will also provide all the instructions you need here to get this up and running. 1. Set Up MeiliSearch First, let's get MeiliSearch running on our workstation. MeiliSearch has extensive docs for . For this tutorial, however, we will give you all the instructions you need to set up MeiliSearch using Docker. getting started docker run -it --rm \
  -p : \
  -v $(pwd)/data.ms: eilisearch 7700 7700 /data.ms \
  getmeili/m That's it! MeiliSearch stores data in , so if you prefer to store it somewhere else, just adjust this path. $(pwd)/data.ms 2. How To Replicate Your Slack Messages to MeiliSearch a. Set Up Airbyte Make sure you have Docker and Docker Compose installed. If you haven’t set Docker up, follow the to set it up on your machine. Then, run the following commands: instructions here git clone https: cd airbyte
docker-compose up //github.com/airbytehq/airbyte.git If you run into any problems, feel free to check out our more extensive for more help. getting started Once you see an Airbyte banner, the UI is ready to go at . Once you have set your user preferences, you will be brought to a page that asks you to set up a source. In the next step, we'll go over how to do that. http://localhost:8000/ b. Set Up Airbyte’s Slack Source Connector In the Airbyte UI, select Slack from the dropdown. We provide step-by-step instructions for setting up the Slack source in Airbyte . These will walk you through how to complete the form on this page. here By the end of these instructions, you should have created a Slack source in the Airbyte UI. For now, just add your Slack app to a single public channel (you can add it to more channels later). Only messages from that channel will be replicated. The Airbyte app will now prompt you to set up a destination. Next, we will walk through how to set up MeiliSearch. c. Set Up Airbyte’s MeiliSearch Destination Connector Head back to the Airbyte UI. It should still be prompting you to set up a destination. Select "MeiliSearch" from the dropdown. For the host field, set: . The can be left blank. http://localhost:7700 api_key d. Set Up the Replication On the next page, you will be asked to select which streams of data you'd like to replicate. We recommend unchecking "files" and "remote files" since you won't really be able to search them easily in this search engine. For frequency, we recommend every 24 hours. 3. Search MeiliSearch After the connection has been saved, Airbyte should start replicating the data immediately. When it completes you should see the following: When the sync is done, you can sanity check that this is all working by making a search request to MeiliSearch. Replication can take several minutes depending on the size of your Slack instance. curl --data 'http://localhost:7700/indexes/messages/search' '{ "q": "<search-term>" }' For example, I have the following message in one of the messages that I replicated: "welcome to airbyte". curl --data # => { :[{ : , : , : , : , : , : , : , : , :[{ : }], :[], : }], : , : , : , : , : , : } 'http://localhost:7700/indexes/messages/search' '{ "q": "welcome to" }' "hits" "_ab_pk" "7ff9a858_6959_45e7_ad6b_16f9e0e91098" "channel_id" "C01M2UUP87P" "client_msg_id" "77022f01-3846-4b9d-a6d3-120a26b2c2ac" "type" "message" "text" "welcome to airbyte." "user" "U01AS8LGX41" "ts" "2021-02-05T17:26:01.000000Z" "team" "T01AB4DDR2N" "blocks" "type" "rich_text" "file_ids" "thread_ts" "1612545961.000800" "offset" 0 "limit" 20 "nbHits" 2 "exhaustiveNbHits" false "processingTimeMs" 21 "query" "test-72" 4. Search via a UI Making curl requests to search your Slack History is a little clunky, so we have modified the example UI that MeiliSearch provides in to search through the Slack results. their docs Download (or copy and paste) this to your workstation. Then, open it using a browser. You should now be able to write search terms in the search bar and get results instantly! html file 5. "Productionizing" Saving Slack History You can find instructions for how to host Airbyte on various cloud platforms . here Documentation on how to host MeiliSearch on cloud platforms can be found . here If you want to use the UI mentioned in the section above, we recommend statically hosting it on S3, GCS, or equivalent. Also published on https://docs.airbyte.io/tutorials/slack-history

Instantly

Slack

Co-founder at Airbyte.io, the new standard for open-source data integration

2021 - HackerNoon Contributor of the Year - STARTUPS TOP STORY

2022 - HackerNoon Contributor of the Year - Open Source

Open-source data integration

Nominated for 2022 - HackerNoon Contributor of the Year - Open Source

Too Long; Didn't Read

Join to Learn About Storytelling for Startups

HackerNoon Mobile

Save and Search Through Your Slack Channel History on a Free Slack Plan

Too Long; Didn't Read

Companies Mentioned

@jean-lafleur

RELATED STORIES