Connecting the dots in style: Build your own Dropbox Sync in 10 minutes!

Integration, or “connecting the dots”, is something that is quite difficult to avoid in the modern era of highly globalized business domains. Fortunately, integration, or “enterprise integration” in more “enterprise-y” terms, is no longer meant to be something that makes your hair stand, thanks to advanced yet user-friendly enterprise integration frameworks such as . Project-X Today, we shall extend our helping hand to Jane, a nice Public Relations officer of the (never heard the name? yup, neither have I :)) in setting up a portion of her latest customer feedback aggregation mechanism. No worries, though, since I will be helping and guiding you all the way to the end! HappiShoppin supermarket service The PR Department of HappiShoppin supermarket service has opened up new channels for receiving customer feedback. In addition to the former, conventional paperback feedback drop-ins, they now accept electronic feedback via their website as well as via a public folder (in addition to social media, Google Drive, Forms etc). Jane, who is heading the Dropbox-driven feedback initiative, would like to set up an automated system to sync any newly added Dropbox feedback to her computer so that she can check them offline whenever it is convenient for her, rather than having to keep an eye on the Dropbox folder all the time. Dropbox Google Jane has decided to compose a simple “Dropbox sync” integration flow that would periodically sync new content from the feedback accumulation Dropbox folder, to a local folder on her computer. On HappiShoppin’s shared Dropbox account, is the folder where customers can place feedback documents, and Jane hopes to sync the new arrivals into on her computer. /Feedback/Inbox /home/jane/dropbox-feedback Jane has estimated that it is sufficient to sync content once a day, as the company receives only a limited number of feedback over a given day; however, during the coming Christmas season, the company is expecting a spike in customer purchases, which would probably mean an accompanied increase in feedback submissions as well. For easier tracking and maintenance, she wants the feedback files to be organized into daily subfolders. In order to avoid repetitively syncing the same feedback file, Jane has to ensure that the successfully synced files are removed from the inbox, which she hopes to address by moving them to a different Dropbox folder: . /Feedback/Synced Dropbox Sync: Solution Overview Now, before we begin, a bit about what Project-X is and what we are about to do with it: Project-X is a , which one could also call an (which is also valid for the scenario we are about to tackle). messaging engine enterprise service bus Project-X ingests (or ) from , subjects them to various transformations via , and emits them to other systems via . For a single message, any number of such transformations and emissions can happen, in any order. events messages ingress connectors processing elements egress connectors The message lifecycle described above, is represented as an . It is somewhat similar to a conveyor belt in a production line, although it can be much more flexible with stuff like cloning, conditional branching, looping and try-catch flows. integration flow A set of integration projects make up an , which is the basic deployment unit when it comes to Project-X runtimes such as . integration project UltraESB-X So, in our case, we should: create a new integration project create an integration flow inside the project, to represent Jane’s scenario add the necessary connectors and processors, and configure and wire them together test the flow to see if what we assembled is actually capable of doing what Jane is expecting build the project into a deployable artifact, ready to be deployed in UltraESB-X While the above may sound like quite a bit of work, we already have a cool IDE that can do most of the work for us. With UltraStudio on your side, all you have to do is to the required connectors and processing elements, and everything else will be magically done for you. You can even try out your brand-new solution right there, inside the IDE, and trace your events or messages real-time as they pass through your integration flow. UltraStudio drag, drop and connect So, before we begin, let’s (unless you already have it, of course!). get UltraStudio installed on your system Once you are ready, using option on the menu bar and selecting . While creating the project, select the following components on the respective wizard pages (don’t worry, in a moment we’ll actually get to know what they actually are): create a new Ultra Project File → New → Project… Empty Ultra Project and on the Connectors page Timer Task Connector Dropbox Connector and processor on the Processors page JSON Processor Flow Control New Empty Ultra Project Wizard: Connector Selection If you were impatient and had already created a project, you could always add the above components later on via the menu option . Tools → Ultra Studio → Component Registry Now we can start by creating a new integration flow , by opening the side pane and right-clicking the directory. dropbox-sync-flow Project src/main/conf Creating a New Integration Flow Again, a few tips on using the graphical flow UI (in case you’re wondering where on earth it is) before you begin: Inside, an integration flow is a XML ( ) configuration, which UltraStudio can alternatively represent as a composable diagram for your convenience. Spring You can switch between the XML and graphical views using the two small tabs that would appear at the bottom of an integration flow file while it is opened in the IDE. (These tabs might be missing at certain times, e.g. when the IDE is performing or ; at such times, patience is a virtue!) indexing Maven dependency resolution The graphical view contains a side palette with all the components (connectors and processors) that have currently been added to your project (at creation or through the Component Registry). You can browse them by clicking on the collapsible labels on the palette, and add them to the flow by simply dragging-and-dropping them into the canvas. In order to mimic the message flow, components should be connected together using lines drawn between their ports (small dots of different colors that appear around the component’s icon). You will get the hang of it, when you have had a look at some of the , or at the image of the flow that would be developing (appearing later in this article). existing integration flows When a component requires configuration parameters, a configuration pane gets automatically opened as soon as you drop an element into the canvas (you can also open it by clicking on the component later on). If the labels or descriptions on the configuration pane are not clear enough, just switch to the tab and click on the “Read more” URL to visit the complete documentation of the element (on your favourite web browser). Also, make sure that you click the Save button (at the bottom or on the side pane) once you have made any changes. Documentation Start the flow with a . This is a connector used to trigger a periodic event (similar to a clock tick) for a time-driven message flow. Let’s configure it to trigger an event that would set the sync process in motion. For flexibility, we will use a instead of a simple periodic trigger. Timer Ingress Connector cron expression Scheduling tab: Polling CRON Expression 0/30 * * ? * * Dropbox Sync: Timer Ingress Connector Configuration Although Jane wanted to run the check only at 6 PM each day, we have set the polling time to every 30 seconds, for the sake of convenience; otherwise you’ll simply have to wait until 6 PM to see if things are working :) Next add a with a Connector operation element added to the side port. You can find the connector operations by clicking on the down arrow icon against the on the component palette, which will expand a list of available connector operations. Dropbox Egress Connector List Entities Dropbox Connector Dropbox Egress Connector A is an appendage that you can, well, append to a connector, which will perform some additional processing on the outgoing message in a connector-specific way. For example, for Dropbox we have a main connector, with a bunch of connector operations that represent different API operations that you can perform against your Dropbox account, such as managing files, searching, downloading, etc. connector operation Configure the with the shared Dropbox account credentials ( and ), and the connector operation with the . Dropbox Connector App ID Access Token Path /Feedback/Inbox , Basic tab: Dropbox Connector {client ID for your Dropbox app; visit to create a new app} Client ID https://www.dropbox.com/developers/apps/create {access token for your Dropbox account, under the above app; follow to obtain an access token for personal use against your own app} Access Token https://blogs.dropbox.com/developers/2014/05/generate-an-access-token-for-your-own-account/ Dropbox Sync: Connector Configuration , Basic tab: List Entities Path /Feedback/Inbox Dropbox Sync Flow: Progress So Far The above contraption will return a , containing all files that are currently inside , as a wrapped payload: List Folder response /Feedback/Inbox JSON {"entries": [{".tag": "file","name": ,"id": "id:12345_67_890ABCDEFGHIJ",...}, {".tag": "file","name": ,"id": "id:JIHGF_ED_CBA9876543210",...}],...} "johndoe.docx" "janedoe.txt" Ah, now there’s the info that we have been looking for; sitting there in boldface. Now we need to somehow pull them out. Next add a processor to extract out the file paths list from the above JSON response, using a pattern: . This will store the resulting file name list in a scope variable named , for further processing. A scope variable is a kind of temporary storage where you can retain simple values for referring later in the flow. JSON Path Extractor JSON Path $.entries[*].name files Variable Name files _JSON Path_ $.entries[*].name Then add a to iterate over the previously mentioned scope variable, so that we can process each of the observed files separately. The next processing operations will each take place within a single iteration of the loop. ForEach Loop _Collection Variable Name_ files Collection Type COLLECTION _Iterating Variable Name_ file Now add a new Dropbox Connector (configured with your app and account credentials as before), along with a, connector operation, to download the file ( ) corresponding to the current iteration from Dropbox into the local directory. Download Entity file When you are drawing outgoing connections from , note that the topmost out port is for the loop termination (exit) path, and not for the next iteration! Tip: ForEach Loop , Basic tab: Dropbox Connector Client ID {client ID for your Dropbox app} _Access Token_ {access token for your Dropbox account, under the above app} , Basic tab: Download Entity _Path_ /Feedback/Inbox/@{variable.file} Destination /home/jane/dropbox-feedback/@{current.timestamp.yyyy-MM-dd_HH-mm} Next add another Dropbox Connector (configured with your app and account credentials) with a connector operation, to move the original file to so that we would not process it again. We will set the property of the connector to 3, to make a best effort to move the file (in case we face any temporary errors, such as network failures, during the initial move). We will also enable on the connector operation to avoid any possible issues resulting from files with same name being placed at at different times (which could cause conflicts during movement). Move Entity /Feedback/Synced Retry Count Auto-Rename /Feedback/Inbox , Basic tab: Dropbox Connector Client ID {client ID for your Dropbox app} _Access Token_ {access token for your Dropbox account, under the above app} , Advanced tab: Dropbox Connector Retry Count 3 , Basic tab: Move Entity Path /Feedback/Inbox/@{variable.file} Destination /Feedback/Synced/@{variable.file} Now add a element to signify that the message flow has completed successfully. Successful Flow End Now we need to connect the processing elements together, to resemble the following final flow diagram: Dropbox Sync: Complete Flow Finally, now we are ready to test our brand new Dropbox sync flow! Before proceeding, ensure that your Dropbox account contains the and directories. /Feedback/Inbox /Feedback/Synced Create an by clicking on the menu, and selecting under the (+) button on the top left. UltraStudio run configuration Run → Edit Configurations… UltraESB-X Server Add New Configuration Now, with everything in place, select from the menu to launch your project! Run → Run configuration name If everything goes fine, after a series of blue-colored logs, you’ll see the following line at the end of the window: Run 2017-11-23T11:45:27,554 [127.0.1.1-janaka-ENVY] [main] [system-] [XEN45001I013] INFO XContainer AdroitLogic UltraStudio UltraESB-X server started successfully in 1 seconds and 650 milliseconds If you get any errors (red) or warnings (yellow) before this, you would have to click (red square) on the window to stop the project, and dig into the logs to get a clue as to what might have gone wrong. Stop Run Once you have things up and running, open your on your favourite web browser, and drop some files into the directory. Dropbox account /Feedback/Inbox After a few seconds (depending on the cron expression that you provided above), the files you dropped there will magically appear in a folder . After this, if you check the Dropbox account again, you will notice that the original files have been moved from to , as we expected. /home/jane/dropbox-feedback/ /Feedback/Inbox /Feedback/Synced Now, if you drop some more files into , they will appear under a different folder (named with the new timestamp) under . This would not be a problem for Jane, as in her case the flow will only be triggered once a day, resulting in a single directory for each day. /Feedback/Inbox /home/jane/dropbox-feedback See? That’s all! Now, all that is left is to call Jane and let her know that her Dropbox integration task is ready to go alive! Originally published at randomizd.blogspot.com on November 25, 2017.