Maximizing efficiency is about knowing how the data science puzzles fit together and then executing them. Today there is a mass of new software packages and repositories arriving on the scene that has made the data science process more interactive, nuanced, and user-driven than ever before. For evidence of this, just check the homepage on any given day. In the face of this new wave of choices, it is important to understand the basic structure of development pipelines. Towards Data Science Data Scientists have become newly minted developers in their own right. In becoming developers, it is useful to understand development principles that software engineers use to iteratively test, construct and shape their deployed code. In this article, we’ll talk about some often-misunderstood development principles that will guide you to developing more resilient, production-ready development pipelines using CI/CD tools. Then, we’ll make it concrete with a tutorial about how to set up your own pipeline using . Buddy A Representation of the Modern Data Science Workflow. Created with Notability for iPad. The Development Pipeline Understanding the components to development is the first step to understanding which pieces go where, and how things fit together. Each of these elements serves as crucial building blocks towards the coveted . In this case, “end-to-end” is jargon for “you make a code-level change, and the end-user experiences the effect”. end-to-end pipeline In a nutshell, you as the data scientist would use the development pipeline to push changes from your local machine to a version control tool, and have these changes be reflected in the cloud deployment service for your end users. Next, we’ll break down an example development pipeline step by step. Let’s say you have created an awesome new Data Science dashboard that uses data to answer a burning question within your organization. In this example, we’ll use a sample project from Streamlit, that visualizes Uber rides in New York City. In this case, this is a project folder that lives on your local machine that contains images, python scripts, datasets, and more. Dashboard: The next step is to use a version control tool like Github to host your project. This helps you and your team track code changes and edits over time, write documentation and enable collaboration with the wider community. Many data scientists are already intimately familiar with GitHub as a place to host and showcase their projects. Version Control: Finally, you use a Cloud Server Service to deploy your project. In this example, we’ll use AWS to deploy the original dashboard visualization and share it with your end-users. Cloud Deployment: At this point, many data scientists stop and call it a good day at the office. However, we’re still missing a crucial piece of the puzzle. Uploading code to GitHub and setting up an AWS deployment is great, but if there are changes or upgrades to that codebase, the AWS deployment . Instead, each version for deployment will have to be manually updated. While this is suboptimal in terms of effort and time, there is also a possibility that your flashy new update might the basic functionality of your original dashboard. This mistake is compounded when working with a team of data scientists to create a product. will not automatically reflect them break CI/CD with Buddy To patch this missing puzzle piece, we introduce the concept of — abbreviated as CI/CD. This tool bridges the gap between development and operation activities through automation. It helps you test and integrates your new changes into existing the body of work. is an excellent option for this tool when setting up your deployment pipeline. Continuous Integration / Continuous Deployment Buddy Works You might be wondering — how is this going to stop my development pipeline from breaking? Let’s explore the value of using Buddy. This CI/CD tool is actually a process, that involves adding , and to connect your GitHub repository to the cloud configuration. testing automation, delivery benchmarks Buddy functions as a Swiss Army knife when it comes to deployment operations. Let's examine each element in turn: After installing the relevant environment, you can run unit-testing through a full suite of tools, including Cypress, Split Testing, and tests written natively in Python, PHP, Django, and more. Each time you run the pipeline, all your tests will be re-run to ensure production-quality code is being deployed. Rigorous Testing: Once your GitHub repository is connected, the pipeline will run according to your setup instructions. Just by pushing to the master-branch or using a recurrent trigger, the pipeline will automatically start, run tests and update your cloud instance if there are no errors. Automated Deployment: The CI/CD workflow allows you to connect notification settings across a wide array of options, including SMS/email, Slack — as well as mainstream performance and app monitoring tools like Datadog and Sentry. If there are any errors in integrating changes, you can be notified and go straight in to fix the issue. Reliable Workflow: Now that we have established the premise of CI/CD and its uses, let’s dive right into a first look of Buddy’s platform and how you can get a basic pipeline off the ground. Tutorial: The CI/CD Pipeline 1. Setting Up the Pipeline First off, head over to and . It is recommended to use your version control login here, as it will save you the step of connecting your repositories. Either way, connecting your version control software will allow Buddy to connect to any of those repositories. Buddy make an account Buddy conveniently syncs with all of your GitHub repositories, public and private. We’ll use the repository for the purposes of this tutorial, which is an interactive dashboard built with Streamlit. After forking the repository on Github, it will show up in our repo list within Buddy. Clicking on the name will lead us to the next screen. demo-uber-nyc-pickups Buddy scans your repo meta-data to recommend a relevant environment setup. Here, Buddy has already detected that the repository’s contents contain a Python app and shows us more options for setting up the relevant Python environment. At this step, we also have to select how the pipeline should trigger. I went with the ‘on-push’ trigger to master-branch, so all my latest and greatest changes will be acted on. 2. Adding Actions to the Pipeline After naming the pipeline, we can choose what action will trigger the pipeline. Since we care about deploying new changes to the AWS instance, we can set it to run the pipeline every time a new push is made to the master branch. Alternatively, we can set it to only trigger manually, or even on a timed basis (e.g. every day at 5pm, every Friday, etc). This is the home for pipeline building. Add new actions to your pipeline either by searching or clicking on icons. As mentioned, Buddy has detected that our app is written in Python, so we’ll click on that icon first. Here’s where we can configure the environment, choose the relevant Python version (in this case, it’s ). A quick look in the README.md of the project tells us the BASH lines needed to get the app up and running: python3.7 pip install --upgrade streamlit pip install -r requirements.txt The first line ensures that we are running the latest version of streamlit, and the contains the remaining dependencies we need to be able to run our app. requirements.txt At the bottom, we can also notice the — this allows for a way of helpfully identifying behavior in case of errors at any step in the pipeline. We can either solider on (not recommended for obvious reasons), or stop the pipeline where it broke and send a notification that something went wrong, or try running different commands. Identifying something has broken is perhaps the most frustrating part of fixing a broken process. Proactively setting error-handling behavior and notifications as a priority will help keep frustrations at a minimum going forward, when some element inevitably breaks. Exit Code Handling section where The build commands allow you to write any BASH or SH scripts you need to get the environment set up right. 3. Building the environment Next, we’ll try running the very basic pipeline so far, and see whether it works. Click “Run Pipeline” in the top left, and sit back as Buddy integrates the latest commit on the master-branch, prepares the environment, and executes the BASH setup. On subsequent runs, the cache will continue to only update changes, and this process will run faster over time. Raw Logs allow you to see exactly what happened during execution, and the timer is a convenient method for estimating runtimes. Awesome! The build is complete and without errors. If you are following along with the tutorial to this stage and faced errors, check that the Python version is exactly , because that is required for this particular app’s dependencies. python3.7 4. Adding testing functionality Testing is a central element in software development, but it is unfortunately not prioritized or taught in most data science curriculums. “Unit tests give you the confidence that your code does what you think it does” Adding unit tests can be simple as adding python files in the same repository. In order to run these tests, we’ll return to , and add in our new line to run the tests here. Step 3: Building the environment Adding in “python run unittest.py” will run your files when the environment is built. If all tests pass, the pipeline will continue. When the tests have been implemented, this is where we would expect to see the results. In this case, error handling setup becomes particularly important, as Buddy can share notifications if some tests fail. If all tests pass, the compiler will end with “Build finished successfully”. 5. Adding notification functionality Adding in notifications is critical to ensuring we know where breaks in the pipeline occur, or which tests have failed. From the pipeline overview, click on the “ section, where we can decide what actions will run if there is an error anywhere in the pipeline. For our purposes, it will be sufficient to set this up using environmental variables that will indicate which execution or test broke the pipeline execution. Actions Run On Failure” gives us the name of the pipeline that is broken $BUDDY_PIPELINE_NAME gives us the unique identifier of the instance of the pipeline that created an error, including the $BUDDY_EXECUTION_ID will give an extensive overview of the logs of what went wrong, which is convenient because it helps in diagnosing any issues that pop up. It may even help solve the issue by just glancing in the email, fixing the code, and making a new commit to patch the issue — without even needing to visit the CI/CD tool at all. $BUDDY_FAILED_ACTION_LOGS An extensive array of environmental variables are , and more can be developed with ease. available here Below is a sample message sent to my email, informing me that there was something amiss with the built environment, and sharing the logs here. With the use of additional environmental variables, you can make this output incredibly specific to allow you to hone in on the error. Here, the issue is clearly that my build environment is deprecated, and thus I need to choose a new python version that is being maintained. In this case, that’s . python 3.7 6. Add More Actions + Iterate This is the last step we will take in setting up the pipeline. By connecting this pipeline to a , we will arrive at an end-to-end pipeline, as per the overall goal. free-tier AWS EC2 machine In order to do this, select the action, and make the connection between Buddy and the public IPv4 address of the EC2 machine. SFTP Using the Pipeline Filesystem is super important here because it makes use of the tested files. Here, I’ve entered my Hostname & Port, and Login information, as well as used my Private SSH key to actually give Buddy access to the EC2 machine. There are two caveats to mention here: The EC2 instance must be running at the time you run your pipeline, to ensure that the Host is live, and can be found. This will not happen if the EC2 instance is stopped or interrupted. It is useful to have tmux running streamlit on your EC2 configuration, since the Buddy pipeline will update the files, and allow the visualization to reflect those changes without any manual input. Having completed this final step, we can run the pipeline by simply making a change at the code-level, and then committing the change to the master branch. git commit "app.py" -m "Buddy cicd test" git push We correctly set up the environment, sent across changes from the local machine to GitHub. The code was then executed, ran unit tests, and uploaded to the EC2 machine, where the changes were reflected in our visualization. Hooray! Let’s take a look at the final product: You can also visit Streamlit’s version of this app . here This is the front-end visualization, powered by Streamlit. To review, we’ve taken python code and committed it to a versioning tool (in this case, Github). This repo is then linked to a CI/CD tool (Buddy), which syncs, tests, and integrates our commits to the overall build, hosted on an AWS EC2 machine. Review In conclusion, every time we make a new commit to Github, this will trigger a Buddy pipeline execution which will: Build the environment Run any tests Upload our changes to AWS using . SFTP In the event of any errors or snags along with the execution, we’ll receive an email highlighting exactly what went wrong. With this level of detail and refinement, has elevated our deployment of data science platforms and made it easier than ever to maintain user-driven products. Buddy’s CI/CD tool Happy coding. . I do use Buddy CI/CD in my projects, and have leveraged their technology to develop and deliver end-to-end pipelines to a number of data engineering clients. Full disclosure, this is a sponsored article by Buddy About The Author 🤠 is a data scientist and co-founder of . Say hello on . Saif Bhatti Go Fish Analytics Twitter Other pieces of data science creativity: Building a system to . predict the impact of natural threats on insurance risk assessments for Lloyds of London Developing new optimisation models for EV and biofuels fuel allocations across the United States Predicting . Epilogue outcomes of early-stage startups Epilogue It’s worth noting that the specifics of the EC2 setup are not in the purview of this article, but some helpful advice and content on setting up Streamlit applications on AWS is available below. Also published on . Medium

Sentry

Slack

Super

Twitter

Uber

Sign up for Command-T now!

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Top 6 CI/CD Practices for End-to-End Development Pipelines

Too Long; Didn't Read

People Mentioned

Companies Mentioned

Saif Bhatti

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Top 6 CI/CD Practices for End-to-End Development Pipelines

Too Long; Didn't Read

People Mentioned

Companies Mentioned

Saif Bhatti

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES