DataPelago

datapelago

In a famous scene towards the end of Indy, the story’s protagonist (along with his cohorts and a few rivals) set out to find the Holy Grail, a fabled cup thought to grant immortality to anyone who drinks from it. When he and his fellow Grail seekers finally reach the end of their odyssey, they find themselves in a room containing many Grail-like objects, guarded by a Knight. Here they learn that their trials are not yet over. “Choose wisely,” the Knight says, “for while the true Grail will bring you life, the false Grail will take it from you.” That is to say, drinking from the real Holy Grail will indeed have the desired effects, but sipping from any other goblet will result in . Indiana Jones and the Last Crusade, having a very bad day Your company’s data plan is the Holy Grail of your organization’s data strategy. While developing this document is not necessarily as harrowing as Indy’s quest, it does entail a significant amount of strategizing, collaborating, and creative thinking on part of the stakeholders involved. The “life” the Grail delivers is akin to the benefits that teams reap from the data plan––mainly, the ability to leverage quality, complete, and accurate data. Realizing these benefits hinges on a choice––that is, developers must choose to use the most recent version of the plan as the basis for implementing data collection. Finally, Indy was able to recognize the true Grail by its simple, unadorned appearance that made it look like the “cup of a carpenter” as he remarked. A data plan is similarly simple yet powerful––a straightforward solution to a complex problem. If you push event-collection code based on an outdated data plan version to production, the work that went into creating that latest version of the data plan will not come to fruition, and marketers and PMs may make decisions based on outdated data. Worst of all, engineers will need to spend time rectifying this error, which could entail not only finding and eliminating the source, but purging it from pipelines and downstream systems as well. This is why we at mParticle built . The tool came to life thanks to the creativity of Alex Sapountzis, an engineer who likes to make life easier for other engineers, especially those who spend much of their time working in web environments. In addition to Smartype Hubs, Alex has built developer-centric , , , and a that can be used to communicate with various mParticle services. Before we dive into what Smartype Hubs are, how they work, and how you can get them set up in your projects, it’s worth revisiting the tool whose functionality they help extend and improve: Smartype. Smartype Hubs Web SDKs Media-focused data collection SDKs Linting tools Command Line Interface Smartype: A brief overview is a tool that automatically translates a data plan represented as JSON schema into usable code for iOS, Android, and Web platforms, exposes these methods via an API, and enables developers to call events within the auto-complete function in their IDE of choice, rather than implementing them manually. Using Smartype, engineers do not have to reference the details of their data model (like attribute names, data types, etc.) in order to correctly implement the data plan. This benefits developers not only by streamlining the data plan implementation process, but by avoiding time-consuming debugging that could result from these errors as well. Smartype To learn more about Smartype, check out this , or of how to use it to send user events to any third-party database or library. overview of its functionality this walkthrough Extending the audience and use cases for Smartype with Smartype Hubs Smartype Hubs are a collection of scripts and Github Actions that provide a way for engineering teams to automatically synchronize data plan versions as part of their CI/CD process. Additionally, Smartype Hubs generate custom Smartype SDKs based on this latest data plan version that can be imported into your organization’s Github projects via NPM. The latter benefit is particularly useful for engineers working on web projects, who commonly work with stacks in the JavaScript/Typescript ecosystem. Since Smartype is built on Java and Kotlin, having a tool that bridges the gap between Smartype’s Java foundation and the web-native languages with which front-end developers commonly work is beneficial. “As engineering projects evolve,” Sapountzis says, “it becomes computationally and mentally expensive to require an engineer to introduce another layer to their stack. Over the last decade, many “Full Stack” engineering roles are replacing other “backend” technologies with JavaScript/TypeScript across the stack. I thought it would be great to let them not have to complicate their stack to get the benefits of Smartype.” Getting started with Smartype Hubs To start using Smartype Hubs, you need to have access to an mParticle workspace. Once you do, within this workspace, then . Next, create the actual hub (a Github repository) by selecting the “Use this template” button at the top of . create one or more Data Plans generate an API key and secret this example Next, clone your hub locally, and edit the file at the root of this repo with the Data Plan ID and versions you would like to sync to the Hub: mp.config.json {
  "smartypeHubConfig": {
    "dataPlans": [
      {
        "dataPlanId": "example_data_plan",
        "versions": [1, 2, 3]
      },
      {
        "dataPlanId": "another_example_data_plan",
        "versions": [1]
      }
    ],

    // This is the directory where data plans will be created
    // The directory must exist in your file system before you run the process
    "dataplanOutputDirectory": "dataplans"
  }
} Now you’ll need to tell your hub which mParticle workspace/client ID and client secret to look for when fetching your data plan by adding some encrypted secrets to your Hub. To do this, click Settings => Secrets => ”New repository secret”, and add the following: MP_WORKSPACE_ID: The integer ID of the workspace that contains your Data Plan(s)
MP_CLIENT_ID: A client ID with access to the Data Planning API
MP_CLIENT_SECRET: A matching client secret with access to the Data Planning API Now that your Smartype Hub knows where to find your data plan, it’s time for the magic to happen. Smartype Hubs uses Github Actions to fetch the specified Data Plan and generate your Smartype SDKs. Within the “Actions” tab of your Hub’s repo, click “Generate Data Plans,” select the “Run Workflow” dropdown, choose your desired branch, the click the “Run Workflow” button: Once the job is complete, your Smartype Hub will have done three things: Retrieved the latest version of your specified Data Plan from your mParticle workspace Created Smartype SDKs based on this version Published these SDKs as NPM packages to your organization’s Github Package Registry under a name based on your Github owner account, the name of your Hub, and a Data Plan ID. Since the npm packages that Smartype Hubs generates are not published to the NPM registry but to your Github Package Registry, anyone importing them into a project must be part of your organization. Importing a Smartype Hub Package into your projects Now that your packages have been generated, you can import both the latest version of your data plan and the Smartype SDKs based on this plan into your project. To do this, first create an file in the root directory of your project with your Github account owner and registry URL. For example, if your company name were Higgs, you would add the following to your file: .npmrc .npmrc @higgs:registry=https://npm.pkg.github.com/ Now you can run an from the command: npm install npm install @<higgs>/<smartype-package-distribution> This will automatically update your package.json and add the latest version of your data plan to your project. Note that if you made your Smartype Hub private, you will need to make sure you configure authentication. All of the benefits of Smartype without switching development environments And that’s it! Now you have your latest data plan and Smartype SDKs in your projects, and you did this using web-centric languages and development tools––no Java installation was necessary. Smartype, and all of its code-completing, error-preventing, time-saving benefits, is now a part of your CI/CD process, as is the process of synchronizing your project with the latest version of the Data Plan in your mParticle workspace. Perhaps Alex sums it up best: “If Smartype can export your data plan into code that works for your platform, why bother having to use Java when you can just import your data plan as a package in your language of choice?” Want to receive curated reads from the data engineering landscape? Sign up for mParticle’s . Written by developers, for developers, including content such as technical blog posts, documentation updates, and curated reads from the data engineering landscape. mPulse Developer Digest

Stories, News & Mentions about DATAPELAGO

Datapelago’s Stories on HackerNoon