Places Dot

placesdot

This new Unity package will teach you how to build a multiplayer AR game in exactly minutes 20 Basic placenote multiplayer sample with a Moon and some Cross Wings We just released a new Unity package that will teach you how to build a multiplayer AR game in exactly 20 minutes. If you’re building (or want to build) a multiplayer AR game or app, this article is for you. Multiplayer AR has been considered by many in the industry to be the key ingredient to making AR apps more social, and ultimately more “sticky” with users. That sounds amazing, but how do you go about building a Multiplayer AR game? What are the design challenges with building an app like this? While there are several SDKs in the market building the tech needed to enable multiplayer AR, that’s far from all you need, as a developer, to build a multiplayer AR app. There’s still a ton of work you need to do to build the networking, UX, server back-end, and game mechanics before you can launch a reasonably good-looking multiplayer experience. So we decided to make this easier. The Problem At Placenote, we built the first multiplayer AR game ever released on the App Store — — a quirky clone. It was no Pokemon Go, but it helped us learn a lot about the problems with building multiplayer AR experiences. We learnt that it was hard to build multiplayer AR, not just because of the tech needed to align many phones to a single world origin, but also because building a networked online game requires a ton of set up to help users communicate with each other. Place Invaders Space Invaders With the latest ARKit 2.0 release, Apple has included capabilities similar to Placenote, but as a developer, using this means you need to manage the storage and sharing of maps yourself, as well as build all the player communication into your game from scratch (or use iCloud). ARWorldMap If you plan to build a multiplayer AR game that can scale up to a lot of users, you’ll likely want to use a mature networking SDK like . Photon Engine Our Solution At Placenote, we want to make it insanely easy to build amazing AR apps. Placenote SDK provides a fully managed cloud solution for mapping and localization, but we wanted to go much further. We wanted to simplify building all the pieces of a multiplayer game, so you can focus on your game rather than wasting time building infrastructure. Introducing an extension to the existing PlacenoteSDK that offers everything you need to get your first multiplayer app up and running. Placenote-Multiplayer. — Placenote Multiplayer Manager Screenshot It’s a Unity Package that includes prefabs to quickly setup Player position tracking, Player Networking (Using Photon) and a simple Game UI elements that you can easily extend to build any Multiplayer AR experience you can dream of. We’ve also created a guided video tutorial to help you build a simple multiplayer experience in 20 minutes. Getting Started It’s time to get our hands dirty with code. To get started, check out the step-by-step tutorial we created to help you set up a simple, yet scalable multiplayer AR session in 20 minutes. Build a Multiplayer AR Game in 20 minutes If you want to jump right in check out our Github repo, PlacenoteSDK-Multiplayer. _PlacenoteSDK-Multiplayer - A Unity project template to get you started creating a multiplayer augmented reality game…_github.com Placenote/PlacenoteSDK-Multiplayer If you’re new to Placenote SDK: If you’re new to Placenote SDK, you’ll want to first learn a bit about Placenote and you can do that with our starter Sample app here: Getting started with Placenote SDK Diving Deeper Once you’ve gone though these videos and tried the sample code, it’s time to broaden your understanding of multiplayer AR development and start experimenting with new designs and game mechanics to build the ultimate mobile AR experience. Here’s a few bits of information that might serve as a useful starting point. Anatomy of a Multiplayer AR Game At a basic level, all multiplayer AR games have the following components: Allowing players to find each other and connect on the same server Setting up a world mapping session to track each player’s position Sharing the world map across multiple players through the cloud Communicating the relative position of one player to other players Synchronizing game assets and actions between multiple players Placenote-Multiplayer includes Prefabs to accomplish each of these tasks in a simple manner but you can now extend and customize these modules to suit your game. Technologies Being Used In the sample’s above, we’re using the following . You can dive into each of these to learn more about them. tools : The best IDE for AR game development Unity3D : Cloud Mapping and localization service to simplify AR persistence and multi-user synchronization. Placenote SDK : iOS libraries for AR experiences. ARKit One of the most popular cloud APIs for multiplayer game networking. Placenote-Multiplayer will handle all the Photon integration, so you do not have to worry about it. Photon Engine: Example of a Real Game Built with Placenote-Multiplayer We created a simple game called “Place Invaders” using the multiplayer SDK as an example to show you how to extend the Placenote-Multiplayer. This game is an alien defense survival game where you use your phone to defend against approaching aliens. You can try it on the App Store here: _‎Read reviews, compare customer ratings, see screenshots, and learn more about Place Invaders. Download Place Invaders…_itunes.apple.com ‎Place Invaders on the App Store Or checkout the source code on our repo here: _PlaceInvaders - Multiplayer AR game sample_github.com Placenote/PlaceInvaders Next Steps There are of course, many more challenges with building multiplayer AR games like robust position tracking and seamless gameplay but for now, we wanted to give developers an easy way to get started about multiplayer AR development and dreaming up new, delightful experiences that can usher in an new era of Social AR. learning So go ahead, get started and build the next generation of highly engaging, social, mobile AR experiences! We, at Placenote are here to help you at every step of the way, so if you have a question or just want to brainstorm ideas for games, . join our 250 member strong slack community See you on Slack!

Building a Multiplayer ARKit Game with Placenote SDK

Hi guys, I want to introduce . It's a React.js for . With it you can build your own places autocomplete UI like the following demo easily! Hope you guys 👍🏻 it. use-places-autocomplete hook Google Maps Places Autocomplete ⚡️ Try yourself: https://use-places-autocomplete.netlify.app Features 🧠 Provides intelligent places suggestions powered by . Google Maps Places API 🎣 Builds your own customized autocomplete UI by React . hook 🔧 to do geocoding and get geographic coordinates using . Utility functions Google Maps Geocoding API 🤑 Built-in for you to save the cost of Google APIs. cache mechanism 💰 Built-in debounce mechanism for you to lower the cost of Google APIs. 🚀 Supports asynchronous Google script loading. 📜 Supports type definition. TypeScript ⌨️ Builds a UX-rich component (e.g. and keyword support) via comprehensive . WAI-ARIA compliant demo code 🦔 Tiny size ( ). No external dependencies, aside for the . ~ 1.7KB gzipped react How does it work? First, use the script tag to load the library in your project. <script src= "https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&libraries=places" ></script> Start to build our component. Check the out to learn more. API getGeocode,
  getLatLng, ready,
    value, setValue,
    clearSuggestions,
  } = usePlacesAutocomplete({ }, }); clearSuggestions();
  }); setValue(e.target.value);
  }; clearSuggestions(); });
    }; place_id, } = suggestion; );
    }); );
}; import usePlacesAutocomplete, { } from "use-places-autocomplete" ; import useOnclickOutside from "react-cool-onclickoutside" ; const PlacesAutocomplete = () => { const { suggestions : { status, data }, requestOptions : { /* Define search scope here */ debounce : 300 , const ref = useOnclickOutside( () => { // When user clicks outside of the component, we can dismiss // the searched suggestions by calling this method const handleInput = ( e ) => { // Update the keyword of the input element const handleSelect = ( { description } ) => () => { // When user selects a place, we can replace the keyword without request data from API // by setting the second parameter to "false" setValue(description, false ); // Get latitude and longitude via utility functions getGeocode({ address : description }).then( ( results ) => { const { lat, lng } = getLatLng(results[ 0 ]); console .log( "📍 Coordinates: " , { lat, lng }); const renderSuggestions = () => data.map( ( suggestion ) => { const { structured_formatting : { main_text, secondary_text }, return ( < li key = {place_id} onClick = {handleSelect(suggestion)} > < strong > {main_text} </ strong > < small > {secondary_text} </ small > </ li > return ( < div ref = {ref} > < input value = {value} onChange = {handleInput} disabled = {!ready} placeholder = "Where are you going?" /> {/* We can use the "status" to decide whether we should display the dropdown or not */} {status === "OK" && < ul > {renderSuggestions()} </ ul > } </ div > Easy right? This is the magic of the ✨. I just show you how does it work via the minimal example. However there're more things you can do for an UX rich autocomplete component, like and keyword support as my (check the ), a keyword clear button, search history etc. usePlacesAutocomplete WAI-ARIA compliant demo code 💡 is my other hook library, which can help you handle the interaction of user clicks outside of the component(s). react-cool-onclickoutside Thanks for reading, for more usage details checkout the project's GitHub page: https://github.com/wellyshen/use-places-autocomplete You can also install this package is distributed via . npm $ npm install --save use-places-autocomplete $ yarn add use-places-autocomplete # or When working with TypeScript you need to install the as a . @types/googlemaps devDependencies $ yarn add --dev @types /googlemaps # or $ npm install --save-dev @types /googlemaps

Create your React.js Address Autocomplete Component in 10 minutes ⏰

When I first started KinkBNB in 2015, I bought an off-the-shelf AirBNB clone codebase to build the first version on. One of the things we had to do was hook it up to with an API key. It was amazing to me how it transformed the dead piece of software we’d gotten from an off-shore programming company into something that seemed to work. It was only after we got into operation that the problem became clear — despite the fact that the software worked, it didn’t actually monetize our traffic because AirBNB’s business model only works at scale. It took us two different packages to come to this realization in 2017. Google Places API This became the impetus for — a completely custom written piece of software. Based on my research I chose the Django framework to write this version. There were a few advantages to this. I’m a pretty hands on guy and I know python like the back of my hand from writing IRC bots in the late 90’s. It allowed me to keep the old databases and userbase intact, and that was pretty cool. But above all I figured out how to drop this Google API into our codebase. This gives KinkBNB the spark of life it would not have otherwise — you can type in an address and it will not only autocomplete based on your location, it will give us the GPS coordinates to do a Point based search in . KinkBNB Mark III Haystack I’ve revisited this particular API a few times for KinkBNB. I have used it in other projects for clients as well. It’s pretty good for building dealer locators as well as finding the nearest restaurants to your sex dungeon. I am still all about this API. It’s Google that I have a problem with. Specifically, their non-communication in making changes. I don’t have much of a reason to pay attention to the backend of this API in the — at least I didn’t before June of 2018. It was then that Google decided they wanted to make money off this API and started charging for it. We didn’t actually notice this for a few reasons. Google Cloud Console We’re in a very weird area with KinkBNB. We get a lot of traffic and we are profitable, but it’s that weird kind of profitability you get in that first year of profitability. Namely, everything affects your bottom line. We exhibit at the and it’s not an inexpensive proposition — but this year we paid for it completely with profits. It was about this time that I started getting notices in my personal mail about our credit card being declined for small payments. Folsom Street Fair Of course, several years ago when I wasn’t thinking I’d signed up for this with my personal email. I shrugged it off until after the Fair, how bad could it be? This photo by the captures me (left, in my trusty hat) on my phone trying to figure out why Google is charging us so much despite all the stuff going on around me. SF Chronicle As it turned out, it was pretty bad. Not only were we badly behind, for some reason Google had given us a credit or something going forward several months. We were pulling a LOT of requests and at their full pricing tier. My best guess is they knew folks would need time to fix it, but in the meantime here is $500 a month credit for several months to make your bills not pile up. I sat up and took notice when that credit stopped. Here’s the fun code portion of the article! The very basic call to Places is simple. You grab an and throw up this piece of HTML and Javascript which is and VOILA — you have a map. API key from the API Dashboard currently on their website <!DOCTYPE html>
<html>
  <head>
    <title>Simple Map</title>
    <meta name="viewport" content="initial-scale=1.0">
    <meta charset="utf-8">
    <style>
      /* Always set the map height explicitly to define the size of the div
       * element that contains the map. */
      #map {
        height: 100%;
      }
      /* Optional: Makes the sample page fill the window. */
      html, body {
        height: 100%;
        margin: 0;
        padding: 0;
      }
    </style>
  </head>
  <body>
    <div id="map"></div>
    <script>
      var map;
      function initMap() {
        map = new google.maps.Map(document.getElementById('map'), {
          center: {lat: -34.397, lng: 150.644},
          zoom: 8
        });
      }
    </script>
    <script src="https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&callback=initMap"
    async defer></script>
  </body>
</html> Now, the piece of code to focus on is that maps.googleapis.com call at the end. <script src="https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&callback=initMap"
    async defer></script> That piece of code right there is the very BASE call. You can use that as the basis of every call to Places API, whether it is the maps or the geocoding or the autocomplete. Notice also that is only has two values being passed — the API key and the callback. Folks, that piece of code controls how much you are being charged. If you just call it with your API key, it charges you the maximum amount they can, per request — because ostensibly it returns all the data in their databases about what’s near you or in your coordinates. Google Maps and Places now has a new If you want just basic address capabilities, that’s a very low rate. But if you grab everything in their API, the bill piles up VERY quickly. Luckily you can staunch the flow of money pretty quickly. pricing structure according to their website. December is my traditional month to do a good code sprint on KinkBNB. This year I had to fix this problem ASAP, and code a new GPS enabled search function for our user directory. I took the time to figure out and understand what was going on. It takes a few days sometimes to figure out if your code actually worked when you have an external API, and I can report that the following problem was definitely fixed by some simple additions to the Places API call lines. First off — you need to make your Google Places API calls priced by session instead of by every API call. Luckily in Django it’s just a matter of adding a value to your API call. <script type="text/javascript" src="//maps.googleapis.com/maps/api/js?libraries=places&&v=3.34&key={{ GOOGLE_MAPS_API_KEY }}&sessiontoken={{request.session.session_key}}"></script> Next, you need to limit what data you are calling from Places. I wrote the following Javascript function that runs when the page loads — it lets me access the location data from the browser, stores it for the user who is logged in, and sets up the Autocompleting search bar on our search pages. {% block head_js %}
<script type="text/javascript" src="//maps.googleapis.com/maps/api/js?libraries=places&&v=3.34&key={{ GOOGLE_MAPS_API_KEY }}&sessiontoken={{request.session.session_key}}"></script>
<script type="text/javascript">
    function showError(error) {
        switch(error.code) {
            case error.PERMISSION_DENIED:
                alert("User denied the request for Geolocation.");
                    window.location.href = "https://www.kinkbnb.com/explore/";
                break;
            case error.POSITION_UNAVAILABLE:
               alert("Location information is unavailable.");
                    window.location.href = "https://www.kinkbnb.com/explore/";
                break;
            case error.TIMEOUT:
                alert("The request to get user location timed out.");
                    window.location.href = "https://www.kinkbnb.com/explore/";
                break;
            case error.UNKNOWN_ERROR:
                alert( "An unknown error occurred.");
                    window.location.href = "https://www.kinkbnb.com/explore/";
                break;
        }
    }
        function initialize() {
            var input = document.getElementById('geocomplete');
             if (navigator.geolocation) {              navigator.geolocation.getCurrentPosition(function(position) {
                  var pos = {
                     lat: position.coords.latitude,
                     lng: position.coords.longitude,
               };
                   document.getElementById('id_lat').value = position.coords.latitude;
                   document.getElementById('id_lng').value = position.coords.longitude;
                      $.ajax({
                    url : '/kinksters/{{ request.user.id }}/gps/',
                    type : "POST",
                    data : {
                        'csrfmiddlewaretoken' : "{{  csrf_token  }}",
                     'lat':position.coords.latitude,
                     'lng':position.coords.longitude,
                    },
                    success : function(result) {}
                   },showError);
                   }, function() {
                             });
            } else {
              // Browser doesn't support Geolocation
                    window.location.href = "https://www.kinkbnb.com/explore/";
             }
            var autocomplete = new google.maps.places.Autocomplete(input,{fields: ["address_components", "geometry"],
      types: ["geocode"],});
            google.maps.event.addListener(autocomplete, 'place_changed', function () {
                var place = autocomplete.getPlace({placeIdOnly: true});
                var lat = place.geometry.location.lat();
                var long = place.geometry.location.lng()
                document.getElementById('id_lat').value = place.geometry.location.lat();
                document.getElementById('id_lng').value = place.geometry.location.lng();
            });
        }
        google.maps.event.addDomListener(window, 'load', initialize);
    </script>

{% endblock head_js %} As in the previous little codeblock, there are specific lines you need to pay attention to here. var autocomplete = new google.maps.places.Autocomplete(input,{fields: ["address_components", "geometry"],
      types: ["geocode"],});
var place = autocomplete.getPlace({placeIdOnly: true}); This is how you limit what data you get back from Places API. You are limiting your fields to geometry and address data, and the type of call you are making is “geocode”. The second line is to specify that you don’t want all the Places Details in your call. If you leave these lines in without specifying the code type, placeIDOnly, or the address fields, then Google will charge you at the full data rate for their Places Details API — and this is what took a few days to figure out after I figured out to make all API calls part of a session. Naturally if you want to get more data back from them this is where you do it as well — but all the tiers cost more on top of what Google is charging as a base. Per Google’s website, here are the available fields and pricing tiers: Basic The Basic category includes the following fields:address_component, adr_address, alt_id, formatted_address, geometry, icon, id, name, permanently_closed, photo, place_id, plus_code, scope, type, url, utc_offset, vicinity Contact The Contact category includes the following fields:formatted_phone_number, international_phone_number, opening_hours, website Atmosphere The Atmosphere category includes the following fields: price_level, rating, review If you want to include reviews and ratings, then you can do this. Just be warned that this is charged at the maximum rate possible. If you get between 20,000 and 30,000 users a month on your search bar and every API call is at maximum rate, that’s going to add about $1000+ a month to your bill. The minimum rate is much more palatable so far. I had virtually no help from Google to solve this. I say virtually, because I actually used Google to search the internet for the answers — and those were not published officially by Google except in their dry documentation that just assumes you’re starting from scratch. I didn’t get or notice an email warning about the change in pricing, nor did I find anything explaining this change and how it would affect older customers. It to be a trick to get as much surprise revenue as possible before their users fixed it. Nobody would be that evil, would they? seems And so it comes full circle to my article’s title. Google used to have a motto. It was “Don’t be evil.” Unfortunately they seem to have left this motto by the wayside, sometime around the time they bought that company that builds scary military robots. Surprising people with pricing changes is about as evil as it gets. I hope someone at Google sees this and tries to do better next time they want to implement a large change that will affect small businesses like mine if we don’t jump right on fixing our code.

Dear Google: Don’t Be Evil

is written text or illustration that accompanies computer or is embedded in the source code. It either explains how it operates or how to use it, and may mean different things to people in different roles. is an important part of engineering. Software documentation software Documentation software https://en.wikipedia.org/wiki/Software_documentation Stage 0 — Google docs plus keeping functionality in my head It’s easy when you have only one person that coding your project.Main problem — every time that I want to write something new:a) I should open my Gmail b) then go to Google Docsc) then I should pick the folder where I save all data, d) open my document Disadvantage It’s a bit unmanageable. If you want to share a link with someone — you should generate a sharable link, copy and send it. Some people don’t have a google account(crazy right?). And it can be a problem too. Content didn’t index by web spiders so nobody can find it accidentally. Doesn’t have a cool search(at least I use it mostly for title search only) When you work on a big document you need to create a table of contents yourself. : Link https://www.google.com/docs/about/ GitHub Stage 1 I keep my code and documents/files at main GitHub repositories. A lot of mess because it’s for storing your code. Sometimes having everything in one place are very pressing.Hard to search. You need to remember where you keep it. Disadvantage : Link https://github.com/ Stage 2 Instead of having a development plan at markdown files — I move some tasks to GitHub issues, add labels and milestones. It is a good approach if you still working alone.But when a project will grow — you’ll drown. As I did. I have a big long list of issues. it’s hard to manage it quickly.But it’s better to have a documented issue on Github, instead of storing it in memory or papers. I create a few separated repositories for documentation only.As I was still alone — I just moved all files without any structure. Stage 3 It’s still doesn’t look like proper documentation. Nobody can navigate it, I store content in different languages, etc. Disadvantage Stage 4 Github has a good tool for documentation — pages/wikis.It’s a good idea, but it didn’t work well for me. Not sure why. Maybe it requires a lot of time for sharing and storing data there. I think it can work for many people. Wikis didn’t require a lot of tech knowledge to start working with. Hard to upload images/screenshots. But! it was an opportunity for us.I create separated repositories for keeping all assets in one place.So logos, brand book, database schemas stored at repository.It’s also good way to share information with designers. And I make it more easy to navigate for humans. And when I needed to share something — I just go to that repository and copy an URL. Disadvantage “creative” : Link https://pages.github.com/ Stage 5 — Gitbook Really cool product. I like it. It syncs with Github.And it looks good. Works fine with Markdown formatting. Disadvantage It has some UI issues. Like it has a custom online editor aka draft.js And it’s not very intuitive how to add cross-links to subpages. You need to make a 2–3 moves if you want to do it. Again it looks good and has more features that I didn’t explore, like slack integration It’s better to separate data between different scopes(sections of documentation). Because if you’ll have 3 main pages with 10 subpages each — it will be hard to navigate or drag pages between sections. For the not-coding person, it’s definitely a good way to go. Why I leave it? They finish their beta testing and introduce their paid plans, and we cannot afford them right now.So it was time for new moves. : Link https://www.gitbook.com/ At some point, between stage 3–4 I read about Amazon development approach.Briefly — you explain in simple sentences what release will have in a result.And I recommend you to read more about it. : Link https://www.quora.com/What-is-Amazons-approach-to-product-development-and-product-management Current stage This is not a linear process, so we still stuck with some documentation at previous stages. Right now we moving everything to Docusaurus. I like that it’s a tool baked by Facebook. It’s free and open-sourced.It builds with React so we can adjust the code if it is necessary.Similar to Gitbook — works with Markdown. But it’s more for coders.Or developer should set it up and the manager will just work with content.With markdown format, you can read/view files at Github too.It also works great with GitHub pages. You just need to deploy it on And you’ll have a separated URL with SEO stuff that can drive more attention to our project. gh-pages : Links https://docusaurus.io/en/ https://docusaurus.io/docs/en/installation https://docusaurus.io/en/users Summary Documentation is a very important tool in the development process and you should use it as much as possible.If you have long meetings, 1-to-1 conversations, kilometers of chats in Slack — then you need to improve it ASAP. I still don’t feel like this is the end of our “evolution process”. A major goal will be to increase modularity and drive more attention to it. We still use it only if it necessary. Instead of “update a documentation file -> share data with team members” we are “having long conversations and then copy some of our comments into documentation for the future needs”. If you want to know about how I manage my projects — read it here: https://medium.com/quick-code/how-i-manage-my-projects-c17c78c796c4 Clap if you like it (Or I will delete your facebook account) From: https://dribbble.com/shots/3005865-Moustached-nanny

How I set up a startup documentation

Yesterday Apple kicked off its annual World Wide Developer’s Conference (WWDC), and the keynote held some big surprises for its ARKit platform. Among other improvements, ARKit 2 will hold the following new functionality: 3D object detection Persistent experiences Shared experiences Now, we’ve all been hearing about the ARCloud lately, with companies like Google pushing their services at this year’s Google I/O, as well as smaller startups like 6D.ai, and Placenote trying to wedge themselves into the space as the authoritative platform. Apple, in the last few years, has been working silently (normal modus operandi) to first create a framework for detecting surfaces, and now, for detecting objects, serializing, and sharing the recognized world. Let’s take a closer look at each of these new features. Detecting Objects As Lego demonstrated in the keynote, ARKit is now able to detect physical objects. The way that this is done is by initially training the model (as in machine learning model, not 3D model) presumably using the Vision and Machine Learning facilities that Apple has developed under the hood. The object that is to be detected is first analyzed by pointing your phone camera at it, and moving all around the areas that are to be recognized. This allows the framework to analyze and save the feature points that it finds on your object. The framework is then able to recognize the same feature points in a different session by comparing the two datasets. Interacting in the same world To train the machine learning model, Apple is providing the source code to a full-featured app that they wrote to simplify the process of scanning the object. From there, the process is simple. Load the model into your app, start an object detection session and when the object is detected, you’re given the proper transforms for laying out your world around the object. Virtual content overlaid on top of real object Persistent Experiences Along with the new detection capabilities, the major feature is the ability to persist the recognized world. That is, once you detect enough feature points in the area that you’re scanning, you’re able to save the data that describes the world. You can then reload the world, not just in the same device, but any other device as well, allowing for a quick startup when looking at the same area at a later time. Save a world map when your app becomes inactive, then restore it the next time your app launches in the same physical environment. -Apple ARKit docs This is a really important feature. This allows for serializing the current scene, feature points and anchors in all, making restarting a session nearly instantaneous. This is also used with the shared experience functionality, whereby the same world can be scanned once and used by multiple users, as described in the next section. Shared Experiences I’m currently writing this while watching a match between two conference-goers in a game that Apple created to showcase multiuser AR. SwiftShot is a take on a tower battle, where each player has to knock down the other’s towers using AR slingshots that they are controlling with iPads. The players are on each side of a large wooden table, which is holding the virtual content. The match is simulcast via a third iPad that a staff member is holding, acting as the camera person. There is a large television showing the match, which includes the players and the virtual content. The play area is rock solid on the table, showing the advanced tracking capabilities of ARKit. Craig Federighi on shared AR experiences This is a great example of a virtual world being shared by multiple participants. They are both seeing the same world, watching their actions affect the scene objects in realtime. This opens up amazing new possibilities for games and experiences, including being able to collaborate and work on the same subject together. With two devices tracking the same world map, you can build a networked experience where both users can see and interact with the same virtual content. -Apple ARKit docs What all this means Apple came out of nowhere with ARKit, and they are quickly iterating on the framework, even going as far as updating to ARKit 1.5 during an iOS point release, which included static image and vertical plane detection. “Simply put, we believe augmented reality is going to change the way we use technology forever,” — Apple CEO, Tim Cook With ARKit 2.0 they raising the bar and moving quicker than some startups, releasing technology that may render many of them obsolete even before they launch. This train seems unstoppable, with Apple making cohesive frameworks that open up new possibilities for developers to create amazing experiences. is a developer who’s worked on more platforms and languages than he cares to admit to. He believes in building products that create memorable user experiences. To Dan, the users are first, the platforms are second. Follow Dan on or to hear more from him. Also check out the blog, where a lot of great content by Nike’s Digital Innovation team gets posted. Daniel Wyszynski Medium Twitter s23NYC: Engineering Author’s Note: Views are my own and do not necessarily represent those of my employer

The Apple AR Train

How to redesign your AR app for persistent sessions. IKEA Place was one of the first ARKit app on the iOS app store One of the first ARKit apps to be released on the App Store was , a furniture shopping app, that let people virtually their own homes with IKEA’s furniture catalogue before making a purchase. IKEA Place design Since then, product visualization for furniture and home decor has become one of the most popular applications of AR on mobile phones and even big retailers like Lowe’s and Amazon have jumped in to experiment with the power of AR to influence furniture shopping behavior among consumers. If you’ve never tried one of these apps, check out IKEA Place or Housecraft . So what is the “Resume Session” problem? The “Resume Session” problem refers to a common problem faced by all AR interior design apps on the market today—It is that they have no way to save the positions of virtual furniture placed in a physical location from one app session to the next. That means every time a user accidentally or intentionally closes the app, they lose all progress and need to start re-designing their space from scratch! This post is about why this is a problem with current apps and how you can potentially solve it using a new SDK available for ARKit — Placenote SDK . Placenote SDK enables the persistent placement of AR content in any physical space. To start, let’s first look at how a basic AR interior design app is built. How do AR Interior Design apps work? These apps basically work by letting you virtually place 3D models of furniture on flat surfaces around you in AR. So, if you’re thinking of buying a new couch, you can see what it will look like from different angles and visualize whether the color and size of the couch suit your room. To understand how these apps work at a technical level, you need to understand how to build a simple ARKit-based iOS app that can detect horizontal planes and place 3D objects on it. If you’re new to this, that will show you how to build it. here’s a great tutorial Once you can build an app that can place a 3D object on a flat surface, you can simply replace the 3D object with a model of a chair, . like this one Now, at a very basic level, every furniture app has two main components **1. An interface to place and move 3D models on a flat plane**ARKit can easily detect flat surfaces and let you place 3D objects on them. Screen record of furniture placement in Housecraft on the App Store **2. A catalogue of 3D models to choose from**3D models will generally be provided by the furniture retailers. For the sake of prototyping, you can use free 3D models from websites like , or . Sketchfab Google Poly Turbosquid 3D models from Google Poly The combination of these elements is the basis for all interior design apps available on the App Store today and while many of these are beautifully well designed and fun to use, there’s one big problem with all these apps today. The Elephant in the Room If you’ve tried one of these apps on an iPhone or iPad, this will sound familiar to you. You painstakingly add various items of furniture around your room, align them correctly and choose just the right colors that match your decor. After 15 minutes of virtually decorating your room, if you accidentally (or intentionally) close the app and open it again…and , your design has disappeared and you need to redo everything again. Not only that, but you can’t share these designs with anyone, except taking a picture or video of the AR scene. POOF From a UX perspective, this isn’t a great experience given that the two main functions of most visual design tools are the ability to save and resume progress and collaborate with others. So let’s take a technical dive into what the problem is. The Problem with ARKit If you’ve worked with ARKit you know that each time an AR app is opened, ARKit sets an arbitrary (0,0,0) point at the phone’s current position and places all content in the scene relative to that point. This is because, although ARKit detects visual features in the environment to track the phone’s motion, it has no ability to “remember” these features. This means that you cannot build a system to save and resume your design session if the locations of your objects relative to the real world are important. You need to realign your furniture every time you open the app. In most user flows, this is unwanted friction. So what is the problem? The issue with ARKit is that the visual features it uses to track motion are not “distinct” enough to be repeatably found in the same place every time it observes an area. In order to save digital content in specific areas of a house, we need a way to permanently fix the positions of AR content in a physical space, so that every time an app is opened, the origin point of your scene snaps to its correct absolute position in the real world. Let’s call this feature “ ”. persistence Persistence in AR is the ability to let digital objects permanently retain their position in the real world across multiple app sessions. One way to solve this problem, is Placenote SDK . The rest of this post will teach you how to build an interior design app that incorporates persistent sessions in its workflow. Solving the Persistence Problem To give digital objects a persistent position in the real world, your app will need some representation of the real world (i.e. a map) and a way to find it’s position on that map. Apps like Pokemon GO did this by using GPS to geolocate Pokemon near specific real world landmarks like shops and parks. This kind of geolocation is sufficient for rough positioning but since GPS is so inaccurate, the positions of content can be between 5 to 20 meters off target every time you open the app. Further, GPS doesn’t work indoors which is where most “interior” design happens. Pokemon go uses GPS for geolocation with low accuracy. Placenote SDK takes a different approach to solving this problem. Rather than GPS, it uses a camera to visually scan a physical space and turn it into a map for retrieval. It also gives a camera the ability to position itself in a map by comparing its current image with a pre-created map. This way, Placenote implements a visual positioning system that is self sufficient (doesn’t need external beacons) and above all, . future works indoors How Placenote Does Visual Positioning Placenote divides the positioning process into 2 main steps — The camera is used to visually scan a physical area. Once mapping is complete, Placenote saves and uploads the map to the cloud, and returns a mapID that can be used to reload or share a map with multiple devices. Mapping: A pre-constructed map is downloaded using a mapID and Placenote attempts to match the image currently seen by the camera with a portion of the map. When it finds a match, it returns a position estimate of the phone/ tablet within that map. Relocalization: If you’re a computer vision geek like me, this will sound familiar to you. (SLAM). This system, while mostly seamless, will likely change the design of your user workflow a bit. Let’s look at how you can redesign an interior design app to incorporate the visual positioning paradigm. Redesigning your App for Visual Positioning Here is a potential redesign for an interior design app that incorporates a visual positioning workflow into its UX. 1. Designing a Room (and Mapping Simultaneously) With Placenote, the design stage needs to be modified slightly to run the mapping procedure in the background. That means, as a user is placing items of furniture in the room, Placenote can start building a 3D map of the environment by running a parallel feature extraction and mapping process in the background. The good news is, this can be done without ay extra prompts to the user. It should seamlessly fit into any existing workflow. If you’re coding in , you can run this line at the end of your viewDidLoad() function in your View Controller. This will start the mapping process immediately when a user opens the app. Swift on Xcode LibPlacenote.instance. startSession() Also add a new function to send image frames to Placenote from ARKit. func (_ session: ARSession, didUpdate: ARFrame) { session let : CVPixelBuffer = didUpdate.capturedImagelet : matrix_float4x4 = didUpdate.camera.transform image pose LibPlacenote.instance. (image: image, pose: pose)} setFrame The design stage can run mapping in the background (green dots are feature points) 2. Saving a Design Session “Saving” a session might be the only additional step added to your workflow if you don’t already have a “save” or “share” button in your app. Since no interior design apps today can really save designs in a specific location, there hasn’t been a real need for a “save” button. However, if you have a “share” feature that lets you capture a screen shot or video of a designed space, you can link the save function to this share button using this code snippet: LibPlacenote.instance. ( saveMap : {(mapId: String?) -> Void in savedCb print("Saved Map ID: " + String(mapId))LibPlacenote.instance. ()}, stopSession : {(completed: Bool, faulted: Bool, percentage: Float) -> Void in uploadProgressCb (completed) {print("Uploaded map!")}}) if In the context of Placenote’s mapping process, the save function saves and uploads a map created during the design process. Once the map is uploaded you will receive a that can be used to reload the map or share it with other devices to make your app more social. mapID From a development perspective, you will need to save the positions of content in your app as well. This can be done locally or on your own cloud backend using a simple JSON file. Save a design session by saving a map and the positions of content in the map. 3. Resuming a Design Session Once a map is saved, you can reload it whenever you want to open a saved design in that area. One way to incorporate this into your UX is to check if there is a previously saved mapID each time the app is opened (in the viewDidLoad() function). If a mapID exists, load the latest map and attempt to localize the device within that map. As soon as Placenote finds a match, the scene can be loaded exactly where it was left off so that the user can continue designing. You can add this line to your function in your View Controller. viewDidLoad() // if you saved a mapID in user defaults, retrieve it like this guard let savedID = defaults.string(forKey: "MapID")else { return false } LibPlacenote.instance. (mapId: “Map ID”, loadMap : {(completed: Bool, faulted: Bool, percentage: Float) -> Void in downloadProgressCb if (completed) { LibPlacenote.instance.**startSession**() }}) The session can be resumed as soon as the app is reopened. And that’s it! With a few small changes you can add a entirely new dimension to your AR experience, making it persistent, more social and overall a better solution for the interior design application it’s built for. The code examples above were in for Native SceneKit development, but Placenote SDK is available as a plugin as well. To learn more about Placenote or try the sample apps, check out . Swift Unity Placenote.com Getting Started with Placenote SDK You can learn more about how Placenote helps with building v apps here. I will be releasing the sample code for the app example above in the next few days so if you’re interested, sign up for an API key and try Placenote SDK for Unity or SceneKit, using the install guide linked below. irtual furniture design and staging _Click the link below to open your Developer Portal. Sign up for an account and create a new API key. Then return to…_placenote.com Get API Key and Choose Platform | Placenote How to Get in Touch To get in touch with me about Placenote, ARKit or anything AR/VR related, feel free to reach out at: Email me: neil [at] vertical.ai Placenote Slack Community: Join our slack group

Supercharge your ARKit Interior Design App with a “Resume Session” feature.

“Occlusion” means hiding virtual objects behind real things. One of the biggest and most elusive pieces of the augmented reality puzzle is occlusion. In other words, the ability to hide virtual objects behind real things. This post is about why occlusion in AR is so hard and why deep learning might be the key to solving it in the future. When you look at a purely real or a purely virtual world, you tend to accept the “rules” of that world or as long as it satisfies some basic notions of reality like gravity, lighting, shadows etc. You’ll notice when these rules are broken because it’s jarring and feels like something . That’s why it’s so instinctual to cringe at bad special effects in movies. suspend disbelief, “doesn’t look right” In VR, it’s actually quite easy to achieve suspension of disbelief because you have complete control over all elements in the scene. Unfortunately, as an AR developer, you don’t have this luxury because most of your app’s screen real estate (i.e. the real world) is totally out of your control. As an AR developer, most of your app’s screen real estate (i.e. the real world) is totally out of your control. In the mobile world, has achieved incredibly fast motion tracking as well as realistic lighting and shadows, but it’s still lacking when it comes to occlusion. Apple’s ARKit Here’s an example: It’s because the dragon looks like it’s further away from the chair but still appears in front of the chair. Does the screenshot below look strange to you? Without occlusion, this dragon looks weird overlapping the chair. This isn’t just a problem with mobile AR. It’s also a problem on every headset available today. How does occlusion in AR work? The goal of occlusion is to preserve the rules of line-of-sight when creating AR scenes. That means any virtual object that is behind a real object, should be “ ” or hidden behind that real object. occluded So how is this done in AR ? Basically, we selectively prevent parts of the virtual scene from rendering on the screen based some knowledge of the 3D structure of the real world. Doing this involves 3 main functions: the 3D structure of the real world. Sensing a digital 3D model of the world. Reconstructing that model as a transparent mask that hides virtual objects. Rendering But what is so hard about it? Assuming you have a good reconstruction of all the objects in your real environment, occlusion involves simply rendering that model as a transparent mask in your scene. That’s the easy part. It’s getting to that point where things start to get unwieldy. Consider a common street scene. There are people, vehicles, trees and all kinds of objects at various distances from you. Further away, there are larger structures like bridges and buildings, each with their own unique features. The real world is a complicated and dynamic 3D scene The hardest thing about creating a realistic occlusion mask is actually reconstructing a good enough model of the real world to apply that mask. That’s because no AR device available today has the ability to perceive its environment precisely or quickly enough for realistic occlusion. How does 3D sensing work? Sensing 3D structure really boils down to one important ability — depth sensing. Depth sensors come in many flavors, the common ones being , and . Structured Light Time of Flight Stereo Cameras In terms of hardware, Structured light and Time of Flight involve an Infrared projector and sensor pair, while Stereo requires two cameras at a fixed distance from each other, pointing in the same direction. At a high level, here’s how they work: Structured Light Sensor Structured Light sensing works by projecting an IR light pattern onto a 3D surface and using the distortions to reconstruct surface contours. Time-of-Flight Sensor This sensor works by emitting rapid pulses of IR light that are reflected by objects in its field of view. The delay in the reflected light is used by an image sensor to calculate the depth at each pixel. Stereo Cameras Stereo cameras simulate human binocular vision by measuring the displacement of pixels between the two cameras placed a fixed distance apart and use that to triangulate distances to points in the scene. Of course, all these sensors have their limitations. IR based sensors like have a harder time functioning outdoors because bright sunlight (lots of IR) can wash out or add noise to the measurements. Stereo cameras have no problems working outdoors and consume less power, but they work best in well-lit areas with a lot of features and stark contrast. All you need to do to confuse a stereo camera is point it at a flat white wall. Since all these sensors work on pixel-based measurements, any noise or error in the measurements creates holes in the depth image. Also, at the size and capacity of phones and headset devices today, the maximum range achieved so far has been about . 3–4 meters The image below is an example of a depth map created with a stereo camera. The colors represent distance from the camera. See how the measurements are good at a close range while further objects are too noisy or ignored? 3D perception doesn’t end at depth sensing. The next step is to take the 2D depth image and turn it into a 3D point cloud model where each pixel in the depth image gets a 3D position relative to the camera. Next, all the camera relative point clouds are fused with an estimate of camera motion to create a 3D point cloud map of the world around the sensor. The video below illustrates the complete point cloud mapping process. Mapping an museum exhibit in 3D - From a real project we worked on at Placenote. Now that you understand the complete pipeline of 3D perception, let’s look at how this translates to implementing occlusion. Using Depth Sensor Data for Occlusion There’s a few ways 3D depth information can be used to occlude virtual objects. Method 1 Directly use the 2D depth map coming in from the sensor. In this method, we align the camera image and the depth map and hide parts of the scene that should be behind any pixels of the depth map. This method doesn’t really need a full 3D reconstruction since it just uses the depth image. This makes it faster but has a few problems: The sensor can only detect objects up to Anything further won’t be occluded. 4 meters away. The depth map has and isn’t perfect holes The resolution of the depth map is much lower than the camera which means scaling and aligning the two images will create for occlusion. pixelated, jagged edges The video below is an example of depth map based occlusion. Notice the irregularities in the Occlusion mask as the red cube is moved around. Occlusion masking done with a simple depth map. Method 2 Re-construct and use the 3D point cloud model. Since the point cloud is a geometrically accurate map of the real world, we can use it to create an occlusion mask. Note that a point cloud itself isn’t sufficient for occlusion, but point clouds can be processed to create meshes that essentially fit a surface onto the point map (like a blanket covering your 3D point cloud). Meshes are much less computationally intensive than point clouds and are the go-to mechanism for calculations like detecting collisions in 3D games. Environment mesh created with a Microsoft Hololens This mesh can now be used to create the transparent mask we need to occlude virtual elements in our scene. Well that sounds like we have a good enough solution for Occlusion! So what’s the problem? The problem with AR devices today The 3 AR devices that I think have the most impressive tracking and mapping capabilities today are and Here’s how their sensors stack up against each other. Google Tango, Microsoft Hololens, Apple iPhone X. **Google Tango (Discontinued by Google)** Depth Sensor — IR time-of-flightRange — 4m **Microsoft Hololens** Depth Sensor — IR time-of-flightRange — 4m **Apple iPhone X** _Back facing depth sensor — Stereo CamerasRange — 4m_ Forward facing depth sensor — IR Structured light The main problem with all the above systems is that in terms of depth sensing, they have: The size and power limitations of mobile devices restrict the range of IR and Stereo depth sensors. Poor range (<4m): Smaller objects in the scene are not discernible in the point cloud and it’s really hard to achieve crisp and reliable occlusion surfaces. Low resolution: Current methods of generating meshes from point clouds are too slow for real-time occlusion on these devices. Slow mesh reconstruction: Generating a mesh from a point cloud, currently, isn’t fast enough for real-time occlusion on any tablet or headset device. So how does a developer today hack together a reasonable solution to get around these issues? How can you hack occlusion today ? Perfect occlusion is an elusive target, but we can get close to it in some situations, especially when we can . relax the real-time constraint If the application allows pre-mapping the environment, it’s possible to use a pre-built mesh as an occlusion mask for the larger prominent objects in the scene, provided they don’t move. This means that you’re not limited to the 4m range of the depth sensor, at least for occlusion behind static objects. Moving objects are still a problem and the only solution right now it to use the depth map masking method for close range moving objects like your hands. A sample 3D environment mesh built by pre-mapping an indoor space Now it’s clear from the example mesh above that a big problem with pre-built meshes is that although they’re lighter than point clouds, they can cause more than a ten-fold increase in the complexity of your 3D content. The way to simply a 3D mesh is to approximate its structure with simpler objects like walls and blocks that envelope complex structures. and the way we hacked occlusion was to At Placenote, we’ve built guided tours of large museums in AR manually draw planes to cover specific walls in the space that might get in the way of our virtual content. A Unity scene that shows how we hacked Occlusion at Placenote. Visit placenote (https://placenote.com) Here’s an example of occlusion with a high-quality, pre-built environment mesh on Google Tango. Of course, this method assumes that either the developer or the user will take the time to map the environment before the AR session. Since this might be a bit overwhelming for the average user, it likely works best in where the map can be pre-built by the developer. location-based AR experiences In an extreme scenario, you might want to occlude an AR experience at a larger scale, like rendering a dinosaur walking among buildings in New York City. Perhaps, the way to do this is to use known 3D models of buildings from services like Google Maps or Mapbox to create occlusion surfaces at the city scale. much Our friends at have built a unique way of creating city scale augmented reality experiences, using satellite imagery to reconstruct large buildings and static structures in 3D. at Sturfee has . Sturfee Sheng Huang written about their platform here Of course, this means you need to be able to accurately localize the device in 3D, which is quite challenging at that scale. GPS position is simply not good enough for occlusion since it’s slow (1Hz) and highly inaccurate (measurement error of 5–20 meters). In fact, centimeter-level position tracking indoors and outdoors is a critical component of occlusion and through our work with Placenote , we’re working towards a cloud-based visual positioning system that can solve some of these problems. Occluding buildings is a crazy idea. Or is it? We already have 3D maps (like Google maps) that could be used for Occlusion. What does the future look like? While pre-built meshes are great for AR experiences tied to a single location, occluding moving objects still requires instant depth measurements at a range greater than just 4m. What’s needed to create a realistic AR experience is a sensor that produces a high-resolution depth map with near infinite range. Improvements in sensing hardware can certainly help squeeze greater resolution and range from IR or Stereo sensors, but these improvements will likely hit a ceiling and produce diminishing returns in the near . future Interestingly, an alternative approach has emerged in 3D sensing research, that turns this hardware problem into a software problem by leveraging deep learning to improve the speed and quality of 3D reconstruction. Neural networks might be the key to solving occlusion in the future. This method uses neural networks that can pick out visual cues in the scene to estimate 3D structure, much like the way we as humans estimate distance. (i.e. guessing distance by using our general knowledge of the sizes of things in the real world. The networks are trained on a large dataset of images and are capable of segmenting out objects in a scene and then recognizing them to estimate depth. That means, if we can the neural networks and train them on a good enough dataset, we might be able to bypass a lot of limitations in resolution and range present in current depth sensing technologies, with no added hardware costs. design Neural network scene segmentation ( ) http://www.ais.uni-bonn.de/papers/KI_2014_Hoeft_RGB-D_Semantic_Segmentation.pdf The image above is from a paper that explores methods to segment and label scenes using neural networks in combination with depth sensors to improve the quality of generated maps. You can find the full text of the paper here. In Summary Occlusion is, by far, one of the biggest pieces of the AR puzzle because it makes the biggest leap towards realism for AR experiences. Depth sensors today are too slow, have limited range and low resolution for real-time occlusion. You can get around these limitations by building AR apps in areas with pre-built environment meshes. Try Placenote SDK to build location-based AR experiences. The key to solving the range and speed limitations of depth sensors in the future might be deep learning and this approach is already showing promising results. If you’re a new AR developer looking to build compelling AR experiences, don’t let occlusion stop you. Remember Pokemon Go? Poor occlusion in Pokemon Go resulted in some hilarious AR screenshots that spread all over the internet and helped with the meteoric rise of the game. So have fun with it! If you want to build amazing AR experiences on iOS or Unity, partner with us, or join our team, let’s connect! Contact me at neil@vertical.ai Who are we? We’re building an SDK for persistent, shared augmented reality experiences. We call it . Placenote SDK Special thanks to Sheng Huang , Dominikus Baur , David Smooke and Peter Feld for your help with reviewing this article!

Why is Occlusion in Augmented Reality So Hard?

If you’re looking to move to Microservices, I have some extremely helpful advice for you that we have learned during our migration from a monolith to a scalable, maintainable microservice . architecture Background We’ve been building our platform now for over 18 months. It started off relatively simple. We had a go server (using ) with a number of REST endpoints. It looked a little bit like this. small Gin https://oursite.com/api/v1/placeshttps://oursite.com/api/v1/usershttps://oursite.com/api/v1/bookings Every resource had it’s own versioned URL. They would each support , , , as necessary. We had 2 seperate mobile clients connecting to these APIs and we had a centralized database. We used for user authentication and everything was going fine. We were handling a fair number of requests per hour. However, we needed to change our platform. The mobile apps weren’t working out. POST GET PATCH DELETE JWT This is where we started to hit issues. We had a large number of new requirements involving 2 web applications. There were two choices, only one of which was to be viable. We could either rewrite our entire architecture and spend a number of months writing a new server with required functionality — or we could use our existing server and add new functionality to it. Given our time constraints, we used our existing server. After a few weeks, our legacy code was already starting to grow and we were starting to encouter issues. Changing models was becoming difficult, We couldn’t keep this up much longer and we came to another turn in the road. Our choices were now either: keep building the current server and encounter hard problems upon scaling migrate to a more scalable approach that was also maintainable. A , if you will. everything seemed like a chore. or silver bullet What are Microservices? Microservices are not a by any stretch of the imagination. I don’t believe there are any silver bullets when it comes to . If someone tells you there is, be skeptical. Basically, microservices do what they say on the tin. They’re small, isolated services that represent an equally small portion of your business domain. silver bullet software development Everybody has a different opinion on how a microservice should work and communicate but some beliefs are more common than others. To say that they’re simple would be a huge understatement. The concept of a microservice architecture is simple and ideal. The initial execution? Far from it. The diagram above explains the difference between a monolithic architecture and a simple microservice architecture. With a monolithic architecture, you have one large server responsible for handling all the requests. This is going to hit you at scale. It’s going to hit you hard. Microservices, however, can balance traffic due to your business’ needs. If you are receiving a large number of payments, you can scale up your payment service and keep the other services using a smaller number of resources. It’s at it’s finest. very, very horizontal scaling Microservices sound great, why are they hard? Microservices are very idealistic and you hear a relatively small number of large companies preaching the benefits. I feel as if a large number of smaller companies mistakingly take this information as gospel. These large companies such as , , , amongst others can execute microservices well because they are . They have the resources and time to invest into building this architecture. Uber Google AirBnb Square absolutely loaded When you start looking into Microservices, you enter a rabbit hole that’s absolutely massive. It makes Alice in Wonderland seem like a piercing. You quickly learn that you will need to manage all of these microservices and balance the load whilst monitoring everything that is happening. Oh, don’t forget that you’re going to need to test each service locally when you’re building new features. After a few weeks of research and testing, here is a list of responsibilities for each service and tools I would recommend researching small Containers ( ) Docker Orchestration ( ) Kubernetes Management ( ) Forge Api Gateway / Canary ( ) Ambassador Edge Proxy ( ) Envoy Monitoring ( ) Prometheus Local Testing ( ) Telepresence Looking at that list, it’s a lot to ingest for one simple service. However, being able to monitor interactions between services and orchestrate them is immensely powerful. There are some terms in there that might seem confusing. I’ll cover the most confusing ones. — An edge proxy is a process that runs alongside your service and proxies your services traffic through it’s own internal system and usually uses some kind of middleware (such as prometheus) to monitor and track this traffic. Ambassador is built on Envoy and provides more functionality. Edge Proxy — You’ve most likely heard of Docker by now. It’s the container standard on the web. It’s actually a play on the meaning of the word container. You can think of docker containers and small shipping containers that are running your application in the cloud with an orchestrator moving them around and optimizing them. They also use linux containers directly under the hood. Containers — Kubernetes is the most popular orchestrator and it works extremely well with Docker. It essentially takes your docker container and looks after them to make sure they’re all working in an efficient manner. Orchestration I’ve heard I can use any language for each micro service, is that correct? Sure**.** You can use any language for a microservice in the same manner that you can run a marathon in boots. It’s possible but it doesn’t mean it’s effective and you’re probably wasting your time. The key to creating a stable and scalable microservice architecture is to make sure that it’s maintainable and that it follows a general trend within your business. Your architecture should have solid foundations. If somebody was to work on your project, you should be able to say: Generally, our services are written in {main language}. They use the following frameworks: {list of frameworks}. If you think a service should use a different language, you will be responsible for ensuring it has the same functionality as our existing services. This keeps consistency within your projects. Sure, there are exceptions. If you had a service that was performing complex mathematics and was handling a lot of traffic, you might want to opt for a language more suited to that task. It won’t be a simple task, however. If you have dependencies in your PSL (primary service language), then it might be an investment of time to rebuild such in the new language. How do microservices communicate? One of the major concern of microservices in the past had been latency issues. Making a number of HTTPS requests between a number of services could cause a fairly noticeable delay to the client. This isn’t a feasible approach and it’s certainly not lightweight. A new approach was needed for microservices and along came our saviour: . gRPC is an RPC framework that is highly efficient and is designed with speed and efficiency in mind. Binary data is sent through the wire at high speeds, alongside rapid serialisation/deserialization. gRPC Using gRPC, you define your models using a definition, which is fairly simple and they have a protobuf compiler for many common languages. Using gRPC has calmed fears about latency. protobuf However, using these protobuf models comes with some confusion. We’ve been looking into this the past few weeks, wondering how different services can share the same protobuf models (for example, a user model). We’ve had ideas such as keeping the protobuf models in a versioned git repository but we’re not convinced. We’re currently investigating the best mechanism for this. Versioning. There are also issues with using gRPC. You will also likely have to maintain a standard REST API for standard web clients to access. We’re making progress in this area using envoy and . It’s still a complicated issue that doesn’t seem to get much coverage in the microservice world. grpc-gateway Migrating If you’re considering building a microservice architecture to begin with, I would not recommend it. You’ll spend way too much time doing premature optimizations for domains you don’t really know exist yet. To create a solid microservice architecture, you’ll need to know exactly why you’re doing it and nobody can really tell you that. However, if you’ve already got a monolith and you’re looking to migrate, here’s some advice. Identify your (bounded contexts) business domains Research Docker, Kubernetes and Envoy thoroughly Research monitoring and testing tools Spend a lot of time creating demos That’s our personal experience. We spend a lot of time in the Google Cloud Platform creating different architectures with new tools, discovering what works best for us. We are currently using the following stack: Primary Service Language: GoPrimary Service Datastore: PostgresqCloud Platform: Google CloudContainers: DockerContainer Orchestration: KubernetesEdge Proxy: EnvoyLocal Testing: TelepresenceMonitoring: PrometheusManagement: Forge It’s been an exciting number of months completing our migration. It’s been difficult. However, more recently we stumbled across a company that smashing the complicated entry barrier building a microservices architecture. They’ve created a number of tools to help you get started with microservices and through our limited experience, we’re enjoying using them. The team are also highly respectful, knowledgable and responsive. Give them some love at . Look at their products and read through their docs. You’ll come out knowing a lot more about Microservices. https://www.datawire.io I would love to know what you think and would appreciate your thoughts on this topic. If you could give us any advice from your experience, do let me know. I’d also be happy to answer any questions about our experience.

Microservices are hard — an invaluable guide to microservices.

K Nearest Neighbor Predictive Classification Javascript implementation Utilizing D3 Visualizations At its heart, the k Nearest Neighbor technique treats the problems it receives as issues of pattern recognition in order to properly classify data. It does this by ‘learning’ the graphical space in which the different categories reside; implicit is the assumption that each classes will be clustered distinctly. Computers learn similarly to living things, based on as much previous data as is available, predictions are made in order to do work more efficiently. Typically, has not been thought of as a good language for the implementation of algorithms, and for good reason. When first developed, the language was incredibly inefficient and a misunderstanding of underlying concepts or blatant disregard for best practices led to a proliferation of terrible libraries and examples. Further, the community had been leaning towards OOP, but there has slowly been a movement to a more functional style. Over the years optimizations have been made, libraries have been refactored or newly created in ways that lend to performing complex processing, the underlying language implementation has had several iterations, and computing machines are also equipped with much greater processing capabilities. Javascript machine learning All of this leads to an interesting space in which we, as Javascript developers, find ourselves. On the cusp of a rapidly transforming widespread technology, the possible applications of which are expanding, we have the opportunity to contribute to and improve a technology which is becoming increasingly fundamental to our daily experience. D3 has been one of the libraries most widely used with the goal of data visualization, and is depended upon several widespread visualization tools today; this is what will be used to plot out k Nearest Neighbors algorithm. If you need an introduction to the library before applying it to something more involved, an introducing it might of use. earlier post of mine Now that we’re all excited about our role in this field, lets explore a simple implementation of arguably the most fundamental machine learning algorithm, k Nearest Neighbor. While on the less complex side of predictive computation algorithms, it is widely used in the data analysis industry, and is one of the best performing for several problems, one such example is the prediction of behavior between proteins, or protein-protein interaction (PPI): “In spite of its simplicity to implement, when a large data set or numerous features are used the computational cost and memory requirement grows rapidly. This method has been used (not widely) in PPI prediction problem.” - , 2013 Zahiri, Bozorgmehr, Masoudi-Nejad To illustrate this, lets explore the example pictured to the left, for which can be cloned down and explored, or you can play with it in ! Note that the dummy data is dumb; as should be inferred, they are not from a valid scientific study nor do the predictions represent a medical diagnosis in reality. It is loosely based on the findings of , and related published research outlining the correlation. which indicates that high levels of the proteins CA15–3 and CA27–29 could indicate the presence of a tumor in the breast; it should also be noted that these tracers also get spiked in cases of ovarian, liver, and other breast diseases. Units are calculated in units per milliliter of blood, with levels over 30U/mL considered high-risk. The training data can be changed in the source code, and the graph will handle the new data by fitting it to the graph as long as it is in the correct format: {x: num, y: num, neighbors: array, identity: num}. the repo this CodePen this medical study To create this graph, there will be a series of functional tasks to accomplish, each of which my be solved differently, and modularity may be implemented in the functions to various degrees or with a different opinions with regards to the ideal separation of concerns. The following is my own opinion on how to best organize the program. The pieces we will address as individual problems in the construction of our larger program include: Elements on the page: A board Training data points Input data points Classification functions: Calibrate(smooth) data to fit to graph Find k nearest neighbors Classify each datum Placement functions: Place training data Place input data OPTIONAL: place circle containing input’s k nearest neighbors To create the board, simply located the element on the page to which you want to append: var = 300;var = 300;var = d3.select(document.getElementById(‘body’)).append(“svg”).attr(“width”, w).attr(“height”, h); h w svgSelection Don’t forget to add a border: var = svgSelection.append(‘rect’).attr(‘x’, 0).attr(‘y’, 0).attr(‘height’, h).attr(‘width’, w).style(“stroke”, ‘black’).style(“fill”, “none”).style(“stroke-width”, 4); border At the top of the js page, include the following variables: and are the dimensions of the board, is an object which holds the highest values for the each of the training variables, into , each data point will be pushed it so that future inputs includes training and input data, and the correspond with the various possible states of classification. h w max addData riskColors To figure out where to place the points on the graph, we need to perform several operations in order to fit the data’s range within the size of the graph. The helper functions and will be utilized in our placement functions. Lets build those out: findMax smooth var = (data) => {var results = {x: 0, y: 0};data.map(datum => {let x = datum.x;let y = datum.y;x > results.x ? results.x = x : false;y > results.y ? results.y = y : false;});return results;} findMax var = (datum, max) => {datum.x = (datum.x/max.x) * h;datum.y = (datum.y/max.y) * h;return datum;} smooth Next, we need a function that will draw the points of data on the graph: var = [];//global container for training and input datavar = [‘mediumturquoise’, ‘crimson’, ‘rebeccapurple’];//indices of riskColors correspond with a classification allData riskColors var = (datum, encircle) => {svgSelection.append('circle').attr('cx', datum.x).attr('cy', datum.y).attr('r', 5).attr('fill', riskColors[datum.identity])encircle ? drawHalo(datum) : false;//draw halo implemented below!}var ;var = (data, adjust=true, encircle=false) => {max === undefined ? max = findMax(data) : max = max;data.map(datum => {allData.push(datum);adjust ? smooth(datum, max) : null;placeDatum(datum, encircle);});} placeDatum max populateGraph will be used set to true for input data, invoking in the , which draws a circle around the data point containing its nearest neighbors. At this point, all that will be on the screen is a blank rectangle. All this needs now is a set of training data and to be invoked. At the bottom of the file, place: Encircle drawHalo placeDatum k //this may be replaced with any data in which there are//identified objects with the three properties found below var = [{x: 32,y: 40, identity: 1},{x: 5,y: 27, identity: 0},{x: 17,y: 9, identity: 0},{x: 50, y: 29, identity: 1},{x: 22,y: 46, identity: 1},{x: 10,y: 27, identity: 0},{x: 5,y: 9, identity: 0}]; trainingData populateGraph(trainingData); The dummy data should now render to the page, clustered and colored. This is still pretty boring, to make it interesting some interface for user interactivity needs to be created. In the body of the HTML, we’ll be adding a form with the id which contains input bars for each of the variables: submitData <div id=”submitData”><form onsubmit=”handleSubmit(event)”><b>Add data:</b> <br/>x:<input id=”x” type=”text” placeholder=”CA15–3 in U/mL”/><br/>y:<input id=”y” type=”text” placeholder=”CA27–29 in U/mL”><button>Submit</button></form></div> The handler should prevent the page from re-rendering, get the inputs from the page, correctly format them to an object to then be passed into a drawing function as a data point: var = function(e) {e.preventDefault();var x = document.getElementById(‘x’).value;var y = document.getElementById(‘y’).value;var = {x: x,y: y,neighbors: [],identity: 0};classifyAndPlot([datum]);//classify and plot not yet defineddocument.getElementById(‘x’).value = ‘’;document.getElementById(‘y’).value = ‘’;//clear the input bars} handleSubmit datum The formatted data then is passed into a function to categorize them based on the points closest to them, which is calculated by looping through all of the data points, calculating distance using good old Pythagoras’s formula. Lets all say it together: “a squared plus b squared equals c squared.” Good! Except this time, it’ll be (xDiff²+yDiff²): √ var = function(data, k) {return data.map(datum => {allData.map(neighbor => {var xDiff = Math.abs(datum.x — neighbor.x);var yDiff = Math.abs(datum.y — neighbor.y);var distance = Math.sqrt(Math.pow(xDiff, 2) + Math.pow(yDiff, 2));var currentFartherNeighbor = datum.neighbors.find(dataNeighbor => dataNeighbor[0] > distance)datum.neighbors.length < k ?datum.neighbors.push([distance, neighbor.identity]) :currentFartherNeighbor ?datum.neighbors.splice(datum.neighbors.indexOf(currentFartherNeighbor), 1, [distance, neighbor.identity]) :false;datum.neighbors.sort((a, b) => b[0] — a[0]);return datum;});});} findNeighbors The above functions loops over the input data, and compares each datum to each of the array, pushing into the neighbors array if there are less then neighbors or if a comparison yields a distance smaller than any currently in the array. In the case of the latter, the farthest neighbor also needs to be spliced out. Then the datum is returned with the neighbors array populated as expected. However, one more component is necessary in order to paint them on the page. We have functions to do the later which may be reused, but before passing data into them to render onto the page, we should consider what the input looks like when this function is invoked, and how to format the input to be successfully accepted by the function. allData k populateGraph var = (data) => {//fit the data to the graph rangedata.map(datum => smooth(datum, max));//populate neighborsfindNeighbors(data, 3);var formattedData = data.map(datum => {var neighborsIdentity = datum.neighbors.reduce((acc, neighbor) => {acc[neighbor[1]] === undefined ? acc[neighbor[1]] = 1 :acc[neighbor[1]] = acc[neighbor[1]] + 1;return acc;}, []);var maximum = Math.max(…neighborsIdentity.filter(neighbor => neighbor));datum.identity = neighborsIdentity.indexOf(maximum);return datum;});populateGraph(formattedData, false, true);} classifyAndPlot That’s it! We have everything we need to train our algorithm and predict unclassified input results based on present clustering trends. While it has been noted that this is not the most efficient solution to most machine learning problems, but is useful in cases where there are discrete categories with clear distinctions. As someone interested in medicine and public health, I’ve learned that cancer diagnosis is one of those cases, in which you are assigned either a positive or negative diagnosis based on various biometric measures related to understood oncological trends. This is the idea that the example data is modeled after, but it can be modified for any training data in which each datum has been classified. What are some ideas for application that come to mind for you? Let me know below in the comments. Thanks for reading, now go make your own improved implementations! Jacob

Associative Machine Learning: Predicting Cancer Risk

Image from placestoseeinyourlifetime At we’ve been fully remote for almost a year now, and it’s been great. Nanobox If you want to work remote, or already are but don’t feel like it’s effective, let me highlight the 3 tools we use that have helped us build a really great remote culture. While these aren’t the only tools we use, they are the ones that have the most impact on how successfully we’re able to work remote. Slack Image by xkcd Slack has become so ubiquitous that I’m sure it needs no introduction. But for that one person who hasn’t heard of it (or refuses to use it), I’ll give a quick overview why it’s a great fit for remote work. Channels Slack allows you to create multiple channels where topics can be discussed in isolation, and has a great way of creating personal or group direct messages. One thing I do wish, however, is that you could invite someone to an existing conversation without creating an entirely new conversation. Chat History Probably my favorite feature of Slack is the chat history. You don’t have to be logged in to be a part of a conversation, you can join at any time. Anyone coming from IRC will immediately know what I’m talking about. Clients and Integrations Slack has really well designed apps for various devices to help you stay connected when you’re not at your computer. It also integrates with a bunch of other services we use like Groove, Pingdom, and even lets you use built in robots (or ) to get extra functionality. build your own Misc. Slack also has a bunch of tiny little things that are just nice to have: back-tic highlighting code/text snippets really great support for embedded content (images, videos, etc). emoji (:party_parrot:) A word of warning …all the pieces and people together so that you can actually get things done. That statement from Slacks own website is somewhat debatable. With all the pieces and people together… things can get a little noisy. Especially if your team has a lot of channels, or if you’re a member of multiple teams. Make sure to update your to ensure you’re not distracted by too much conversation. Notification Settings Discord I think at some point we had tried close to a dozen different voice clients and each one of them had their own set of issues. Configuring and managing servers, disconnects, terrible voice quality, the list goes on. While discord touts itself as an all-in-one voice and text chat client for , in my opinion, it is currently the best voice client out there for . Period. gamers anything Rather than configure and manage your own server, you simply create one via the client and invite people to join. Similar to Slack you can create and configure “rooms” for specific needs. And the most important thing is the voice quality is . outstanding If you forced me find one bad thing to say about Discord, it would be that there is no tray indicator when you speak. So you might not know that you have a hot mic until you sneeze and everyone yells at you. Google Hangouts We initially tried using Google Hangouts for everything. The video and voice quality was just not great. And for whatever reason they decided to get rid of chat… The screen sharing however, is great! Whenever we need to pair program, or do code reviews we’ll just hop on there and share screens. Sometimes we’ll even get up to eight people in a screen share at one time, with three different screens being shared for various different meetings. Just remember… Make sure when using Google Hangouts to mute your mic and shutoff your camera (the latter not being critical). If you keep your mic on and you’re using Discord you’ll get some pretty bad feedback/echo. as Go… Live your dream Hopefully I’ve given you a good place to start if you’re really considering working remote. I’ve been doing it for almost a year now, and I love it. On a side note, if you use some other tools that have really helped you, or your team, I’d love to hear about them!

Stories, News & Mentions about PLACESDOT

Placesdot’s Stories on HackerNoon