Outlogicis in the Geographic Information Systems Industry

Outlogic

outlogic

The 6 Types of Code Refactoring That Every Programmer Should Know

Since the Supreme Court’s reversal of Roe v. Wade in June, location data firms, , and have been facing increased scrutiny from Congress for trading in sensitive user data that could indicate when someone is seeking an abortion. app developers online abortion pill providers The Markup has found that location data company INRIX, which collects, analyzes, and sells aggregated vehicle, traffic, and parking data includes Planned Parenthood clinics in its . INRIX IQ Location Analytics dashboard Using a free trial of the product, The Markup was able to locate at least 71 Planned Parenthood clinics in dozens of states. INRIX is a major, established mobility data company that has been in business for 17 years. It lists the U.S. Federal Highway Administration, Microsoft, the Los Angeles Department of Transportation, BMW, Zillow, and IBM among its clients. The Markup’s findings follow recent reports by Motherboard identifying two other location data firms— and —that also included Planned Parenthood visitor data in their similar dashboard products. Both companies have since removed that data. Safegraph Placer.AI Mark Daymond, the CTO and data protection officer at INRIX told The Markup in an email, “INRIX receives anonymized data and further de-identifies it then aggregates that data. Identities of individuals are irrelevant for our business.” The free version of INRIX’s Location Analytics dashboard listed only the address, hours, and average annual daily traffic counts on nearby streets for each clinic, but the paid version shows more detailed statistics for sample points of interests in its database, including demographic and ethnic breakdowns of visitors, visitor counts by hour and day, aggregated heat maps of the origins and destinations for visitors, and drive times to and from the business location. The Planned Parenthood locations remained in the INRIX dashboard after we contacted the company, and were still there hours before publication of this story. Planned Parenthood did not immediately respond to a request for comment. Daymond said the approach INRIX takes to limit detail on the maps protects users from being identified. “Location Analytics only displays results down to the census block group level. We source data from a variety of map providers whose information is commercially available,” he said. With the legal landscape changing rapidly since Dobbs v. Jackson triggered more restrictive abortion laws, people seeking the procedure face increased risks to their privacy, and many technology companies have been racing to respond. Anti-abortion activists have used tools built by the location data industry, , to dissuade abortion-seeking patients with targeted ads. Now, the stakes are higher as nine states have criminalized the procedure. such as geofencing Scrutiny from Congress Some Democratic lawmakers are demanding action. Following reports by Motherboard, 14 Democratic senators to location data companies Safegraph and Placer.AI seeking details about their data collection and asking them to cease including abortion clinics. sent letters and have removed the clinics and other sensitive locations from their databases following the Motherboard reports. On July 1, Google when a user visits an abortion provider, fertility center, or other sensitive location. Safegraph Placer.AI pledged to erase visitor location data Last week, the location data companies Placer.AI, Safegraph, Babel Street, X-Mode (Outlogic), Gravy Analytics, and five reproductive health app developers and , the sources of their location data, and all data partnerships as part of an investigation into abortion and reproductive health patient data privacy. House Oversight Committee contacted asked them to disclose their data collection practices “This is a great example of how comprehensive privacy laws need to be put in place that restrict the collection of that kind of data that can be weaponized against users by fingerprinting, identifying them, and selling that sort of behavioral data or identifying data to anyone that wants it, law enforcement or otherwise,” said Daly Barnett, a staff technologist at the Electronic Frontier Foundation. Location Analytics Dashboards INRIX, a privately held company based in Kirkland, Wash., offers parking, traffic, and aggregated location data to transportation agencies, car manufacturers, and software developers. In a brochure for its “Vehicle Trips” product, the company boasts that its data “ ” and 36 billion “real-time data points” each day, with updates as frequent as every three seconds. captures over 150 million anonymous trips Location data companies like INRIX, Placer.AI, and Safegraph offer similar products: an easy-to-use dashboard where users can quickly analyze foot and vehicle traffic to millions of retail businesses, highlighting patterns in visits with charts and maps and tools to compare competing businesses. One of the sample locations The Markup was able to explore was a chicken sandwich restaurant in New Jersey. Such data is extremely valuable for city planners, retail businesses, and commercial real estate and private equity firms. The data powering these aggregated insights comes from multiple sources, , connected vehicles, traffic sensors, and data purchased from other brokers. including mobile phone apps While tracking the aggregate number of people visiting a chicken sandwich shop seems innocent enough, when people are seeking an abortion, the privacy risks change drastically. Some location data companies have recognized the risks of including sensitive locations like doctors’ offices and houses of worship. For example, location intelligence firm Cuebiq publishes a public “ ” policy defining the types of locations it excludes to protect the privacy of the users in its datasets. Sensitive Points of Interest Data Partners INRIX’s states that it collects personal data from third-party sources, including “[d]ata brokers from which we purchase data to supplement the data we collect,” though it does not disclose the identity of any of its partners. privacy policy The data that powers the dashboard we used contains clues to the source of INRIX’s points of interest (POI) data. We observed the autocomplete function on INRIX’s dashboard to the API for location data company HERE, which provides location results for a boundary specified by the map coordinates. making a call There also appears to be a HERE-provided unique ID for each location in the data. HERE spokesperson Kasey Farrar told The Markup in an email that the company “… maintains robust ‘privacy by design’ data practices with strict terms and conditions for data usage by customers. HERE does not collect origin-destination data of individuals.” Farrar noted the company’s “ ” which details some of the specific measures taken to protect user privacy. privacy charter, The business review site Yelp also makes an appearance in the POI data. Yelp is listed as the “supplier” within the data powering INRIX’s dashboard. Yelp spokesperson Julianne Rowe said that the company does not have a partnership with INRIX and that businesses “can obtain Yelp content and data through our free API or from an approved partner.” She noted that the business location, hours, and contact information are all publicly available, including on Planned Parenthood’s website. There are several categories assigned to the Planned Parenthood locations we found on INRIX. Categories included “Medical Services/Clinics,” “Hospital or Health Care Facility,” and “Consumer Services.” While most of the data we could see on INRIX’s dashboard appeared to be aggregated, risks do remain for users. Many companies in the highlight that user privacy is protected, due to the fact that they only sell aggregated data, such as a certain number of people visiting a particular business during a specific week. multibillion-dollar location data industry Daymond said INRIX does not sell data that is tied to individual users. “INRIX is not a data broker…. We don’t store personal identifiers that can be tied back to an individual to help ensure no personal data is discovered.” On the industry-wide use of aggregated data to protect user privacy, EFF’s Barnett said that even aggregated data carries risks for user privacy, and individuals . can still be identified in some circumstances INRIX’s Daymond said, “Our terms of use explicitly prohibit our customers from using our products in a way that violates laws and regulations.” While business location and visitor traffic may seem innocuous, Barnett said, the context matters. “These seemingly … observed-in-a-vacuum, harmless datasets don’t exist in vacuums. They can be snowballed together too.” Written By Jon Keegan Also published here Photo by on Alina Grubnyak Unsplash

Your Planned Parenthood Data Is Being Collected, Analyzed, and Sold

আপনার পরিকল্পিত পিতামাতার ডেটা সংগ্রহ করা, বিশ্লেষণ করা এবং বিক্রি করা হচ্ছে

Ihre Daten zur geplanten Elternschaft werden gesammelt, analysiert und verkauft

Sus datos de Planned Parenthood están siendo recopilados, analizados y vendidos

Vos données sur la planification familiale sont collectées, analysées et vendues

आपका नियोजित पितृत्व डेटा एकत्र, विश्लेषण और बेचा जा रहा है

あなたの計画された親子関係のデータが収集され、分析され、販売されています

귀하의 가족 계획 데이터가 수집, 분석 및 판매되고 있습니다.

Seus dados de paternidade planejada estão sendo coletados, analisados e vendidos

Planlı Ebeveynlik Verileriniz Toplanıyor, Analiz Ediliyor ve Satılıyor

Dữ liệu Kế hoạch làm cha mẹ của bạn đang được thu thập, phân tích và bán

您的计划生育数据正在被收集、分析和出售

This article will show how I built a functional static web app with user authentication and a dynamic database for FREE using Webflow, Vault Vision, Cyclic, and Airtable with LowCode Over the past few weeks, I've spoken to many technology founders starting out with an idea for a software business or service that requires interaction with their customers over the web. One was a service that provides charities a way to publish donation requests, and another was a service that connected delivery drivers to nearby public bathrooms. I discovered a significant obstacle awaiting these founders when they make the transition from a simple static brochure-style website to creating a SaaS application. The transition to a modern SaaS app that needs individual user authentication and controlled access to records typically required a development effort that was out of reach of budgets, thus these entrepreneurs were forced to use email submission forms or online spreadsheets that did not sufficiently meet their auth needs and were cumbersome. Like any founder, I knew there had to be a better way;  with all the new SaaS LowCode tools that offer a FREE plan, surely there is a lowcode and user-friendly solution that doesn’t require hiring a developer to build a custom 3-tier application for thousands of dollars. To build a fully functional and secure authentication system into SaaS applications, we settled on the following requirements: low cost or FREE #LowCode, has to be copy/paste-able and quick to self-edit without relying on expensive  developers All managed infrastructure. Servers, networks, systems, and disks have to be someone else's problem.  Once I get it working at noon, it better stays that way through the night, and the job of some other unsung sysadmin if it doesn't App users need to be able to login and logout App users need to be able to add/update/see their records and data (and ONLY their data) Needs to have some pre-built easy admin interfaces for service owners themselves to manage and have access to ALL the data. Needs to be point-and-click, copy-and-paste usable for standard features, with an option to have advanced customizable features available with small bits of code. A modern, secure and reasonable solution is a 3-tier app and how can we make it simple? Front End (easy to make simple with web builders like Wix, Webflow, GoDaddy) Middle Tier (hmm, this is the struggle, most current options will violate one of the requirements, specifically needing custom development) Data Tier (easy to make simple with Airtable) Let's break down what we need the Middle Tier to actually do. Hold an API Key and keep it secure to access Airtable, we can have this in the Front End because that is public, so it has to be in the Middle Tier Handle user auth to issue sessions and access tokens to securely identify and authenticate users for both the Front End and the Data Tier Luckily there are auth services like Vault Vision (disclaimer: the author of this fine article is affiliated with Vault Vision. Please check us out :) ) https://vaultvision.com So Vault Vision can handle the user management and auth with a static website just fine, but there still needs to be a component to store the data API key and to tie the user authentication to the data such that user access to records is controlled and restricted to just the users that should have access.  This is where things get interesting and to sum up the problem: 'Can we make a generic, re-usable, middle tier component that does not need "on-going" dev that can hold an API key, validate user authentication tokens and enforce data access constraints, and NOT require developers or system admins to manage and deploy, and can be FREE?' YES!! It took a bit of inspired determination but where there is a will there is a 'Node-Auth-Airtable' way open-sourced by Vault Vision, designed to run as an Express API serverless App on the Cyclic application platform that does offer a FREE plan. Here is the GitHub link to the Vault Vision ' ', which you can easily copy/clone to run on Cyclic. Generic Node Auth Airtable Bridge This solution might not be for everyone, but let's go over the pros and cons of this solution vs a standard 3-tier app. Pros FREE, can not overstate how awesome this is for the fledging founder, it is a great way to test out business ideas and service concepts with workable betas. #LowCode, can be self-installed and self-edited with a minimal amount of tech knowledge Modular, most of the services can be provided by many vendors Expandable with simple javascript, complex customization can be added with minimal effort Can scale with smaller piece-meal replacements Quick to deploy and setup can be done in under 1 hour Support available through discord and forums Data structures and admin interfaces are simple and excel like Low complexity, no abstractions, very understandable Cons Specifically built for Airtable's data API Esoteric, not a common architecture pattern DELETEs are not really supported, can be done, but there is no 'recycle bin' if a user wipes their own data Scalability is limited before upsizing components and infrastructure customization will be needed Complex data structures, ACID transactions, and foreign key constraints aren't available Components are tightly coupled, and the Front End does need to match data formats to the Data Tier interfaces. Is this right for you? If you are looking for a fast and easy way to make a low-cost dynamic web app that users can create and view their own records, all in simple data structures, then YES! Great use cases are: A school that wants to publish grades for students Voting or polling application Ticket/work request application Expense report submission So on to the nitty gritty details, below are the 11 steps needed to try this out and to get a working example on your own accounts and websites. Details of the Steps (Warning, many screenshots) You are going to create 4 accounts and keep track of the following 8 values of id, domains, and URLs, you should keep track of these values in a notepad window for easy reference you will copy and paste them frequently. API key (from Airtable) baseid (from Airtable) table (from Airtable) website URL (from Webflow) bridge connector URL/domain (from Cyclic) tenant domain (site where your login page will be shown) (from Vault Vision) client_id (from Vault Vision) client_secret (from Vault Vision) Step 1 Create the Airtable, and make note of these 3 values from the following 2 screens: APIKEY baseid table When you create the table; make sure your table has a single text column called vv_id (THIS IS VERY IMPORTANT) Visit: https://airtable.com Get the API key . here Step 2 Create your static WebHost on Webflow, make note of: Front End website URL Create a website on Webflow and start with a blank site Visit: https://webflow.com/dashboard/sites/new Once created, you can get the website URL here ![Note the website url - in this case: https://demo-site-6a158e.webflow.io ](https://cdn.hackernoon.com/images/AOlJRvEucFTafhEGDZ1zCOoPUXa2-81d3l51.png) **Caveat** For the little bit of js code that we will be using you will need to use the paid version of Webflow OR there are 2 other options, 1) stick to static HTML buttons and HTML forms that you post to your bridge connector and move some of the logic there, OR 2) you can host the static assets on GitHub pages and create a repo named {your_github_username}.github.io and it will served out of the same https://{your_github_username}.github.io endpoint https://pages.github.com/ Don't worry about the content for now, we will add that later.  For now, we just need to get the website URL. Step 3 Create your user authentication bridge connector on Cyclic, make note of: connector domain name Create a GitHub account if you don't have one already, Cyclic uses that account for auth and for publishing Visit: https://github.com/ Fork this repo into your account: https://github.com/vaultvision/node-auth-airtable Create a Cyclic account and authorize it to access GitHub Visit: https://www.cyclic.sh/ You will need to give the Cyclic account access to your public GitHub repos. Select ‘node-auth-airtable’ assuming you forked it from https://github.com/vaultvision/node-auth-airtable After you select the right repo, Cyclic will ask for final approval to install. Make note of the bridge connector domain name on this screen ![Note the bridge connector URL - in this case: https://adventurous-jade-chimpanzee.cyclic.app ](https://cdn.hackernoon.com/images/AOlJRvEucFTafhEGDZ1zCOoPUXa2-mri3lnz.png) Step 4 Create your Vault Vision authentication tenant and application Visit: https://manage.vaultvision.com/register Pick a name for your login page, this will be the domain/URL for page where your users login and signup Fill in a name for your company, it can be anything you want Make sure to choose 'Traditional Web App' for the Application Type. And Pick a name for your application, it can be anything it is never shown to a user, and fill in these values for your application: The Callback URL must end with /oidc/callback The Logout URL must end with /oidc/logout The Login URL is just your homepage URL Callback URL ------------------- https://adventurous-jade-chimpanzee.cyclic.app/oidc/callback Login URL ------------------- https://demo-site-6a158e.webflow.io (if you are on Webflow) https://vaultvisiondemo.github.io (if you are on GitHub Pages) Logout URL ------------------- https://adventurous-jade-chimpanzee.cyclic.app/oidc/logout Mmake note of: auth tenant domain (site where your login page will be shown) client_id client_secret Step 5 Go back to the Cyclic application, and update these variables (under the ‘Variables’ tab in the menu) Visit: https://app.cyclic.sh/#/app/ Step 6 Go back to Webflow or your GitHub Page and copy and paste the html fragment and script fragments and update these 3 values in the javascript script tag: const authhostdomain const baseid const table You can get the HTML and script fragment from here: https://github.com/vaultvisiondemo/vaultvisiondemo.github.io/blob/main/index.html There are 3 sections, an HTML block, a style block and a script block In the script block there are 3 const that need to  be updated with your unique values {these are the values for this demo, but you will need to use your unique values} const authhostdomain="adventurous-jade-chimpanzee.cyclic.app"; const baseid = "app8Bj4fSoM2Hv3s9"; const table = "demo-table"; const authhostdomain -> provided by Cyclic, it is the domain of your bridge connector (NOT full URL, there should not be https:// added here) const baseid -> provided by Airtable, the baseid of your 'base' at Airtable const table -> provided by Airtable, the name you chose for your table All the configuration and setup is now done!! Time to try it out. Step 7 Create a new account from your front-end (make sure you are not in Incognitio mode, or if you are you enable 3rd party cookies ) https://knowledge.kaltura.com/help/enabling-3rd-party-cookies-for-chrome-incognito-mode Visit: {whatever your front end url is - in our case it was GitHub pages https://vaultvisiondemo.github.io} Step 8 Add some new data content The "add Air Table" button will add some static data to your Airtable and automatically attach the correct vv_id (user_id) for the user that you are logged in as. Step 9 List the data content The "list Air Table" button will read out all the records that are attached to the correct vv_id (user_id) for the current user that is logged in.  You can logout, create a new user and repeat the same process and you will see that each users records are dynamic and only accessible by that user.  One user cannot access another users records. Step 10 Airtable Admin sees all Back in Airtable you can new see and modify ALL records as the Airtable Admin, and you can see the individual records now attached to each user. Step 11 User management, you can also manage these users with blocking/deleteing/credential reseting in the User management panel. Voila!!! Even though there are several components to this solution, they are all understandable, single purpose based, #LowCode and have some FREE options yet give you dynamic user-based features of full application with the convivence of Webflow and #LowCode building.

How I Built a Static Website with User Authentication and Dynamic Database for Free

Logic as the Essence of Philosophy

On this day, , also known as the , , , and , traditional Catholics following the traditional liturgical calendar in the Roman Rite, ponder the Gospel of St. John (here taken from the 1945 St. Andrew Missal): Second Sunday of Easter Octave Day of Easter White Sunday Quasimodo Sunday Low Sunday “Now Thomas, one of the 12, who is called Didymus was not with them when Jesus came. The other disciple therefore said to him: We have seen the Lord. But he said to them: Except I shall see in His hands the print of the nails, and put my finger into the place of the nails, and put my hand into His side, I will not believe. And after eight days, again His disciples were within, and Thomas with them. Jesus cometh, the doors being shut, and stood in the midst, and said: Peace be to you. Then He saith to Thomas: Put in thy finger hither thy hand, and put it into my side; and be not faithless, but believing. Thomas answered and said to Him: My Lord and my God. Jesus saith to him: Because thou has seen Me, Thomas, thou hast believed: blessed are they that have not seen, and have believed.” It’s an interesting grounding for a critique on academia, the world of the peer-review, what we call science, industry, and now it seems even the everyday. I’ve subjected myself to what academe calls ‘rigor’ in order to get published. But what exactly is this ‘rigor’? A simple google search for a definition yields, “the quality of being thorough, exhaustive, or accurate”, “severity or strictness”, “demanding, difficult, or extreme conditions”. In the world of the scientific peer-review, your work is evaluated by the editor of an academic journal, who decides whether your paper will be sent out to be critiqued (via a blind review) by three or four other academics. Your paper is sent back to the editor, who evaluates the critique and ultimately decides whether the journal will proceed to accept your paper for publication (oft subject to revisions) or not. This is essentially what rigor is—one is subjected to ‘strict’ rules of ones peers, who consider one another ‘experts’ in evaluating and critiquing a piece of work. The system is inductive, where general principles (also decisions, impressions, and critiques) are derived from specific observations. Just from an everyday common-sense perspective, one should see the highly subjective nature of the process. At the same time, from a logical perspective, it is a fallacy—because it assumes everyone in the whole must have the same characteristics as the parts. It should prove disturbing to anyone how the industry of knowledge isn’t instead premised on reason; and that science as we know it isn’t primarily logical or reasonable. What is logical is empirical. What isn't logical can’t be empirical. A prime test of reality and truth is whether it passes the logic test, where human folly, impressions, and biases can’t figure into determining whether something is or isn’t. Inductive ‘logic’, however, isn’t confined to academia. A majority rules or peer system governs both board rooms and labor unions. They used to call this sophistry. And as this global health issue has shown, the survey has become the primary arbiter of decisions and choices. Indeed, numbers are part and parcel of Math and arithmetic is logic; but haven’t the numbers been robbed of the essence of logic itself? Have not the graphs, charts, percentage points, various quantities cited from and by ‘expert’ upon ‘expert' culled so ‘rigorously’, which have come to be the plaque and emblem of trust been stripped of the essence of logic down to mere consensus, impression, and rule of the majority, or in some cases the majority of the elite? But rigor is not reason; and what isn’t logical can’t be empirical. Christ was/ is Logos Incarnate; and the civilization, which built what we have today (from science to architecture, engineering, law, education, and commerce) was (is) a Catholic Christian one, which is inevitably premised on this idea of ‘Logos Incarnate’. It should be paradoxically both refreshing and daunting to all (whether Catholic or not) that the very basis of civilization and its consequences came off this--'truth' for Catholics, 'concept' for non-Catholics. And that we should turn our world around (upside down, really) and premise it on inductive (vs. deductive logic) is a defiance of both reason and sense. This is why experience has been touted the best teacher, which contradicts Logos Incarnate and ‘what is logical is empirical’. Truth is, experience isn’t the default and inherent best teacher. It is the inefficient teacher. Logic oft acquired through much research and reading is the best teacher. Indeed, it does require practice (thus, experience); but the ‘experience’ touted today is not this type experience but an immersion with what is referred to as real-world situations (which is good in itself). Problem is, if this fondness and bias for ‘experience’ comes at the expense of logic. What if one is 'learning' in an environment that is anti-logic? Logic has become so difficult for many and in fact, most. Thus, while ‘what is logical is empirical’, the pain of achieving and maintaining logic (true rigor in the acquisition of habits of the mind towards consistency in thought and fidelity to the flow of these thoughts) is not an appealing proposition. Acting before thinking has trumped the more prudent prudence. In reality, however, ‘experience’ is an inefficient and slow teacher as one has to go through various events of pain, oft stretching an entire lifetime to learn and gain. You don’t have to jump off a cliff or touch fire to know how that’s going to feel or turn out. Logic saves you the trouble. But we’ve come to prefer the more circuitous and long route. This is not an affront on practice, praxis, and experience. I do not discount the necessity. I am merely arguing for the superiority of deductive logic. In fact, the affront on logic might be more serious. Let’s take the case of what the invention of bitcoin and its blockchain has enabled—the superiority of code; in other words, logic. Verification (on so public a ledger) vs. ‘trust’ (especially in comparison to legal tender known as ‘fiat’) is certainly an interesting and wonderful proposition. Anyone who has followed and looked into the history of bitcoin, knows while it is a product of human design (the anonymous Satoshi Nakamoto), it's human design that came as a consequence of central planning and the manipulation of ‘money’. One can argue, it was thus, organic and natural. The last straw seemed to have been the 2008 financial crisis. Moreover, this product of human design, which remains to have been created by someone who is either dead or prefers to be permanently invisible, is open source (thus, anyone can work on it), public (anyone and everyone can use and see its activities and transactions), and is built on economic incentives for its growth and survival, has made it practically fool-proof for any sort of compromise and infiltration. Its very foundation, grass roots development, and tokenomics (as the people in the space like to call it), ensure its fidelity to logic not just in its code (which non-coders can’t even understand) but its function and utility as a system and store of value. It’s a real case for ‘what is logical is empirical’. But while bitcoin and other cryptocurrencies and blockchains are based on code (math and logic), ‘what is logical is empirical’ isn’t yet quite clear with other blockchains and alt-coins. Qualitatively, when translated to experience and culture, the consequences and effects can be driven by anti-logic. This is most apparent in the space of non-fungible tokens (NFTs). I sometimes wonder if writers of code are based on logic or Cartesian ‘logic’—which is logical within its realm (and why Descartes lived in his head and the logic he invented was never and can never be empirical)—giving birth to bubbles as the norm and foundation of civilization and society. We might just be devolving from mere Doubting Thomases who need to see in order to believe to disciples of ‘rigor’ (rule by the majority, elite, or both) who don’t need to see because there are others (‘experts’ and ‘chosen ones’), who will do the job of seeing to an almost complete withdrawal from reality, choosing to be in a bubble of hype, hysteria, emotion, and noise now tokenized with possible value or ‘value’. The verdict is still out on this one though. I, myself dabble in NFTs. But I prefer to guard and ensure my inclination to logic and thus remain critical by default. It also helps to do so in determining, which are Cartesian (bubbles) and which are not. Ironically, it has become necessary to become a Doubting Thomas these days of what looks like a third level withdrawal from logic and doubt. Going back to Thomas though, and to be true to ‘what is logical is empirical’, to err is human (the stain of original sin has remained). His folly might just have been a case for this as well as a case against the premium we’ve placed on inductive logic as the backbone of civilization. St. Thomas, however, almost immediately acknowledged his folly, perhaps being first to directly acknowledge and proclaim the Divinity of Christ, with “My Lord and my God”. He went on to create converts to Christianity, preaching from Babylon to Persia and finally to Malabar, India (present-day Madras), where he was reportedly thrown into a pit and pierced with a Brahmin’s spear. There are, in fact, St. Thomas Christians (Mar Thomas Nazranis or Syrian Christians). Indeed, it might seem, our current more circuitous and inductive route of rigor vs. logic, might just be going through a test. Question is, is it a unicorn (bubble), empiricism, or will logic be incarnate?

Perils of Induction: Doubting Thomases, Rigor, NFTs & Descartes

Who is Life360 Selling Your Data to?

By: Jon Keegan and Alfred Ng A huge but little-known industry has cropped up around monetizing people’s movements. Companies that you likely have never heard of are hawking access to the location history on your mobile phone. An estimated , the location data industry has many players: collectors, aggregators, marketplaces, and location intelligence firms, all of which boast about the scale and precision of the data that they’ve amassed. $12 billion market Location firm Near describes itself as “The World's Largest Dataset of People's Behavior in the Real-World,” with data representing “1.6B people across 44 countries.” Mobilewalla boasts “40+ Countries, 1.9B+ Devices, 50B Mobile Signals Daily, 5+ Years of Data.” X-Mode’s website claims its data covers “25%+ of the Adult U.S. population monthly.” In an effort to shed light on this little-monitored industry, The Markup has identified 47 companies that harvest, sell, or trade in mobile phone location data. While hardly comprehensive, the list begins to paint a picture of the interconnected players that do everything from providing code to app developers to monetize user data to offering analytics from “ ” and on hundreds of millions of people. Six companies claimed more than a billion devices in their data, and at least four claimed their data was the “most accurate” in the industry. 1.9 billion devices access to datasets “There isn't a lot of transparency and there is a really, really complex shadowy web of interactions between these companies that’s hard to untangle,” Justin Sherman, a cyber policy fellow at the Duke Tech Policy Lab, said. “They operate on the fact that the general public and people in Washington and other regulatory centers aren't paying attention to what they're doing.” Occasionally, stories illuminate just how invasive this industry can be. In 2020, Motherboard reported that a company that collects location data through apps, was collecting data from Muslim prayer apps and selling it to military contractors. X-Mode, The Wall Street Journal also reported in 2020 that for immigration enforcement. Venntel, a location data provider, was selling location data to federal agencies A Catholic news outlet also </a> to out a priest who had frequented gay bars, though it’s still unknown what company sold that information. used location data from a data vendor Many firms promise that privacy is at the center of their businesses and that they’re careful to never sell information that can be traced back to a person. But researchers studying anonymized location data just how misleading that claim can be. have shown The truth is, it’s hard to know all the ways in which your movements are being tracked and traded. Companies often reveal little about what apps serve as the sources of data they collect, what exactly that data consists of, and how far it travels. To piece together a picture of the ecosystem, The Markup reviewed the websites and marketing language of each of the 47 companies we identified as operating in the location data industry, as well as any information they revealed about how the data got to them. . See our methodology here How the Data Leaves Your Phone Most times, the location data pipeline starts off in your hands, when an app sends a notification asking for permission to access your location data. Apps have all kinds of reasons for using your location. Map apps need to know where you are in order to give you directions to where you’re going. A weather, waves, or wind app checks your location to give you relevant meteorological information. A video streaming app checks where you are to ensure you’re in a country where it’s licensed to stream certain shows. But unbeknownst to most users, some of those apps sell or share location data about their users with companies that analyze the data and sell their insights, like Advan Research. Other companies, like Adsquare, buy or obtain location data from apps for the purpose of aggregating it with other data sources. Companies like real estate firms, hedge funds and retail businesses might then turn and use the data for their own advertising, analytics, investment strategy, or marketing purposes. Serge Egelman, a researcher at UC Berkeley’s ​​International Computer Science Institute and CTO of AppCensus, who has , said it’s hard to tell which apps on your phone simply use the data for their own functional purposes and which ones release your data into the economic ether. researched sensitive data permissions on mobile apps “When the app asks for location, in the moment, because maybe you click the button to find stuff near you and you get a permission dialog, you might reasonably infer that ‘Oh, that's to service that request to provide that functionality,’ but there's no guarantee of that,” Egelman said. “And there's certainly usually never a disclosure that says that the data is going to be limited to that purpose.” Companies that trade in this data are reluctant to share which apps they get data from. The Markup asked spokespeople from . all the companies on our list</a> where they get the location data they obtain Companies like Adsquare and Cuebiq told The Markup that they don’t publicly disclose what apps they get location data from to keep a competitive advantage but maintained that their process of obtaining location data was transparent and with clear consent from app users. “It is all extremely transparent,” said Bill Daddi, a spokesperson for Cuebiq. He added that consumers must know what the apps are doing with their data because so few consent to share it. “The opt-in rates clearly confirm that the users are fully aware of what is happening because the opt-in rates can be as low as less than 20%, depending on the app,” Daddi said in an email. Yiannis Tsiounis, the CEO of the location analytics firm Advan Research, said his company buys from location data aggregators, who collect the data from thousands of apps—but would not say which ones. Tsiounis said the apps he works with do explicitly say that they share location data with third parties somewhere in the privacy policies, though he acknowledged that most people don’t read privacy policies. “There’s only so much you can squeeze into the notification message. You get one line, right? So you can’t say all of that in the notification message,” Tsiounis said. “You only get to explain to the user, ‘I need your location data for X, Y, and Z.’ What you have to do is, there has to be a link to the privacy policy.” Only one company spokesperson, Foursquare’s Ashley Dawkins, actually named any specific apps—Foursquare's own products, like Swarm, CityGuide, and Rewards—as sources for its location data trove. But Foursquare also produces a free software development kit (SDK)—a set of prebuilt tools developers can use in their own apps—that can potentially track location through any app that uses it. is used in apps like , a service that compares prices at nearby gas stations, , a shopping app for coupons, and , another location-based discount app. Foursquare’s Pilgrim SDK GasBuddy Flipp Checkout 51 GasBuddy, Flipp, and Checkout 51 didn’t respond to requests for comment. A search on l a site that analyzes and tracks SDKs in apps, found Foursquare’s Pilgrim SDK in 26 Android apps. Mighty Signa While not every app with Foursquare’s SDK sends location data back to the company, the privacy policies for Flipp, Checkout 51, and GasBuddy all disclose that they share location data with the company. Foursquare’s method of obtaining location data through an embedded SDK is a common practice. Of the 47 companies that The Markup identified, 12 of them advertised SDKs to app developers that could send them location data in exchange for money or services. i says in its marketing that it does foot traffic analysis and that its SDK is installed in more than 500 apps and has insights on more than 20 million devices. Placer.a “We partner with mobile apps providing location services and receive anonymized aggregated data. Very critically, all data is anonymized and stripped of personal identifiers before it reaches us,” Ethan Chernofsky, ’s vice president of marketing, said in an email. Placer.ai Into the Location Data Marketplace Once a person’s location data has been collected from an app and it has entered the location data marketplace, it can be sold over and over again, from the data providers to an aggregator that resells data from multiple sources. It could end up in the hands of a “location intelligence” firm that uses the raw data to analyze foot traffic for retail shopping areas and the demographics associated with its visitors. Or with a hedge fund that wants insights on how many people are going to a certain store. “There are the data aggregators that collect the data from multiple applications and sell in bulk. And then there are analytics companies which buy data either from aggregators or from applications and perform the analytics,” said Tsiounis of Advan Research. “And everybody sells to everybody else.” Some data marketplaces are part of well-known companies, like or , which sell all types of data, not just location data. Oracle boasts its listing as the “world’s largest third-party data marketplace” for targeted advertising, while Amazon claims to “make it easy to find, subscribe to, and use third-party data in the cloud.” Both marketplaces feature listings for several of the location data companies that we examined. Amazon’s AWS Data Exchange Oracle’s Data Marketplace Amazon spokesperson Claude Shy said that data providers have to explain how they gain consent for data and how they monitor people using the data they purchase. “Only qualified data providers will have access to the AWS Data Exchange. Potential data providers are put through a rigorous application process,” Shy said. Oracle declined to comment. Other companies, like Narrative, say they are simply connecting data buyers and sellers by providing a platform. lists location data providers like SafeGraph and Complementics among its 17 providers with more than two billion mobile advertising IDs to buy from. Narrative’s website, for instance But Narrative CEO Nick Jordan said the company doesn’t even look at the data itself. “There’s a number of companies that are using our platform to acquire and/or monetize geolocation data, but we actually don’t have any rights to the data,” he said. “We’re not buying it, we’re not selling it.” To give a sense of how massive the industry is, Amass Insights has 320 location data providers listed on its directory, Jordan Hauer, the company’s CEO, said. While the company doesn’t directly collect or sell any of the data, hedge funds will pay it to guide them through the myriad of location data companies, he said. “The most inefficient part of the whole process is actually not delivering the data,” Hauer said. “It’s actually finding what you’re looking for and making sure that it’s compliant, making sure that it has value and that it is exactly what the provider says it is.” Oh, the Places Your Data Will Go There are a whole slew of potential buyers for location data: investors looking for intel on market trends or what their competitors are up to, political campaigns, stores keeping tabs on customers, and law enforcement agencies, among others. Data from location intelligence firm Thasos Group has been used to measure the l . number of workers pulling extra shifts</a> at Tesla plants. Political campaigns on both sides of the aisle have also used ocation data from people who were at rallies</a> for targeted advertising Fast food restaurants and other businesses have been known to buy location data for advertising purposes down to a person’s steps. For example, in 2018, Burger King ran a promotion in which, if a customer’s phone was within , the Burger King app would let the user buy a Whopper for one cent. 600 feet of a McDonalds The Wall Street Journal and Motherboard have also written extensively about how federal agencies including the , , and bought location data from companies tracking phones. Internal Revenue Service Customs and Border Protection the U.S. military Of the location data firms The Markup examined, the offerings are diverse. Advan Research, for instance, uses historical location data to tell its customers, largely retail businesses or their private equity firm owners, where their visitors came from, and makes guesses about their income, race, and interests based on where they’ve been. “For example, we know that the average income in this neighborhood by census data is $50,000. But then there are two devices—one went to Dollar General, McDonald’s, and Walmart, and the other went to a BMW dealer and Tiffany’s ... so they probably make more money,” Advan Research’s Tsiounis said. Others combine the location data they obtain with other pieces of data gathered from your online activities. Complementics, which boasts data on “more than a billion mobile device IDs,” offers for mobile ad targeting. location data in tandem with cross-device data The prices can be steep. Outlogic (formerly known as X-Mode) offers a license for a titled “Cyber Security Location data” on Datarade for $240,000 per year. The listing says “Outlogic’s accurate and granular location data is collected directly from a mobile device’s GPS.” location dataset At the moment, there are few if any rules limiting who can buy your data. Sherman, of the Duke Tech Policy Lab, </a> finding that data brokers were advertising location information on people based on their political beliefs, as well as data on U.S. government employees and military personnel. published a report in August “There is virtually nothing in U.S. law preventing an American company from selling data on two million service members, let's say, to some Russian company that's just a front for the Russian government,” Sherman said. Existing privacy laws in the U.S., like California’s Consumer Privacy Act, do not limit who can purchase data, though California residents can request that their data not be “sold”—which can be n. Instead, the law focuses on allowing people to opt out of sharing their location in the first place. a tricky definitio The European Union’s General Data Protection Regulation has stricter requirements for when their data is being processed or transferred. notifying users But Ashkan Soltani, a privacy expert and former chief technologist for the Federal Trade Commission, said it’s unrealistic to expect customers to hunt down companies and insist they delete their personal data. “We know in practice that consumers don't take action,” he said. “It's incredibly taxing to opt out of hundreds of data brokers you've never even heard of." Companies like Apple and Google, who control access to the app stores, are in the best position to control the location data market, AppCensus’s Egelman said. “The real danger is the app gets booted from the Google Play store or the iOS app store,” he said.” As a result, your company loses money.” </a> from using location reporting SDKs from several data companies. Google and Apple both recently banned app developers , however, that the companies’ SDKs were still making their way into Google’s app store.  Apple didn’t respond to a request for comment. Researchers found "The Google Play team is always working to strengthen privacy protections through both product and policy improvements. When we find apps or SDK providers that violate our policies, we take action,” Google spokesperson Scott Westover said in an email. Digital privacy has been a key policy issue for U.S. senator Ron Wyden, a Democrat from Oregon, who told The Markup that the big app stores needed to do more. “This is the right move by Google, but they and Apple need to do more than play whack-a-mole with apps that sell Americans’ location information. These companies need a real plan to protect users’ privacy and safety from these malicious apps,” Wyden said in an email. This article was license. originally published on The Markup</a> and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives

Companies Pay Big Money to Have the Location Data From Your Phone

Recently I made a that requires user accounts. I learned quite a bit about setting up authentication with Firebase on the client-side and using it on the server-side to protected API routes with a middleware pattern similar to Express.js. This post is a recap of what I learned based on this project for future reference. You can find the code for this project on GitHub . small web app here Authentication - Client Side Initialization Setting up Firebase is easy. You create a project and enable the sign-in providers you plan to use, along with authorized domains. Grab the credentials from Project Settings in the Firebase console, and we can initialize the Firebase SDK on the client-side like this. here firebase ; ; ; clientCredentials = { : process.env.NEXT_PUBLIC_FIREBASE_API_KEY, : process.env.NEXT_PUBLIC_FIREBASE_AUTH_DOMAIN, : process.env.NEXT_PUBLIC_FIREBASE_DATABASE_URL, : process.env.NEXT_PUBLIC_FIREBASE_PROJECT_ID, : process.env.NEXT_PUBLIC_FIREBASE_APP_ID,
}; (!firebase.apps.length) {
  firebase.initializeApp(clientCredentials);
} firebase //lib/firebase.js import from 'firebase/app' import 'firebase/auth' import 'firebase/firestore' const apiKey authDomain databaseURL projectId appId if export default (See file and folder structure in the actual project) here React Hooks and Context Provider Since the user's authentication status is a "global" state, we can avoid recursively passing it as a prop through many layers of components by using . Context To do this, we need a context Provider and a context Consumer. A Provider comes with a Context created by createContext(). The value prop we pass to the Provider will be accessible by its children. authContext = createContext(); { auth = ; //lib/auth.js const export ( ) function AuthProvider { children } const /* something we'll fill in later */ return {children} ;
} < = > authContext.Provider value {auth} </ > authContext.Provider For the descendant components to use the value, i.e., consume the Context, we can use Context.Consumer, or more conveniently, the useContext . hook useAuth = { useContext(authContext);
}; SomeComponent = { { user, loading } = useAuth(); } //lib/auth.js export const => () return //components/SomeComponent.js const => () const // later we can use the object user to determine authentication status // ... In Next.js, the AuthProvider we implemented above can be inserted in the _app.js so all the pages in the app can use it. See . here Implementation Details of `AuthProvider` In the AuthProvider skeleton above, we passed an auth object as the value prop, and this is the key thing that all the consumers consume. Now we need to figure out what we need to implement this auth object. The key thing auth need to achieve is subscribing to the changes in the user's login status (and associated user info). These changes can be triggered through the Firebase SDK, specifically the sign-in / sign-out functions such as firebase.auth.GoogleAuthProvider() and authentication state observer function firebase.auth().onAuthStateChanged(). So, our minimal implementation could be the following, mainly pay attention to the new getAuth function. We definitely need to return something from getAuth and that'll be the auth object used by AuthProvider. To do this, we implement the handleUser function to update the state user as follows React, { useState, useEffect, useContext, createContext } firebase authContext = createContext() { auth = getAuth() //lib/auth.js import from 'react' import from './firebase' const export ( ) function AuthProvider { children } const return {children} }

export const useAuth = () => {
  return useContext(authContext)
}

function getAuth() {
  const [user, setUser] = useState(null)
  const handleUser = (user) => {
    if(user){
      setUser(user)
    }
  }

  useEffect(() => {
    const unsubscribe = firebase.auth().onAuthStateChanged(handleUser);
    return () => unsubscribe();
  }, []);
  
  /* TBA: some log in and log out function that will also call handleUser */
  
  return {user}
} < = > authContext.Provider value {auth} </ > authContext.Provider Since we are calling other React Hooks, e.g. userEffect,  getAuth needs to be either a React functional component or a custom hook in order to follow the . Since we are not rendering anything, just returning some info,  getAuth is a custom hook and we should thus rename it to something like useFirebaseAuth (i.e the custom hook's name should always start with use, per ). The main function userFirebaseAuth provides us is to share the user state between components. Actually, across all the components since we used a Context Provider in _app.js. rules here note here Below is a fuller implementation of userFirebaseAuth. There are quite a few things we added here: Exposing sign-in and sign-out logic so that context consumers can use them. Since they would trigger changes in user state similarly to firebase.auth().onAuthStateChanged, it is better to put them here. We actually need to change firebase.auth().onAuthStateChanged to firebase.auth().onIdTokenChanged to capture the token refresh events and refresh the user state accordingly with the new access token. Adding some formatting to make the user object only contains our app's necessary info and not everything that Firebase returns. Add redirect to send user to the right pages after sign-in or sign-out. React, { useState, useEffect, useContext, createContext } ; Router ; firebase ; { createUser } ; authContext = createContext(); { auth = useFirebaseAuth(); import from 'react' import from 'next/router' import from './firebase' import from './db' const export ( ) function AuthProvider { children } const return {children} ;
}

export const useAuth = () => {
  return useContext(authContext);
};

function useFirebaseAuth() {
  const [user, setUser] = useState(null);
  const [loading, setLoading] = useState(true);

  const handleUser = async (rawUser) => {
    if (rawUser) {
      const user = await formatUser(rawUser);
      const { token, ...userWithoutToken } = user;

      createUser(user.uid, userWithoutToken);
      setUser(user);

      setLoading(false);
      return user;
    } else {
      setUser(false);
      setLoading(false);
      return false;
    }
  };
  
  const signinWithGoogle = (redirect) => {
    setLoading(true);
    return firebase
      .auth()
      .signInWithPopup(new firebase.auth.GoogleAuthProvider())
      .then((response) => {
        handleUser(response.user);

        if (redirect) {
          Router.push(redirect);
        }
      });
  };

  const signout = () => {
    return firebase
      .auth()
      .signOut()
      .then(() => handleUser(false));
  };

  useEffect(() => {
    const unsubscribe = firebase.auth().onIdTokenChanged(handleUser);
    return () => unsubscribe();
  }, []);

  return {
    user,
    loading,
    signinWithGoogle,
    signout,
  };
}

const formatUser = async (user) => {
  return {
    uid: user.uid,
    email: user.email,
    name: user.displayName,
    provider: user.providerData[0].providerId,
    photoUrl: user.photoURL,
  };
}; < = > authContext.Provider value {auth} </ > authContext.Provider Authorization - Server Side The other use case we need with Firebase authentication is to ensure users have proper access to server-side resources, i.e., specific API routes will be only accessible if certain access criteria is met. I guess this called authorization. An example would be, for /api/users/[uid] route, we would only return results the user requesting their own info. Firestore Security Rules One pattern to manage access to backend resources (mostly database access) is to use Firestore and Firebase authentication together and use Firestore's to enforce access permissions. security rules For example, in the example above, to limit access to user info, on the client-side, we attempt to retrieve the user record as usual { doc = firestore.collection( ).doc(uid).get(); user = { : doc.id, ...doc.data() }; user;
} export async ( ) function getUser uid const await 'users' const id return But we define the following set of security rules to only allow read/write when the user's uid matches the document's uid. rules_version = ;
service cloud.firestore {
  match /databases/{database}/documents {
    match /users/{uid} {
      allow read, : isUser(uid);
    }
  }
} { isSignedIn() && request.auth.uid == uid;
} { request.auth.uid != ;
} '2' write if ( ) function isUser uid return ( ) function isSignedIn return null You can actually do a lot with this setup. For example, in order to determine access to a document, you can do some extra queries on other collections and documents. Here are the I used, which involved a bit of that. security rules With this client-side setup and security rules, there are downsides. Mainly: We are defining access using this security rule syntax, which is less flexible than just writing arbitrary code on the server-side. Firestore also limits the number of queries you can do to verify the access permission on each request. This may limit how complex your permission scheme can be. Some of the database operations can be very heavy, such as recursively deleting a large document collection, and should only be done on the server-side. (See Firestore's documentation for more details.) here Testing security rules require extra work. (Firebase does have a friendly UI and simulator for this). Finally, it gets a little scattered that some database access logic lives on the client-side ( ) and some on the server-side ( ). I probably should consolidate to the server-side. code pointer code pointer Using Firebase Admin on Server Side OK, now the more "classic" way of doing the authorization on the server-side. The general workflow is: The client-side code should send over an access token along with each request. The server-side code has an instance of firebase-admin, which can verify and decode the access token and extract user information, such as the uid of the user Based on that information, the server-side code can do more queries and apply more logic to figure out it should proceed or reject the request. (The firebase-admin will have privileged access to all Firebase resources and will ignore all the security rules, which are only relevant for client-side requests). This is how I initialized firebase-admin * admin ; (!admin.apps.length) {
  admin.initializeApp({ : admin.credential.cert({ : process.env.NEXT_PUBLIC_FIREBASE_PROJECT_ID, : process.env.FIREBASE_CLIENT_EMAIL, : process.env.FIREBASE_PRIVATE_KEY.replace( , ),
    }), : process.env.NEXT_PUBLIC_FIREBASE_DATABASE_URL,
  });
} firestore = admin.firestore(); auth = admin.auth(); { firestore, auth } //lib/firebase-admin.js import as from 'firebase-admin' if credential projectId clientEmail privateKey /\\n/g '\n' databaseURL const const export The documentation suggests generate a private key JSON file. The file contains many different fields, the three fields above: projectId, clientEmail, and privateKey seem to be enough to get it to work. here Now we can extract uid on each request and verify the user's access { auth } ; (req, res) => { (!req.headers.token) { res.status( ).json({ : });
  } { { uid } = auth.verifyIdToken(req.headers.token);
    req.uid = uid;
  } (error) { res.status( ).json({ : error.message });
  } } import from '@/lib/firebase-admin' export default async if return 401 error 'Please include id token' try const await catch return 401 error // more authorization checks based on uid // business logic Authentication Middleware for Next.js API Routes One small annoyance with the above is that as we have more API routes that need authentication, the code need to be repeated in these API routes functions. I find Next.js out of the box doesn't have as strong a support for server-side development. A couple of things from Express.js I wish Next.js have are: routers and . middleware In this scenario, making authentication work as a middleware would be convenient. Middleware is things you can plug into the request handling lifecycle; the middleware would enrich the request and/or the response objects and can terminate the request early if errors occur. It turned out to be pretty straightforward, we just need to create a wrapper for our normal handler function, and in the wrapper we can modify the req and res objects and return early if errors occur. Here is how I defined a withAuth middleware { auth } ; { (req, res) => { authHeader = req.headers.authorization; (!authHeader) { res.status( ).end( );
    } token = authHeader.split( )[ ]; decodedToken; {
      decodedToken = auth.verifyIdToken(token); (!decodedToken || !decodedToken.uid) res.status( ).end( );
      req.uid = decodedToken.uid;
    } (error) { .log(error.errorInfo); errorCode = error.errorInfo.code;
      error.status = ; (errorCode === ) {
        error.status = ;
      } res.status(error.status).json({ : errorCode });
    } handler(req, res);
  };
} import from '@/lib/firebase-admin' export ( ) function withAuth handler return async const if return 401 'Not authenticated. No Auth header' const ' ' 1 let try await if return 401 'Not authenticated' catch console const 401 if 'auth/internal-error' 500 //TODO handlle firebase admin errors in more detail return error return And this is how we can use it, notice instead of exporting handler we are exporting withAuth(handler) { withAuth } ; { getUserSites } ; handler = (req, res) => { { { sites } = getUserSites(req.uid); res.status( ).json({ sites });
  } (error) { .log(error); res.status( ).json({ : error.message });
  }
}; withAuth(handler); // get all sites of a user import from '@/lib/middlewares' import from '@/lib/db-admin' const async try const await return 200 catch console return 500 error export default Here are the relevant files on GitHub: and . middleware.js sites route That's all I learned about authentication on the client and server side with Next.js and Firebase. Overall it's a great developer experience and pretty painless to figure things out. Previously published at https://www.dingran.me/next-js-authentication/

Next.js: Firebase Authentication and Middleware for API Routes

Zeet is sponsoring this blogpost for the next month. I tried it out the other day. It's like serverless but for running entire back ends. You can host and scale apps automagically. Pretty neat. If you’re like me, a sucker for event-driven programming, you’ll want to continue reading. Today we’ll take a look at triggering functions from messages. I’ve covered a few interesting topics regarding serverless architectures and AWS already, but nothing like this yet. Dig down, and get ready. Let’s go. AWS Lambda AWS SNS TL;DR What are we building? What is AWS SNS? Build the API with the Serverless Framework Deploy the API to AWS Lambda Test the API with Dashbird Note : All the code from this tutorial is already on GitHub if you want to check out the end result right away. What are we building? Our focus will solely be on the steps to create the infrastructure components our app will need. The code itself will just mimic the behavior of a random complex computation. I’ve chosen a recursive function that calculates the factorial of the number passed to it. Here’s a nice diagram, because diagrams are awesome of course! The function is the only exposed function, which is hooked up to . It takes a single parameter which it validates, upon success, it publishes an SNS topic and sends along the value. init API Gateway number number The SNS topic will trigger a second function called . This function will perform the calculation and log out the result to the console. This mimics a heavy computational background task such as data processing, image manipulation or machine learning calculations. calculate If the function fails, the Dead Letter Queue SNS topic will receive a message and trigger the function. calculate error Every function invoked will twice retry its execution upon failure. Using the Dead Letter Queue as a pool for your error logs is a smart use-case. asynchronously Now you’re wondering, why all the complication with SNS instead of just invoking the second lambda function from the first one with ? Lambda’s invoke API First of all, it’s a huge anti-pattern for asynchronous workflows, which is our case. Otherwise, it’s fine if you need the response from the second lambda function right away. Another issue is hitting Lambda’s concurrency limits pretty fast. It can result in losing invocations and dropping data. Sending your data through a pub/sub service like SNS or a queue like will make sure you have data integrity. SQS Makes sense now? Sweet, let’s talk a bit more about SNS. What is AWS SNS? Before we start coding we need to cover the basics. We know what AWS Lambda is, but what about SNS? The AWS docs are pretty straightforward. Amazon Simple Notification Service (SNS) is a flexible, fully managed pub/sub messaging and mobile notifications service for coordinating the delivery of messages to subscribing endpoints and clients. - AWS Docs In English, this means it’s a way of sending notifications between services on a publisher/subscriber basis. One service publishes some data about a and sends it along its way. SNS will then funnel it to all the subscribers of that particular . Key focus is on here, you’ll see why a bit further down. topic topic topic Build the API with the Serverless Framework The first thing to do, as always, is to set up the project and install dependencies. 1. Install the Serverless Framework My go-to development and deployment tool for serverless apps is the . Let’s go ahead and install it. Serverless Framework $ npm i -g serverless Note: If you’re using Linux, you may need to run the command as sudo. Once installed globally on your machine, the commands will be available to you from wherever in the terminal. But for it to communicate with your AWS account you need to configure an IAM User. Jump over , then come back and run the command below, with the provided keys. here for the explanation $ serverless config credentials \ 
    --provider aws \ 
    --key xxxxxxxxxxxxxx \ 
    --secret xxxxxxxxxxxxxx Now your Serverless installation knows what account to connect to when you run any terminal command. Let’s jump in and see it in action. 2. Create a service Create a new directory to house your Serverless application services. Fire up a terminal in there. Now you’re ready to create a new service. What’s a service? It’s like a project. It’s where you define AWS Lambda functions, the events that trigger them and any AWS infrastructure resources they require, including SNS which we’ll add today, all in a file called . serverless.yml Back in your terminal type: $ serverless create \
    --template aws-nodejs \
    --path lambda-sns-dlq-error-handling The create command will create a new . What a surprise! We also pick a runtime for the function. This is called the . Passing in will set the runtime to Node.js. Just what we want. The will create a folder for the service. service template aws-nodejs path 3. Explore the service directory with a code editor Open up the folder with your favorite code editor. There should be three files in there, but for now, we’ll only focus on the . It contains all the configuration settings for this service. Here you specify both general configuration settings and per function settings. Your will be full of boilerplate code and comments. Feel free to delete it all and paste this in. lambda-sns-dlq-error-handling serverless.yml serverless.yml service: lambda-sns-dlq-error-handling

plugins:
  - serverless-pseudo-parameters

provider:
  name: aws
  runtime: nodejs8 stage: dev
  region: eu-central memorySize: environment:
    accountId: region: iamRoleStatements:
    - Effect: Resource: Action:
        - functions:
  init:
    handler: init.handler
    events:
      - http:
          path: init
          method: post
          cors: calculate:
    handler: calculate.handler
    events:
      - sns: calculate-topic # created immediately
    onError: arn:aws:sns:#{ ::Region}:#{ ::AccountId}:dlq-topic
  error:
    handler: error.handler
    events:
      - sns: dlq-topic # created immediately .10 -1 128 '#{AWS::AccountId}' '#{AWS::Region}' "Allow" "*" "sns:*" true AWS AWS Let’s break down what’s going one here. Check out the section. There are three functions here. From top to bottom they're , , and . The init function will get triggered by a simple HTTP request, which we invoke through API Gateway. Familiar territory for us. functions init calculate error However, the and functions are triggered by SNS topics. Meaning we will have logic in the function that will messages to a topic named while the function is to the same topic. calculate error init publish calculate-topic calculate subscribed Moving on, the function is subscribed to the while the function will publish messages to this topic if it fails, as you can see with the property. Now stuff makes sense, right? error dlq-topic calculate onError Make a mental note to yourself, once you add the SNS topics as events for your functions, the resources will automatically be created once you deploy the service. What else, take a look at the , they specify that our functions have permission to trigger, and get invoked by SNS topics. While the plugin lets us reference our and with the CloudFormation syntax, making it much easier to keep our SNS ARNs consistent across all resources. iamRoleStatements serverless-pseudo-parameters AccountId Region 4. Install dependencies Luckily, this part will be short. Just one package to install. First, initialize npm and then you can install . serverless-pseudo-parameters $ npm init -y && npm i serverless-pseudo-parameters That’ll do. 5. Write business logic With all things considered, the configuration process was rather simple. The code we’ll write now is just as straightforward. Nothing extraordinary to see, I’m sorry to disappoint. Let’s keep all three functions in separate files, to keep it simple. First of all create an file and paste this snippet in. init.js aws = ( ) sns = aws.SNS({ : }) { .log(payload) { : code, : .stringify(payload)
  }
} { .error(err) generateResponse(code, { : err.message
  })
} { params = { : .stringify(data), : } sns.publish(params).promise()
} .exports.handler = (event) => { data = .parse(event.body) ( data.number !== ) { generateError( , ( ))
  } { metadata = publishSnsTopic(data) generateResponse( , { : , : metadata
    })
  } (err) { generateError( , ( ))
  }
} // init.js const require 'aws-sdk' const new region 'eu-central-1' ( ) function generateResponse code, payload console return statusCode body JSON ( ) function generateError code, err console return message async ( ) function publishSnsTopic data const Message JSON TopicArn `arn:aws:sns: : :calculate-topic` ${process.env.region} ${process.env.accountId} return module async const JSON if typeof 'number' return 400 new Error 'Invalid number.' try const await return 200 message 'Successfully added the calculation.' data catch return 500 new Error 'Couldn\'t add the calculation due to an internal error.' We have a few helper functions and the exported lambda function at the bottom. What’s going on here? The lambda validates input and publishes some data to the SNS topic. That's everything this function is doing. The SNS topic will trigger the lambda function. Let's add that now. calculate-topic calculate-topic calculate Create a file and name it . Paste this snippet in. calculate.js .exports.handler = (event) => { { number } = .parse(event.Records[ ].Sns.Message) factorial = x === ? : x * factorial(x - ) result = factorial(number) .log( ) result
} // calculate.js module async const JSON 0 const ( ) => x 0 1 1 const console `The factorial of is .` ${number} ${result} return As you see this is just a simple factorial calculation implemented with a recursive function. It’ll calculate the factorial of the number we published to the SNS topic from the function. init An important note here is that if the function fails a total of three times, it'll publish messages to the Dead Letter Queue SNS topic we specified with the property in the file. The Dead Letter Queue will then trigger the function. Let's create it now so it can log out errors to CloudWatch. Create an file, and paste these lines in. calculate onError serverless.yml error error.js .exports.handler = (event) => { .error(event)
} // error.js module async console For now, this will do. However, ideally, you would have structured logging with detailed info about everything going on. That’s a topic for another article. Deploy the API to AWS Lambda Here comes the easy part. Deploying the API is as simple as running one command. $ serverless deploy You can see the endpoint get logged to the console. That’s where you will be sending your requests. Test the API with Dashbird The simplest way of testing an API is with CURL. Let’s create a simple CURL command and send a JSON payload to our endpoint. $ curl -H \
    -d \
    https: "Content-Type: application/json" '{"number":1000}' //<id>.execute-api.eu-central-1.amazonaws.com/dev/init If everything works like it should, the result of the calculation will be logged to CloudWatch. If not, well then you’re out of luck. In cases like these, I default to using to debug what’s going on. It’s to set up. Dashbird free and doesn’t require a credit card After hitting the endpoint a few times with a couple of different values here’s the result. The function works as expected. init But, what really interests us is the function. Here's what that looks like when it's successful. calculate When it fails it’ll specify a crash and show the error logs. After two retries it’ll send a message to the Dead Letter Queue and trigger the error function. Sweet! We’ve tested all the different scenarios. Hope this clears things up a bit. Wrapping up That’s all folks. We’ve covered creating SNS triggers for Lambda while also implementing a Dead Letter Queue to catch errors from failed invocations. Using serverless for various intermittent calculations is a valid use-case that will only grow in popularity in the future. There are no servers you need to worry about, and you only pay for the time it runs. Just deploy the code and rest assured it’ll work. If something breaks, you have watching your back, alerting you in Slack or E-mail if something is wrong. You just have to love Slack integration! Dashbird Again, here’s the , if you want to take a look at the code. It can act as a starter for your own use-cases where you need SNS messages triggering lambda functions. Give it a star if you like it and want more people to see it on GitHub. GitHub repo If you want to read some of my previous serverless musings head over to or my profile join my newsletter! Or, take a look at a few of my articles right away: A crash course on serverless-side rendering with Vue.js, Nuxt.js and AWS Lambda Building a serverless contact form with AWS Lambda and AWS SES A crash course on Serverless APIs with Express and MongoDB Solving invisible scaling issues with Serverless and MongoDB How to deploy a Node.js application to AWS Lambda using Serverless Getting started with AWS Lambda and Node.js A crash course on securing Serverless APIs with JSON web tokens Migrating your Node.js REST API to Serverless Building a Serverless REST API with Node.js and MongoDB A crash course on Serverless with Node.js Hope you guys and girls enjoyed reading this as much as I enjoyed writing it. If you liked it, slap that tiny clap so more people here on Medium will see this tutorial. Until next time, be curious and have fun. Originally published at dev.to .

A crash course on Serverless with AWS — Triggering Lambda with SNS Messaging

ZKP can be used for much more than just privacy. ZKPs are so versatile that it may redefine how blockchains work. There are many technical blog posts about Zero Knowledge Proofs (ZKP). I recently wrote one myself comparing . I noted there's a lack of non-technical posts on the use-cases of ZKP. the new general purpose zk-SNARKs Part 1: Succinct blockchain, from gigabytes to kilobytes Blockchains can be big and continuously grow in size every block. That’s by design. We have come to accept that this is the way it is. However, Coda’s recent testnet is different. First of all, Coda’s blockchain is fixed size. It doesn’t grow. Secondly, it’s only 22 kilobytes (!) in size. It would easily fit in a 1980s Commodore 64 or ZX Spectrum. Yet Coda is as secure, or arguably even more secure than traditional blockchains. And more projects are following: Mir and Starling (I’m part of Starling) will launch similar but more versatile ‘succinct blockchains’ soon. How does it work? Anyone who has ever spun up a blockchain node knows the pain: It takes hours or even days to sync a node. Blockchains are often so large that disk space and bandwidth requirements are beyond what most people have at home. This is leading to centralization. Even a popular blockchain like Ethereum consists of only about 10,000 nodes. Most of which are hosted on AWS and the nodes are owned by just a handful of entities. Blockchain isn’t as decentralized as many people think. Why does syncing a blockchain take so long? There are two reasons. The first reason is obvious: It takes a while to download hundreds of gigabytes of data or more. Secondly, the blockchain needs to be validated after its downloaded, because a malicious node could have sent you incorrect data. To validate a blockchain, it has to be replayed from the genesis block: execute the first transaction and confirm that the calculated state is equal to the downloaded state. Move to the next transaction until you’ve checked every transaction in the blockchain. It’s time consuming and wasteful; tens of thousands of nodes have done exactly the same computations before you. The reason this is necessary is because in traditional computing, the only way to know if a computation has been computed correctly, is to redo the computation. That’s fine for small computations, but it’s not great for slow calculations like replaying a blockchain. ZKP to improve efficiency and bandwidth It turns out there is a way to cheaply validate a computed result without redoing the computing: Zero knowledge proofs (ZKP), of which zk-SNARKs are probably the best known ones. How would it work? We’d have to rewrite the blockchain replay function as a zk-SNARK. The zk-SNARK will output two things: the original output (just like the original replay function) and a small mathematical proof that the result was calculated correctly. The proof can be as small as 200 bytes (yes, that’s less than a kilobyte). There is no need for all (or even multiple) computers to run the replay function. One computer can create the proof, an unlimited number of computers can validate at any time they see fit. Validation takes only a few milliseconds, no matter how long the original computation took (even hours, days, or years, it just doesn’t matter). The proof can be distributed online, via a USB stick or even printed on a t-shirt for that matter. If a malicious node changes a balance, the proof will not match the result and any validator will reject the state. If a malicious node changes the zk-SNARK code, the result will be rejected too. (There is a third parameter, a publicly shared string that ties the proof also to the zk-SNARK code. If the code is changed, the proof and shared string won’t be matched and a validator will reject the result.) We’ve eliminated the need to redo expensive computations, and there’s no need to download the blockchain anymore (as we already have the mathematical proof the blockchain exists and is valid). You only need the current state (e.g. the last block) plus a tiny proof that the current state is part of a valid blockchain and spend a few milliseconds to validate the result. Recursive composition It’s fast to validate a proof, but how about creating the proof? It turns out it’s not constant time, and it’s a lot less efficient in terms of computing and memory compared to traditional computing. In fact, while a zk-SNARK version of a replay function may sound good, it’s not a great solution in practice. It will be huge in terms of memory and even slower than the original non-zk-SNARK replay function. But there is another elegant solution. With a few tricks it turns out it’s possible to use recursive zk-SNARKs. With recursion, we don’t have to validate the blockchain from scratch, but we can build on top of the previous state. That’s much faster. Note that recursive zk-SNARKs are less efficient than non-recursive zh-snarks, but recent zk-SNARK constructs have made huge strides. A recursive zk-SNARK program uses the previous state, the proof belonging to the previous state and the new transactions as inputs. It validates the previous state (using the provided proof) and checks if the transactions in the new state are valid. If so, it will output the new state and a proof. Once the new state and proof are distributed to the network, all nodes can simply discard the previous state without any negative consequences at all. New nodes only need to download the latest state and proof. This is how Coda, Mir, and Starling are able to have a tiny fixed-sized blockchain. In our last example, only one node would create a new block and the proof. Obviously it isn’t necessarily that the same single node produces every block. For example, a node can be randomly chosen out of many nodes (with Verifiable Random Functions nodes can even randomly choose themselves without cheating). We could do even better. We can divide the block producing logic into multiple zk-SNARKs. The end result is block producers do not need the full blockchain, it only needs the previous state. How much smaller would such a solution be? A regular Coda node only needs 22 KB to store the proof, the current state, and the Merkle path to one balance. With 22 KB, a node can validate the complete blockchain, query the balance, and create transactions. But to produce blocks, a node needs more: it would need the full balances Merkle tree of the previous state. The Merkle tree size is dependent on the number of wallets. If Coda would have as many wallets as Ethereum, a Coda block producer will still only need about 1 GB. The smallest full node on Ethereum is (as of December 2019) 230 GB. A huge difference. This way, the network has many more active nodes, increasing decentralization, and opens up many new possibilities of programs interacting with the blockchain without solutions like Infura or Metamask. And given that , that could have a huge impact. 99% of new users drop off before installing Metamask : Daniel Lubarov (Mir), Shane Vitarana, and Stan van de Burgt. Thanks for proofreading We’re hiring! If you’re a Rust developer who’s interested in learning zk-SNARKs (but not necessarily have experience with zk-SNARKS): Starling Protocol is hiring. Starling is a programmable succinct layer one protocol based in Berkeley, CA. Starling Protocol is a diverse and inclusive workplace. We encourage people of color, LGBTQ individuals, and women to apply. Email for more information. ronald@starlingprotocol.com ( ) Disclaimer: The author is a Founder at Starling Protocol

ZKPs or Zero Knowledge Proofs Are Evolving Blockchains [Explained in non-technical terms]

This summer, I set off to build a productivity-based browser and ended up building a Chrome extension. Here’s how the process yielded a product that, while smaller than I initially wanted it to be, set me on the best path possible for future success: But first, a little background information. I have been coding since I was ten years old, and haven’t stopped since. I attended multiple bootcamps and academies, learning as many new concepts as I could. A year ago, after ending a long stint of failed projects and multiple camps attended, I decided that I wanted to take my skills and create a full-scaled project aimed towards a subject matter that I had been adamant about: human-computer interaction(or HCI). From my point of view, the way that we currently interact with computers, especially desktops, is too multi-layered and outdated, given how many resources and new outlets of technology are available. In a society that is so heavily dependent on computers, designers and developers alike owe it to users to create interfaces that are powerful and easy to use. In addition, interfaces should not interfere with the user’s life in the physical world, nor should they draw users away from the natural world. Computers should be an efficient and ever-lasting aid instead of a distraction or a threat. Take my MacBook for example: Let’s quickly examine this interface, run and controlled by MacOS software. My main workspace is my browser, in which I have multiple tabs open. The main functionality of the MacOS desktop is a set of windows, each controlled by a different app. For some people, there are already enough readily-accessible functionalities to get certain tasks done. For example, someone who doesn’t use a laptop much might find this layout ok if they only use their MacBook to view photos and listen to music. However, more and more people are starting to live their lives user interfaces. Everything from medical records to project research to communication with family members to tax forms live within one’s “tech ecosystem”. Although many users rely so heavily on these user interfaces, it is becoming difficult to manage where data is coming from and how data should be interpreted. In addition, we shouldn’t forget that technology is a tool — we need to make sure that certain tasks, such as browsing the web to research something or listening to music, can continually be an easy task that can be performed through as many mediums as possible. through For example, let’s observe how interact with computers. I use my laptop to do schoolwork, work on projects, stay in touch with my family, edit designs, read emails, and generally browse the web. I use my phone to view information at-a-glance, read the news, and listen to music.Whenever I want to do something(especially on my laptop), I have to dictate that action and I have to be precise in where/how I would like it to be executed. If I, say, want to begin researching an artist I had just stumbled upon on Spotify, I not only have to integrate this action into a space that holds other, non-related actions(e.g. a browser with multiple tabs), but I have to dictate what I’m looking for. I precisely In essence, the modern human-computer interface is lacking intelligence and ease-of use. So, you might be wondering. As I mentioned earlier, I started workshopping this idea about a year ago. It took me a while to get to the point where I felt comfortable enough with the idea that I could start formulating an actual product. What does this all have to do with a Chrome extension? At the beginning of the summer, I set out to take all of these ideas that I had workshopped and create a new browser using Electron. The idea itself very closely follow the description from earlier, but I ran into many technical challenges while beginning: Databases As I started developing this project, I was thinking about which database I wanted to choose and what security features should be integral to the product. I ended up stumbling across a super-exciting library called GunJS, which is an offline-first, decentralized database. Not only does it have military-grade encryption and is written in JavaScript, but it’s very powerful and rather easy to use. As I started experimenting more and more with GunJS, I realized that I did not have enough time or desire to include as many GunJS-based features as I had thought. Anyways, after spending too much time solving bugs on some of these features, I realized that I needed to put those ideas to the side and actually finish the product. 2. Electron + React I chose to use Electron for this project because I could build for MacOS, Windows, and Linux simultaneously. I also chose to use React with Electron, so I could make sure that my project was organized and responsive. Initially, my plan was to utilize Electron’s <webview> tag as a way to load websites and track history. Although I would be losing the encryption and privacy standards that I would have if I was building off of Chromium, <webview> seemed like it could be suitable for all of the app’s functionalities. However, it took me a while to figure out how I could render the tag, which is a part of Electron’s API, into React’s DOM(and, therefore, as JSX). This issue, just like some of the others, only sped development time down, and made it unable to just build out the important features of the app, such as online action management. 3. Main Functionality I knew what I wanted to make, but I kept striving for too much on my first build. I couldn’t stop adding new functionalities that I lost focus on the main purpose of the app: to help users keep actions online organized. So, instead of writing out this entire browser, I decided to get my ideas into the hands of the people and validate my value proposition. From there on out, I decided that I would create a simple Chrome extension that allows users to manage what they’re working on by grouping related tabs together. The rest of the extension’s development was rather easy. I removed extraneous features, including using standard React state and browser local storage instead of GunJS, and focused more on the logic of the app. So, what’s the point? Why am I telling you this story? Why did I waste my time on a project that I ended up scrapping? Why did I vie so hard to validate my value proposition? Although there are many features of this idea that I would like to develop, the most important part of this product is user action. Unlike other services, which are built around a business(take Postmates or Uber, for example), this product is designed to make it easier for users to browse the web and keep their online lives organized. The importance of this Chrome extension isn’t necessarily the functionality, but it is the user response. It is seeing how users use this product and learning how to adapt those patterns in a way that makes our standard GUI as responsive and straightforward as possible. And although the final product turned out much different than the original idea, I feel like I’ve found a way to test my idea and use the experience to develop what’s next. Install the Chrome extension and let me know what you think of it! here

I spent all summer building a Chrome extension - Here's What I Built

Do you remember the days when Nokia and RIM (That was the maker Blackberry, back in the days) ignored the advent of real Smartphones, in particular the first iPhone? This is evident in : “It’s OK — we’ll be fine”. History clearly shows us what happened next — Both RIM/Blackberry and Nokia have exited the handset business. But this post is not about how the mobile phone industry was disrupted, and how the innovator’s dilemma played out text book style. You can read about that in much more detail and accuracy in Clay Christensen’s . those famous last words by Mr. Lazaridis, the c0-CEO of RIM work However, I would like to stick to the industry of handset makers as a comparison and analogy of what we see today with another technology that is still somewhat in its infancy and often ignored and even ridiculed by the incumbents. The technology I am talking about, you guessed it from the title 😀, is Blockchain. The Distributed Ledger Technology was first introduced as a technical foundation for Bitcoin (as the first app for Blockchain) and is now used by literally hundreds, if not thousands, of startups, projects and consortiums. I will not explain what a Blockchain is as there are fantastic explanations available . in other places So what does this Blockchain thing have to do with mobile phones? On the surface, obviously, not a lot. Or to be precise, nothing. However, the journey that those two technologies went through (for Smartphones) and possibly will go through (for Blockchains) are remarkably similar in my opinion. Let me explain. In the early 2000’s, almost seven years before the iPhone saw the light of the world, “web-enabling” technologies on cell phones became somewhat popular. One of them was (Wireless Access Protocol) which allowed mobile users to access web content via their phones. It was neither pretty nor very usable, but it was a bridge from a very successful technology (the WWW) to mobile phones. Still, the vast majority of the web was built and meant for traditional browsers running on desktops, and in some cases, early adopters could access those apps and their content via their phones. It was a start. Shortly after step, the first “real” apps started to become available for a few handsets. For example, J2ME applications. This was an improvement to some extent, but still cumbersome to deploy and use. After those somewhat clumsy solutions to bring content and applications to the phone, the first real apps were introduced by Apple via the App Store in 2008. Developers started building dedicated, optimized apps for the iPhone AND had the distribution channel from Apple. The UI and the entire experience looked different than on a computer’s browser — they were optimized for screen size, battery life, limited internet access, touch screen functions and more. Still, those early mobile apps were afterthoughts to their big brothers: the browser / web apps. WAP Around 2009, a new design approach started to become popular: . This meant that mobile applications were now developed from the ground up for smartphones. In many cases the mobile experience now came before developing or launching a traditional web application. This is where I want to make a comparison to the Blockchain world as we see it today. This is where I believe the development of Blockchain is heading over the coming years: To a Blockchain first approach. Mobile First Let’s use Blockchains and the enterprise software space as the example to elaborate. I am using that as a comparison because, first, this is my area of expertise, and second, the comparison is clearest and most advanced in this business. enterprise The biggest consulting firms into the space in order to leverage the Distributed Ledger Technology to reduce inefficiencies within supply chains. Optimizing the Procure to Pay (P2P) process (how companies purchase goods, receive, and pay for goods and services) is one of the key areas of focus. The idea is to provide access to all participants of the P2P process (the buyers and their suppliers) through a shared ledger. In addition the rules of the process, the logic, and the checks are coded into Smart Contracts, which live on the chain as well. This substantially reduces the inefficiencies of today’s labor intensive processes for everyone involved. pour millions Other examples of enterprise use of Blockchain include invoice financing, supply chain finance, factoring or invoice discounting. In essence, leveraging the Blockchain to be the record for financial assets that can be funded by third parties. Blockchain provides clear advantages over existing systems and reduces risk involved in those financing transactions. This means the cost of funding is lower and results in better financing offers. What do those examples in the enterprise space have in common? The sources of the data for these use-cases are always third party systems. Sometimes one, often (and therefore worse) multiple systems. The data always originates outside of a Blockchain, is transferred onto the ledger and processed there. Often it is then copied into another external system later. To illustrate, let’s look at the process of sending an invoice from a supplying company A to their customer B. There is a version of the invoice in company’s A billing system (the “receivable”) and a version of this invoice in the receiving customer’s accounting or ERP system (a “payable”). Often, a middleman service is used for the submission / transfer of that invoice (like a supplier network or an EDI / eInvoicing provider) which stores a representation of that invoice in their database. If this invoice then gets financed (“paid earlier for a fee”) by a financial institution, then a FOURTH representation of that document is stored in the bank or factoring company’s environment. Now we have up to four different versions of the same document, an invoice transaction. Each of those records then gets changed, adjusted, canceled etc independently from each other. Very often, all the data is simply “out of sync”. This creates huge inefficiencies to sort out, which increases the risks for funders. With that, costs are going up for all participants of this flow. This is why the current initiatives to use a single source of truth, a Blockchain, really seem to be good ideas. One location, one valid document, everyone has access to the same version. But even with a shared ledger that everyone points to, we still have the issue that the existing legacy systems hold their own copy in their own database. In addition to the invoice on the Blockchain these systems produce a multitude of copies again. Which one is used now as the “source of truth”? Using a Distributed Ledger for all parties to access a common source of truth for “shared processes” is already solving some of the problems but everyone still keeps local records and pushes/pulls and synchronizes their local state with the Blockchain again. The Blockchain FIRST approach will soon become a reality in many business processes. Applications will be designed where all the business logic and data storage primarily happens on a Blockchain. All other systems and participants will have access to this logic and data and mirror the necessary objects back into their systems. For example, an accounting system that has all of its accounts and business logic on a chain. When a supplier generates an invoice it is created on the chain FIRST and the buying organization reads the that invoice from the Blockchain to use it for further processing in their own ERP system. All involved applications agree that only updates on the Blockchain are valid for further processing (like financing the invoice). With that, there is no receivable or payable anymore. The Receivable and the Payable become the same transaction. There is simply a unique invoice living on the Blockchain, visible and accessible by all authorized parties. To give another example of a common and conflict-laden business process, a created by one system and sent to a supplier where it is represented as a become the same: simply an order, represented as a Smart Contract, where all business logic is embedded that the parties agreed on. purchase order sales order This is where I see this striking similarity to the mobile world. It started as a “let’s do mobile in addition to desktop”, which is comparable with “let’s use a Blockchain as an immutable backup”. We are right now moving into the Blockchain first world, that again very is comparable to the Smartphone world, which now treats the new technology and paradigm as a primary consideration instead of a secondary design aspect. Many apps on Smartphone now follow a “Mobile Only” approach (Snapchat, N26, Tinder and others), due to the ease of use, but also because of the superior capabilities of phones (sensors, location tracking, camera, mobility etc.). Let’s guess when we will see a “Blockchain Only” movement!

Blockchain First

As internet heavy user I organize my games, spams, and services in different email providers such as Yahoo, Microsoft, and due to the facility to organize my things. So, Microsoft has offered me a great feature to create an alias account along with to my main account. Google FANTASTIC !!! However, In few weeks ago I started to receive emails that were supposed to be mine but wasn’t. Later, I found out that a user has been using my outlook.com email that I’ve created to play my Blizzard games. alias No worries, It's just an alias email, I thought. Nevertheless, I was wondering which sort of problem this would cause? Even though I created the alias account for gaming purposes, somebody would create for interception purposes, why not? At that moment I was thinking WTF Microsoft?! Then, I decided to do a test myself. Most of all emails I was receiving in my alias account come from facebook in which the email was registered. Only to illustrate, the test that I did was composed by resetting the Facebook password of the user which was using my alias email account. Facebook did very well into creating different types of validations and layers during my attempt to change the password resulting in my waive.On the other hand, Microsoft demonstrated that could have an issue related to users privacy and security what made me worried and create this new question, how secure my and other accounts are? security I’ll let you think more about it, and also think about what should be done… All things considered, I'd say that Microsoft still not interested in solve this issue because I still receive emails from my shared-alias friend after 1 year by now. :( Cheers! is how hackers start their afternoons. We’re a part of the family. We are now and happy to opportunities. Hacker Noon @AMI accepting submissions discuss advertising & sponsorship To learn more, , , or simply, read our about page like/message us on Facebook tweet/DM @HackerNoon. If you enjoyed this story, we recommend reading our and . Until next time, don’t take the realities of the world for granted! latest tech stories trending tech stories

Should we use trust in outlook.com?

Code Book for Annotation of Diverse Cross-Document Coreference: Annotation Tool

This paper is available on arxiv under CC 4.0 license. Authors: (1) Vogt, Lars, TIB Leibniz Information Centre for Science and Technology; (2) Konrad, Marcel, TIB Leibniz Information Centre for Science and Technology; (3) Prinz, Manuel, TIB Leibniz Information Centre for Science and Technology. Table of Links Abstract & Introduction Interoperability Semantic interoperability and what natural languages like English can teach us Requirements for successfully communicating terms and statements Parallels between the structure of natural language statements and data schemata with implications for semantic interoperability What makes a term a good term and a schema a good schema? The need for a machine-actionable Rosetta Stone for (meta)data that acts as an interlingua for specifying reference terms and reference schemata to support cognitive and semantic interoperability Rosetta Stone and machine-readability: UPRIs, XML Schema datatypes, and RDF for communicating terms, datatypes, and statements Rosetta Stone and machine-interpretability: Wikidata and a modeling paradigm for (meta)data statements based on English Rosetta Stone and semantic interoperability: Specifying term mappings and schema crosswalks Rosetta Stone and cognitive interoperability: Specifying display templates and using a query builder Discussion Related work Conclusion, Acknowledgements, & References Requirements for successfully communicating terms and statements What is needed to communicate terms and statements efficiently and reliably? For communication between human beings to be successful, both the sender and the receiver of the information need to share the same relevant background knowledge. For a given , sender and receiver have to share the same lexical competences. They need to share the same (35) and thus knowledge about the of a given term in the form of an that answers the question (36). Inferential lexical competences are needed for the term inferential lexical competences meaning ontological definition ‘What is it?’ human-interpretability of terms. Sender and receiver also need to share the same (35) and thus about the of a given term. A refers to an individual entity (i.e., a particular) and a or to a set of individuals that meet the defining properties of the term, sometimes also called its extension. A or refers to a specific type of action or attribute. Diagnostic knowledge is often communicated in the form of method-dependent recognition criteria, images, or exemplars that answer the question (36), enabling the receiver to use the term correctly in (i.e., the object is given, and the matching term has to be found) or (i.e., the term is given, and the matching object has to be identified) 1 . Referential lexical competences are needed to use a term correctly in different contexts, and thus for the referential lexical competences diagnostic knowledge reference proper name general term kind term verb predicate ‘How does it look, how to recognize or identify it?’ designation recognition tasks human-actionability of terms. Given that the meaning of terms is provided by their ontological definitions, one could argue that terms are only placeholders, i.e., surrogates, for these definitions and thus statements, and that only statements carry meaning. In other words, the communication of meaning requires more than a single term, but rather several terms placed in context and related by predicates. The communication of meaning thus requires statements. This becomes obvious when the positions of terms in a given sentence are changed, such as versus —the same set of terms carries two different meanings. Therefore, for the efficient and reliable communication of statements, the sender and the receiver of the information must share a set of rules and conventions for formulating sentences using terms. Statements carry meaning in addition to the meaning of the terms that compose them. “Peter travels from Berlin to Paris” “Peter travels from Paris to Berlin” But how is the meaning of a statement represented in the human brain? Understanding how the human brain creates cognitive representations of semantic content is a very active area of research (38). The human brain is a highly interconnected complex system that is continually influenced by input signals from the body and the world, so that a given neuron does not function in isolation but is substantially influenced by its neural context (39). It is therefore not surprising that there is evidence for at least two forms of object knowledge representation in the human brain, supported by different brain systems, i.e., motor-sensory-derived and conceptual-cognition-derived knowledge (40,41), and that lexical concepts are stored as patterns of neural activity that are encoded in multiple areas of the brain, including taxonomic and distributional structures as well as experience-based representational structures that encode information about sensory-motor, affective, and other features of phenomenal experience (42). These findings suggest that the cognitive representation of the meaning of a statement is likely to take the form of a complex network of associations, analogous to a multidimensional mind-map. Thus, when attempting to communicate a statement, the sender must first translate this multidimensional mind-map into a one-dimensional sequence of terms, i.e., a sentence. This translation step is supported by a set of syntactic and grammatical conventions, shared by the sender and the receiver, for formulating sentences using terms. According to the of linguistics (43,44), the main verb of a statement, together with its auxiliaries, forms the statement’s predicate. A predicate has a that determines the number and types of it requires to complete its meaning. may be additionally related to the predicate, but they are not necessary to complete the predicate’s meaning. Adjuncts provide optional information, such as a time specification in a parthood statement. Therefore, every statement has a subject phrase as one of its arguments, and can have one or more object phrases as further arguments and additional adjuncts, depending on the underlying predicate. predicate-argument-structure valence arguments Adjuncts In the syntax of a statement, each argument and adjunct of a predicate can be understood as having a specific with a specific . This is related to Kipper et al.’s (45) verb lexicon , where they extend Levin verb classes (46) to include abstract representations of syntactic frames for each class, with explicit correspondences between (i.e., positions in a syntax tree) and the (i.e., thematic roles sensu (45)) that these positions express (Fig. 2B). Each verb class lists the semantic roles allowed by the predicate-argument-structure of the instances of the class and the basic syntactic frames shared by the instances. position semantic role VerbNet syntactic positions semantic roles The list of arguments of a predicate-argument-structure can be described by a list of taken from a set of predefined possible labels (e.g., AGENT, PATIENT, THEME, etc.), and the syntactic frames are represented by an ordered sequence of such thematic labels. The thematic labels function as descriptors of semantic roles that are mapped onto positions in a given syntactic frame (45). thematic labels In , Palmer et al. (47) define semantic roles at the level of individual verbs by numbering the arguments of the verb, with the first argument generally taking the semantic role of an AGENT and the second argument typically, but not necessarily, taking the semantic role of a PATIENT or a THEME. For higher numbered arguments, there are no consistently defined roles. PropBank Syntax trees, with their different syntactic positions and associated semantic roles, contribute substantially to the meaning of their sentences, and they are used to translate a web of ideas in the mind of a sender into a string of words that can be understood by a receiver to translate it back into a web of ideas (48). The clearer the semantic roles of the different positions are, the easier it is for a human being to understand the information. In a sense, we can understand syntax trees as the first knowledge graphs created by humans, and their use seems to be quite straightforward, providing a structure that is interoperable with human cognitive conditions, thus satisfying the need for cognitive interoperability. To sum it up, whenever information needs to be communicated efficiently and reliably, the sender and receiver of the information must not only share the same inferential and referential lexical competences regarding the terms used in their communication, but also the same set of syntactic and grammatical conventions for formulating sentences with these terms, resulting in the same syntax tree in both sender and receiver. The knowledge shared by the sender and the receiver of the message ensures that the message is: 1. : The receiver must be able to in a message, i.e., where a term or a statement begins and where it ends; readable identify the basic units of information 2. : The receiver must be able to understand each term and statement and thus recognize its ; interpretable meaning 3. actionable: The receiver must be able to correctly apply terms and statements from the message, including correctly designating or recognizing of terms and correctly placing a statement in . referents context

Requirements for Successfully Communicating Terms and Statements

Rosetta Stone and machine-interpretability: Wikidata and a modeling paradigm 

This paper is available on arxiv under CC 4.0 license. Authors: (1) Vogt, Lars, TIB Leibniz Information Centre for Science and Technology; (2) Konrad, Marcel, TIB Leibniz Information Centre for Science and Technology; (3) Prinz, Manuel, TIB Leibniz Information Centre for Science and Technology. Table of Links Abstract & Introduction Interoperability Semantic interoperability and what natural languages like English can teach us Requirements for successfully communicating terms and statements Parallels between the structure of natural language statements and data schemata with implications for semantic interoperability What makes a term a good term and a schema a good schema? The need for a machine-actionable Rosetta Stone for (meta)data that acts as an interlingua for specifying reference terms and reference schemata to support cognitive and semantic interoperability Rosetta Stone and machine-readability: UPRIs, XML Schema datatypes, and RDF for communicating terms, datatypes, and statements Rosetta Stone and machine-interpretability: Wikidata and a modeling paradigm for (meta)data statements based on English Rosetta Stone and semantic interoperability: Specifying term mappings and schema crosswalks Rosetta Stone and cognitive interoperability: Specifying display templates and using a query builder Discussion Related work Conclusion, Acknowledgements, & References Rosetta Stone and semantic interoperability: Specifying term mappings and schema crosswalks Different controlled vocabularies and ontologies may contain terms that have the same meaning and referent, and are therefore strict synonyms. Unfortunately, if their UPRIs differ, a machine will not be able to recognize them as synonyms, and statements using such terms will not be interoperable because the terms they use are not terminologically interoperable (10). The problem of the lack of terminological interoperability can be solved by specifying ontological term mappings for all strictly synonymous terms across different vocabularies and ontologies, as discussed above. Unfortunately, however, terms that appear to be strictly synonymous at first glance sometimes turn out not to be strictly synonymous on closer inspection, or the evaluation of their synonymy involves evaluating chains of interdependencies between numerous terms across different ontologies and is practically not feasible. In such cases, but also when terms refer to the same entities but differ in their ontological definitions, referential term mappings can be specified. In the first implementation of the Rosetta Framework, we suggest being pragmatic and focusing on referential interoperability of terms and thus referential term mappings—ontological term mappings can still be added later. We plan to use Wikidata terms as referent terms for referential term mappings. The mappings and should be stored and made openly accessible and usable in a terminology service such as the . TIB Terminology Service Analogous to referential term mappings, referential schema crosswalks will be specified to establish schematic interoperability between all schemata that model the same type of statement (10). With schema crosswalks specified via the appropriate Rosetta Framework reference schema (cf. Fig. 4), researchers can use a schema that is optimized for the set of operations and tools relevant to their particular project and research topic, while making the (meta)data they create referentially and schematically interoperable with all other statements created with schemata for which schema crosswalks have been specified for the same reference schema (Fig. 8). All provenance and metadata statements associated with a particular dataset or a given data statement can be mapped to any provenance and metadata schema or model, such as PROV-O, PAV ontology (62), Dublin Core Metadata Initiative, or DataCite Schema, if a corresponding reference schema is specified along with the required . metadata crosswalks In addition to specifying schema crosswalks to different (meta)data graph schemata, they can also be specified to different formats such as RDF/OWL, GraphQL, Python or Java data classes, JSON, and CSV. Since all of these formats must provide data slots for a given statement type that map to its positions and their associated semantic roles, mapping to non-graph-based formats should be analogous to mapping to graph-based formats (e.g., Fig. 2D). The Rosetta Editor must allow the specification of new graph-based and tabular (meta)data schemata and formats whenever needed, thus allowing adaption to newly emerging (meta)data formats and standards 5 . This takes into account the observation that FAIRness is not sufficient as an indicator of high (meta)data quality―the use of (meta)data often depends on its fitness-for-use, i.e., data must be available in appropriate formats that comply with established standards and protocols that allow their direct use, e.g., when a specific analysis software requires data in a specific format. With the Rosetta Framework and its Editor, any domain expert can begin to specify their own small Rosetta module for each type of statement that is relevant to them. The module comprises the statement class, a reference schema, and schema crosswalks. Rosetta modules facilitate the production of FAIR machine-actionable (meta)data statements. Using the Editor, any number of additional schemata can be specified at any time and linked to the reference schema through a schema crosswalk. These additional schemata can be used by applications to make their (meta)data, which are stored according to reference schemata, accessible in any other schema and format—either virtually through their UI or as an export option. As a result, (meta)data managed by an application that applies the Rosetta Framework are decoupled from the application’s storage model and can be readily used in other frameworks And since schema crosswalks allow (meta)data conversion in both directions, they can also be used to import (meta)data from any schema into the reference schema. Combined with the concept of semantic units (50), the Rosetta modules resemble Knowledge Graph Building Blocks, i.e., small information modules for knowledge processing (for a discussion of Knowledge Graph Building Blocks, see (64)). The ability to specify schema crosswalks that convert, for example, weight measurement statements that comply with a corresponding reference schema into data graphs that comply with the corresponding OBI schema also opens up the possibility for knowledge graph applications to establish workflows in which statements that meet certain criteria, such as having a certain confidence level or having a documented reference to a relevant source of evidence for the statement, are then converted into data graphs that comply with the OBI schema for weight measurements, thereby converting information from a schema that models statements into a schema that models a human-independent reality. From reference schemata to OWL-based schemata that allow reasoning (Meta)data statements and their corresponding statement classes can be understood as a formalized approach to modeling n-ary predicates. Each statement class refers to an n-ary predicate, and the statement class can be understood as an attempt to model this predicate as an ontology class instead of an ontology property. As a consequence, reasoning over property axioms such as transitivity or domain and range specifications is not straightforward with (meta)data statements resulting from instantiations of reference schemata, and tools established for OWL-based frameworks cannot be readily reused in the Rosetta Framework. Therefore, it is important for the Framework to provide interoperability with OWL and description logics by providing, by default, a corresponding OWL-based schema for each reference schema and its corresponding statement class. When a new reference schema is specified for a new type of statement, the Rosetta Editor uses the information provided for the newly specified statement class and for the reference schema to automatically specify an OWL-based schema and the associated schema crosswalk. When we take a material has-part statement with a material object as the subject and another material object as the object argument, the corresponding reference schema takes on the structure shown in Figure 9, top. We can define a new object property ‘has material part’ as a subproperty of ‘required object position’, with the domain and range specified as ‘MATERIAL OBJECT’ (Fig. 9, bottom). An annotation property indicates that the ‘has material part’ property belongs to the ‘material has-part statement’ class. The domain and range specifications are taken from the constraints of the two slots that the property connects, and any logical property axioms, such as transitivity, are taken from the corresponding specifications on the range object-position class (see discussion above).

Rosetta Stone and Semantic Interoperability: Specifying Term Mappings and Schema Crosswalks

This paper is available on arxiv under CC 4.0 license. Authors: (1) Vogt, Lars, TIB Leibniz Information Centre for Science and Technology; (2) Konrad, Marcel, TIB Leibniz Information Centre for Science and Technology; (3) Prinz, Manuel, TIB Leibniz Information Centre for Science and Technology. Table of Links Abstract & Introduction Interoperability Semantic interoperability and what natural languages like English can teach us Requirements for successfully communicating terms and statements Parallels between the structure of natural language statements and data schemata with implications for semantic interoperability What makes a term a good term and a schema a good schema? The need for a machine-actionable Rosetta Stone for (meta)data that acts as an interlingua for specifying reference terms and reference schemata to support cognitive and semantic interoperability Rosetta Stone and machine-readability: UPRIs, XML Schema datatypes, and RDF for communicating terms, datatypes, and statements Rosetta Stone and machine-interpretability: Wikidata and a modeling paradigm for (meta)data statements based on English Rosetta Stone and semantic interoperability: Specifying term mappings and schema crosswalks Rosetta Stone and cognitive interoperability: Specifying display templates and using a query builder Discussion Related work Conclusion, Acknowledgements, & References What makes a term a good term and a schema a good schema? First and foremost, a good schema for a (meta)data statement must cover all the information that needs to be documented, stored, and represented for the corresponding type of statement. However, beyond that, there are many other criteria for evaluating schemata. Most of these relate to the different operations one wants to perform on the (meta)data, and the formats required by the corresponding tools, which determine the of the (meta)data. These include search operations (i.e., the findability in FAIR), but also reasoning and all kinds of data transformations, such as unit conversion for measurement data. Communicating with humans is another set of operations that needs to be considered when evaluating (meta)data schemata, as it relates to cognitive interoperability and thus the human-actionability of (meta)data. degree of machine-actionability Unfortunately, different operations are likely to have different requirements on a schema, and the tools that execute these operations may have their own requirements. For example, optimizing the findability of measurement data requires a different data schema than optimizing reasoning over them or their reusability. A given schema therefore needs to be evaluated in terms of the operations to be performed and the tools to be used on the (meta)data, often involving trade-offs between different operations that are prioritized differently in order to achieve an overall optimum. An example is the trade-off between reasoning and human-readability, as discussed above in the context of the dilemma between machine-actionable and human-actionable (meta)data schemata (Fig. 1). As a consequence, for a given type of statement, there is likely to be a need for more than one corresponding schema. This is mainly because, besides historical reasons, different research communities often have different and thus emphasize different aspects of a given type of entity, resulting in the need for different terms for the same type of entity (resulting in issues with ontological and thus terminological interoperability, but not necessarily with referential interoperability), but also because different research communities want to perform on the (meta)data, different types of schemata. Since operations on (meta)data can be performed with different sets of tools, not only the structure of the schema is important, but also the format in which it can be communicated with such tools. For example, some tools require (meta)data to be in RDF/OWL, others in JSON, as CSV, or as a Python or Java data class. frames of reference different operations Obviously, FAIRness is not sufficient as an indicator of high quality (meta)data―the use of (meta)data often depends on their , i.e., their availability in appropriate formats that conform to established standards and protocols that allow their direct use, e.g., when a specific analysis software requires data in a specific format. fitness-for-use Therefore, although agreement on a common vocabulary and a common set of schemata would be a solution for semantic interoperability and machine-actionability of (meta)data across different research domains, this is unlikely to happen, and we have to think pragmatically and emphasize the need for ontological and referential term mappings and schema crosswalks for terminological schematic interoperability.

Stories, News & Mentions about OUTLOGIC

Outlogic’s Stories on HackerNoon