LiveAPI

liveapi

*This is a continuation of a series of articles in which I briefly cover the main points of a specific topic in system architecture design. The first article can be read *and the full guide you can find on my . here github API architecture refers to the set of rules, protocols, and tools that dictate how software components should interact. The architecture of an API is not just about facilitating communication; it's also about ensuring that this communication is efficient, secure, and scalable. A well-designed API architecture can significantly enhance the performance of a system, while a poorly designed one can lead to bottlenecks, security vulnerabilities, and maintenance nightmares. Different Styles of API Architecture The most common API design styles: (Representational State Transfer) is the most used style that uses standard methods and HTTP protocols. It's based on principles like statelessness, client-server architecture, and cacheability. It's often used between front-end clients and back-end services. REST is a query language for APIs. Unlike REST, which exposes a fixed set of endpoints for each resource, GraphQL allows clients to request exactly the data they need, reducing over-fetching. GraphQL is a protocol allowing two-way communication over TCP. Clients use web sockets to get real-time updates from back-end systems. WebSocket is a mechanism that allows one system to notify another system about specific events in real-time. It is a user-defined HTTP callback. Webhook is a protocol that one service can use to request a procedure/method from a service located on another computer in a network. Usually, It's designed for low-latency, high-speed communication. RPC (gRPC) is a protocol for exchanging structured information to implement web services. It relies on XML and is known for its robustness and security features, currently considered a legacy protocol. SOAP Let's look at each protocol separately with all their pros, cons, and use cases. REST is an architectural style that uses standard conventions and protocols, making it easy to understand and implement. Its stateless nature and use of standard HTTP methods make it a popular choice for building web-based APIs. REST While REST has been the de facto standard for building APIs for a long time, other styles like GraphQL have emerged, offering different paradigms for querying and manipulating data. : XML, JSON, HTML, plain text Format : HTTP/HTTPS Transport protocol Key Concepts and Characteristics : In REST, everything is a resource. A resource is an object with a type, associated data, relationships to other resources, and a set of methods that operate on it. Resources are identified by their URIs (typically a URL). Resource : REST services often map directly to CRUD (Create, Read, Update, Delete) operations on resources. CRUD Operations : REST systems use standard HTTP methods: HTTP Methods GET: Retrieve a resource. POST: Create a new resource. PUT/PATCH: Update an existing resource. DELETE: Remove a resource. : REST APIs use standard HTTP status codes to indicate the success or failure of an API request: Status Codes 2xx - Acknowledge and Success 200 - OK 201 - Created 202 - Accepted 3xx - Redirection 301 - Moved Permanently 302 - Found 303 - See Other 4xx - Client Error 400 - Bad Request 401 - Unauthorized 403 - Forbidden 404 - Not Found 405 - Method Not Allowed 5xx - Server Error 500 - Internal Server Error 501 - Not Implemented 502 - Bad Gateway 503 - Service Unavailable 504 - Gateway Timeout : Each request from a client to a server must contain all the information needed to understand and process the request. The server should not store anything about the client's state between requests. Stateless : REST is based on the client-server model. The client is responsible for the user interface and experience, while the server is responsible for processing requests, handling business logic, and storing data. Client-Server : Responses from the server can be cached by the client. The server must indicate whether a response is cacheable or not. Cacheable : A client cannot ordinarily tell whether it is connected directly to the end server or an intermediary. Intermediary servers can improve system scalability by enabling load balancing and providing shared caches. Layered System Hypermedia As The Engine Of Application Stat is a REST web service principle that enables clients to interact with and navigate through a web application entirely based on the hypermedia provided dynamically by the server in its responses, promoting loose coupling and discoverability. HATEOAS: Use Cases : Many web services expose their functionality via REST APIs, allowing third-party developers to integrate and extend their services. Web Services : Mobile apps often communicate with backend servers using REST APIs to fetch and send data. Mobile Applications : SPAs use REST APIs to dynamically load content without requiring a full page refresh. Single Page Applications (SPAs) Systems within an organization can communicate and share data using REST APIs. Integration Between Systems: Example Request GET “/user/42” Response {
    "id": 42,
    "name": "Alex",
	"links": {
        "role": "/user/42/role"
    }
} GraphQL offers a more flexible, robust, and efficient approach to building APIs, especially in complex systems or when the frontend needs high flexibility. It shifts some of the responsibility from the server to the client, allowing the client to specify its data requirements. GraphQL While it's not a replacement for REST in all scenarios, it offers a compelling alternative in many situations, particularly as applications become more networked and distributed. : JSON Format : HTTP/HTTPS Transport protocol Key Concepts and Characteristics : It allows clients to request the data they need, making it possible to get all required information in a single request. Query Language for APIs : GraphQL APIs are organized in terms of types and fields, not endpoints. It uses a strong type system to define the capabilities of an API. All the types exposed in an API are written down in a schema using the GraphQL Schema Definition Language (SDL). Type System : Unlike REST, where you might have multiple endpoints for different resources, in GraphQL, you typically expose a single endpoint that expresses the complete set of capabilities of the service. Single Endpoint : On the server side, resolvers gather the data described in a query. Resolvers : Beyond just querying data, GraphQL includes built-in support for real-time updates using subscriptions. Real-time Updates with Subscriptions : A GraphQL server can be queried for the types it supports. This creates a strong contract between client and server, allowing for tooling and better validation. Introspective Use Cases : For applications (especially mobile) with crucial bandwidth, you want to minimize the data fetched from the server. Flexible Frontends : A GraphQL layer can be introduced to aggregate the data from these services into a unified API if you have multiple microservices. Aggregating Microservices : With its subscription system, GraphQL can be an excellent fit for applications that need real-time data, like chat applications, live sports updates, etc. Real-time Applications : With REST, you often need to version your APIs once changes are introduced. With GraphQL, clients only request the data required, so adding new fields or types doesn't create breaking changes. Version-Free APIs Example Request GET “/graphql?query=user(id:42){ name  role {  id  name  } }” Response {
    "data": {
        "user": {
            "id": 42,
            "name": "Alex",
            "role": {
                "id": 1,
                "name": "admin"
            }
        }
    }
} WebSocket provide a full-duplex communication channel over a single, long-lived connection, allowing real-time data exchange between a client and a server. This makes it ideal for interactive and high-performance web applications. WebSockets : Binary Format : TCP Transport protocol Key Concepts and Characteristics : Unlike the traditional request-response model, WebSockets provide a full-duplex communication channel that remains open, allowing for real-time data exchange. Persistent Connection : WebSockets start as an HTTP request, which is then upgraded to a WebSocket connection if the server supports it. This is done via the `Upgrade` header. Upgrade Handshake : Once the connection is established, data is transmitted as frames. Both text and binary data can be sent through these frames. Frames : WebSockets allow for direct communication between the client and server without the overhead of opening a new connection for each exchange. This results in faster data exchange. Low Latency : Both the client and server can send messages to each other independently. Bidirectional : After the initial connection, data frames require fewer bytes to send, leading to less overhead and better performance than repeatedly establishing HTTP connections. Less Overhead : WebSockets support subprotocols and extensions, allowing for standardized and custom protocols on top of the base WebSocket protocol. Protocols and Extensions Use Cases : Real-time multiplayer games where players' actions must be immediately reflected to other players. Online Gaming : Applications like Google Docs, where multiple users can edit a document simultaneously and see each other's changes in real-time. Collaborative Tools : Stock trading platforms where stock prices need to be updated in real-time. Financial Applications : Any application where users need to receive real-time notifications, such as social media platforms or messaging apps. Notifications : News websites or social media platforms where new posts or updates are streamed live to users. Live Feeds Example Request GET “ws://site:8181” Response HTTP/1.1 101 Switching Protocols Webhook is a user-defined HTTP callback triggered by specific web application events, allowing real-time data updates and integrations between different systems. Webhook : XML, JSON, plain text Format : HTTP/HTTPS Transport protocol Key Concepts and Characteristics : Webhooks are typically used to denote that an event has occurred. Instead of requesting data at regular intervals, webhooks provide data as it happens, turning the traditional request-response model on its head. Event-Driven : Webhooks are essentially a user-defined callback mechanism. When a specific event occurs, the source site makes an HTTP callback to the URI provided by the target site, which will then take a specific action. Callback Mechanism : When the webhook is triggered, the source site will send data (payload) to the target site. This data is typically in the form of JSON or XML. Payload : Webhooks allow applications to get real-time data, making them highly responsive. Real-time : Users or developers can typically define what specific events they want to be notified about. Customizable : Since webhooks involve making callbacks to user-defined HTTP endpoints, they can pose security challenges. It's crucial to ensure that the endpoint is secure, the data is validated, and possibly encrypted. Security Use Cases : Triggering builds and deployments when code is pushed, or a pull request is merged. Continuous Integration and Deployment (CI/CD) : Notifying downstream systems when content is updated, published, or deleted. Content Management Systems (CMS) : Informing e-commerce platforms about transaction outcomes, such as successful payments, failed transactions, or refunds. Payment Gateways : Receiving notifications about new posts, mentions, or other relevant events on social media platforms. Social Media Integrations : Devices or sensors can trigger webhooks to notify other systems or services about specific events or data readings. IoT (Internet of Things) Example Request GET “ ” https://external-site/webhooks?url=http://site/service-h/api&name=name Response {
  "webhook_id": 12
} RPC and gRPC (Remote Procedure Call) is a protocol that allows a program to execute a procedure or subroutine in another address space, enabling seamless communication and data exchange between distributed systems. RPC (Google RPC) is a modern, open-source framework built on top of RPC that uses HTTP/2 for transport and Protocol Buffers as the interface description language, providing features like authentication, load balancing, and more to facilitate efficient and robust communication between microservices. gRPC RPC : JSON, XML, Protobuf, Thrift, FlatBuffers Format : Various Transport protocol Key Concepts and Characteristics : RPC allows a program to cause a procedure (subroutine) to execute in another address space (commonly on another computer on a shared network). It's like calling a function performed on a different machine than the caller's. Definition : In the context of RPC, stubs are pieces of code generated by tools that allow local and remote procedure calls to appear the same. The client has a stub that looks like the remote procedure, and the server has a stub that unpacks arguments, calls the actual procedure, and then packs the results to send back. Stubs : Traditional RPC calls are blocking, meaning the client sends a request to the server and gets blocked waiting for a response from the server. Synchronous by default : Many RPC systems allow different client and server implementations to communicate regardless of the language they're written in. Language Neutral : RPC often requires the client and server to know the procedure being called, its parameters, and its return type. Tight Coupling Use Cases : RPC is commonly used in distributed systems where parts of a system are spread across different machines or networks but need to communicate as if they're local. Distributed Systems : NFS (Network File System) is an example of RPCs performing file operations remotely. Network File Systems Example Request {
    "method": "addUser",
    "params": [
        "Alex"
    ]
} Response {    
    "id": 42,
    "name": "Alex",
	"error": null
} gRPC : Protobuf Format : HTTP/2 Transport protocol Key Concepts and Characteristics : gRPC is an open-source RPC framework developed by Google. It uses HTTP/2 for transport, Protocol Buffers (Protobuf) as the interface description language, and provides authentication, load balancing features, and more. Definition : This is a language-neutral, platform-neutral, extensible mechanism for serializing structured data. With gRPC, you define service methods and message types using Protobuf. Protocol Buffers : gRPC is designed for low latency and high throughput communication. HTTP/2 allows for multiplexing multiple calls over a single connection and reduces overhead. Performance : gRPC supports streaming requests and responses, allowing for more complex use cases like long-lived connections, real-time updates, etc. Streaming : gRPC allows clients to specify how long they will wait for an RPC to complete. The server can check this and decide whether to complete the operation or abort if it will likely take too long. Deadlines/Timeouts : gRPC is designed to support pluggable authentication, load balancing, retries, etc. Pluggable : Like RPC, gRPC is language agnostic. However, with Protobuf and the gRPC tooling, generating client and server code in multiple languages is easy. Language Neutral Use Cases : gRPC is commonly used in microservices architectures due to its performance characteristics and ability to define service contracts easily. Microservices : Given its support for streaming, gRPC is suitable for real-time applications where servers push data to clients in real-time. Real-time Applications : gRPC's performance benefits and streaming capabilities make it a good fit for mobile clients communicating with backend services. Mobile Clients Example message User {
  int id = 1
  string name = 2
}

service UserService {
  rpc AddUser(User) returns (User);
} SOAP , which stands for Simple Object Access Protocol, is a protocol for exchanging structured information to implement web services in computer networks. It's an XML-based protocol that allows programs running on disparate operating systems to communicate with each other. SOAP : XML Format : HTTP/HTTPS, JMS, SMTP, and more Transport protocol Key Concepts and Characteristics : SOAP messages are formatted in XML and contain the following elements: XML-Based : The root element of a SOAP message that defines the XML document as a SOAP message. Envelope : Contains any optional attributes of the message used in processing the message, either at an intermediary point or the ultimate end-point. Header : Contains the XML data comprising the message being sent. Body : An optional Fault element that provides information about errors while processing the message. Fault : SOAP can be used with any programming model and is not tied to a specific one. Neutrality : It can run on any operating system and in any language. Independence : Each request from a client to a server must contain all the information needed to understand and process the request. Stateless : The Fault element in a SOAP message is used for error reporting. Built-in Error Handling : Operates based on well-defined standards, including the SOAP specification itself, as well as related standards like WS-ReliableMessaging for ensuring message delivery, WS-Security for message security, and more. Standardized Use Cases : SOAP is often used in enterprise settings due to its robustness, extensibility, and ability to traverse firewalls and proxies. Enterprise Applications : Many web services, especially older ones, use SOAP. This includes services offered by major companies like Microsoft and IBM. Web Services : SOAP's built-in security and extensibility make it a good choice for financial transactions, where data integrity and security are paramount. Financial Transactions : Telecom companies might use SOAP for processes like billing, where different systems must communicate reliably. Telecommunications Example Request <?xml version="1.0"?>
<soap:Envelope>
	<soap:Body>
		<m:AddUserRequest>
			<m:Name>Alex</m:Name>
		</m:AddUserRequest>
	</soap:Body>
</soap:Envelope> Response <?xml version="1.0"?>
<soap:Envelope>
	<soap:Body>
		<m:AddUserResponse>
			<m:Id>42</m:Id>
			<m:Name>Alex</m:Name>
		</m:AddUserResponse>
	</soap:Body>
</soap:Envelope> Conclusion The landscape of API architecture styles is diverse, offering various approaches like REST, SOAP, RPC, and more, each with unique strengths and use cases, enabling developers to choose the most suitable paradigm for building scalable, efficient, and robust integrations between different software components and systems.

The System Design Cheat Sheet: API Styles - REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

সিস্টেম ডিজাইন চিট শীট: API শৈলী - REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

Der Spickzettel für das Systemdesign: API-Stile – REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

La hoja de trucos de diseño del sistema: estilos de API: REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

L'aide-mémoire pour la conception du système : styles d'API - REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

सिस्टम डिज़ाइन चीट शीट: एपीआई शैलियाँ - REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

システム設計チートシート: API スタイル - REST、GraphQL、WebSocket、Webhook、RPC/gRPC、SOAP

시스템 설계 치트 시트: API 스타일 - REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

A folha de referências do design do sistema: estilos de API - REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

Памятка по проектированию системы: стили API — REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

Sistem Tasarımı Hile Sayfası: API Stilleri - REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

Bảng tóm tắt về thiết kế hệ thống: Kiểu API - REST, GraphQL, WebSocket, Webhook, RPC/gRPC, SOAP

系统设计备忘单：API 样式 - REST、GraphQL、WebSocket、Webhook、RPC/gRPC、SOAP

It is no secret that the blockchain sector moves at pace. In 2020, we have seen the launch of several significant protocols, each with their particular nuances and strengths which offer value propositions across industries, ranging from blockchain and enterprise to finance and supply chain. At , we understand that the need to communicate across multiple blockchain networks is pivotal for many network users and developers, that is why we launched earlier this year, an API tool that provides a unified syntax for communicating across many different protocols. Ubiquity eliminates the need for users of multiple networks to always switch contexts and re-learn a new set of commands and APIs. Blockdaemon Ubiquity However, with constant innovation and development in the blockchain sector, it is essential that Ubiquity also stays ahead of these developments, that is why I'm thrilled to discuss the next exciting phase in the development of the service. Ubiquity System Architecture One of the initial challenges we had developing Ubiquity was in finding the right system architecture. The first version of the architecture has proven to suit our needs effectively. CockroachDB serves as a performance index backed on top of Ceph which stores large amounts of unstructured data, while RabbitMQ, a reliable message broker, powers the Event-Driven-Architecture for relaying syncing data across the microservice architecture. Live APIs We incorporated feedback on customers' demand for a live API, and to that end, Ubiquity will offer some exciting new features in the next iteration of its development. The engineering team focused on Ubiquity v1.0 as a historical data API. Our next iteration provides users with real-time notifications on transactions via a WebSocket API. Network Reorganizations Blockchain development has many specific and challenging nuances that don't exist in other areas of software development. One of these is in accounting for widespread network reorganizations which can occur both in terms of attack type situations and also planned forks, where large volumes of blocks are reverted. Blockchain networks running proof-of-work (PoW) consensus perform smaller forks naturally during normal operations. We made sure to account for these possibilities by enabling support for these reverts in Ubiquity and notifying users if previously sent transactions have been reverted. Staking Data We expect the API generic model to evolve and import more data types. The newest addition is staking reward data, incubating for internal use. A unified staking dashboard on the Blockdaemon Marketplace, backed by Ubiquity, enables users to monitor node efficiency across multiple protocols--all within one unified dashboard. Data Extraction with Zombies Extracting rich historical data from blockchain nodes is difficult and can be extremely slow due to poor data locality. While extracting state change information is as easy as parsing a block, getting detailed information such as historical balances from archive nodes incurs random reads ranging over terabytes of cold data, which takes months. To address this challenge, we reimagined the data extraction pipeline to work with less storage space by replaying blockchain sync on special-purpose "zombie" nodes and tracing state data. These zombie nodes exist solely for state/tracing computation and are otherwise idle (have no free will to sync blocks). Although applicable to any blockchain, this approach has proven incredibly valuable on Bitcoin, where data extraction was sped up by two orders of magnitude, by fully leveraging the UTXO set in-memory cache provided by Bitcoin Core. The zombie syncing mechanism will also enable the quick onboarding of new protocols. High Availability Even the best server hardware on the market requires occasional maintenance. So from day one, Ubiquity has been designed to eliminate any single points of failure and is resilient to partial outages, conquering the availability problem by dividing the architecture into layers with clear separation. Multiple blockchain nodes feeding new data into the API. Source Layer: Multi-zone Kubernetes cluster and RabbitMQ event broker. Service Layer: Redundant batch processing and live jobs for syncing blockchain data to the database. Syncing Layer: Swarm of web servers querying the database. API Layer: Ingestion Phase Presentation Phase Testing Testing Testing We continually test API performance to iteratively push improvements based on insights gained through thousands of VictoriaMetrics time series and millions of Jaeger traces. The critical metric optimization on the live API was latency--the time between when the block gets created on the network, to when the customer receives information on the event via Ubiquity. A valuable few hundred milliseconds are saved by converting the notifiers for blockchain changes from polling to push-based delivery. We are happy with the performance of the API for the most part, and stress-tested it to maximize raw request per second throughput, mainly by optimizing SQL queries. A test involving scaling the SQL indexing part of the syncing layer yielded the most exciting results, increasing the throughput for Bitcoin from just ~100 to sustained ~1700 transactions per second. Changes included using binary encodings for primary keys, dropping foreign key constraints and switching to an eventually consistent database model--data inconsistencies during updates are now gracefully handled on the query layer instead. Another syncing component to be tested was the blockchain tracker, which resolves re-branches and states which blocks to index. A trace below shows how performance was improved by 40% by adding parallel processing. Without Parallel Processing With Parallel Processing A Final Note As a blockchain infrastructure provider, we both must anticipate and move with developments in the ever-evolving blockchain sector. Our latest updates to Ubiquity are our endeavour to do just that. With support currently provided for Bitcoin, Ethereum, Stellar and Ripple, we are planning to add several additional protocols to the list soon. If you'd like to try Ubiquity out, or learn more about its latest features, please visit the for additional details. Ubiquity page in our Marketplace This article was written by Richard Patel, Blockchain Developer at Blockdaemon

The Return of the API: Analyzing the Latest Developments in Ubiquity 

API Analytics are key for any platform company that wants to gain insights into their API and platform usage. These insights can be leveraged by product owners, growth teams, developer relations, and more to make more strategic decisions based on the raw health of your platform business rather than just gut feelings. However, many API platforms have a very high volume of API calls per day. This volume creates a set of unique challenges when designing an analytics system that scales without crippling their platform or having sticker shock when looking at their cloud vendor’s bill. This post goes into some of the inner workings of how we designs Moesif’s API analytics platform to handle companies with billions of API calls a day. Agents/SDKs Designing for scale starts with a well-designed agent to collect metrics from various applications and send to your analytics platform. Since the agent or SDK is embedded in your application, a poorly designed one could cripple your core services if your analytics service gets overloaded. In order to prevent this, your agent should have automatic fail safes and kill switches. For starters, the agent should collect any metrics asynchronous from your core application code and should sit out-of-band, This way, even if your analytics service is down for extended periods of time, your application availability is not affected. Ensure any errors in processing metrics data are handled and not propagated back to the application itself. Since you’ll be collecting metrics asynchronously, you’ll want to leverage local queuing. This can be done in memory or to disc depending on the data consistency required. Keep in mind though, any resources that need to be allocated locally such as memory or disc required for a queue needs to be constrained. Otherwise, a queue could continue to grow without bounds if your analytics service goes down for an extended period of time eventually causing the process to run out of memory and crash. One way to solve this via fixed queue size and drop older events. Dropping older events during a network disconnect means you sacrifice consistency for availability, a key pillar of which is usually referenced in when discussing distributed data stores, but is applicable here also. Queueing also enables you to batch events which reduces network overhead. Brewer’s CAP theorem For more info, read Best practices for building SDKs for APIs Sampling If your API is processing billions of API calls a day, even with queueing and batching, you may quickly exceed your compute and and storage requirements. The compute cost to run such a system could easily be in the hundreds of thousands or millions of dollars a year. One way to reduce this pressure is via a smart sampling algorithm. While a rudimentary algorithm that consistently chooses a random percentage of events to log vs drop may work for the short term, this can heavily reduce the usefulness of your analytics system. This is because most products exhibit a power law curve when it comes to customer usage. Your top 20% of customers will create 80% of your traffic. This also means you have a of customers that send very little traffic. Heavily sampling customers who send very little data in the first place can create distorted views of your customer journey. long tail One way to reduce this issue is by implementing a more intelligent sampling scheme. One way is sample on a per user or per API key basis. For example, any user who is a top 20% customer will have a sample rate of 10% whereas the rest of your customers will have a sample rate of 100% (i.e. capture all activity). This means you need a mechanism to propagate the sample rates to your agent. Collection Network Regardless if your analytics service is accessible only from your own infrastructure only or is accessible from the intranet, you should have a system that can handle fail over in case one of the collection nodes goes down. A well designed network will leverage DNS-based load balancing across multiple data centers around the world such as AWS Route 53 or Azure Traffic Manager with a relatively short TTL. Within each data center, you can then leverage a proxy server like HAProxy, NGINX, AWS ELB to distribute traffic among your collector nodes. We at Moesif are heavy users of HAProxy to load balance within each data center and Azure’s Traffic Manager to load balance across different data centers. Your collector logic should be very light wight. It’s job should be only authentication, some lightweight input validation, and persisting to disc as quick as possible. This is where solutions like Kafka can be extremely useful. Roll Ups Even if you sample, you may find that your storage costs are ever increasing and queries take exceedingly long time. Modern databases like Elasticsearch and Cassandra can hold hundreds of terabytes of data with their distributed architecture, yet you’ll run into increasingly long queries (or worse: out of memory errors) when you attempt to perform aggregations on these massive data sets. In addition, the storage requirements continue to increase. When it comes to analytics, you can choose the resolution you want to store the data. This can be the raw events as they come in, or rolled up into for all activity that happened in a particular period such as every minute or every hour. Rolling up a metric can drastically reduce storage requirements, but it sacrifices metric resolution. stories Many analytics systems have a tiered roll up strategy. For example, all data in last 7 days is stored down to the event resolution or on 5 second intervals. Older data greater than 7 days ago can be rolled up into hourly intervals. While even data that’s older than a year ago may be rolled up in daily intervals. It’s unlikely you’ll be looking at events down to the exact timestamp. colder Metric Estimation Sometimes you don’t need an exact count and can live within a certain level of error. Let’s take the example calculating weekly active API keys. Let’s say you have an ordered API log that contains the request time stamp and the API key used to access your API. A naive approach would be to create a map indexed by the weeks and then for each , create a set that contains the unique keys for each bucket. bucket An issue with this approach is you can quickly run out of memory if you have a large number of unique API keys (also known as high-cardinality). A different way to approach this would be leveraging a probabilistic data structure to infer or estimate the uniqueness of the set rather than directly counting. Example probabilistic data structures include and . HyperLogLog Bloom Filters After all, you probably are interested in if there was 100 or 10,000 API keys active last week, but if your analytics system reported 101 instead of 100, your decision making wouldn’t differ. For this case, we can leverage HyperLogLog which approximates the number of unique values by estimating the number of leading zeros when the API keys are hashed together via a hashing algorithm. Accuracy is dependent on how many bits are allocated for this bit vector required to store the hashed values. Many modern databases for analytics like Druid and Elasticsearch have HyperLogLog built in, but it’s up to you to ensure your data can leverage them. Previously published at https://www.moesif.com/blog/technical/scaling-analytics/How-to-Design-API-Analytics-Data-Collection-for-High-Volume-APIs/

How to Design API Analytics Data Collection for High Volume APIs

Gatsby and Drupal : Match made in heaven? Gatsby is a popular static site generator that can communicate with any backend. The front-end landscape has exploded in the last three years. Today you have various libraries/front end frameworks like React, Angular, VueJS. You have tightly coupled full stack frameworks NEXT, NUXT etc. Of all these options Gatsby finds a sweet spot with its JAM stack approach. JAMstack is “a modern web development architecture based on client-side JavaScript, reusable APIs, and prebuilt Markup.” In this article we will discuss how we can make use of the JAM stack using Gatsby and Drupal. We will also cover the general questions that are generally not answered in the blogs. Gatsby is static site generator, so what are the scenarios where I can use Gatsby. My site has a lot of dynamic content, how do I make it work well with Gatsby. I have read that Gatsby is based on React, so how can I decide when to use Gatsby and when to use React only front end. Can I use gatsby to progressive decouple my Drupal website? How can I configure Gatsby so that most of the configurations are read from the backend and a parity is maintained with current existing website. What is server side rendering and how can I leverage it with my Gatsby and Drupal set up. Is Gatsby good only for the anonymous user or can I leverage it for authenticated flows as well? Setting up OAuth on Drupal and leveraging the same for Gatsby. What is the relationship between jsonapi and graphql. Setting up Gatsby What are the other alternatives that play well with Drupal. Should I use Gatsby or Tome? Feedback There are many todo articles that already explain how to get started with Drupal and Gatsby. I will just link to them here instead of repeating and answer the questions that I had when following them. If you have already made your decision just visit the Setting up Gatsby section. What is JAM stack JAMstack is “a modern web development architecture based on client-side JavaScript, reusable APIs, and prebuilt Markup.” In a crude analogy you can think of it like the static files deployment that you were doing if you were in software development say a decade ago. This is not entirely true but JAM stack tries to combine the ease of static sites along with the dynamicness provided by APIs. Javascript is used for the client side interactions, better handling of dynamic rendering of elements on the client side and also for interaction with the backend if any. Think AJAX. Javascript : Any interaction with the backend is abstracted into reusable APIs. One more advantage is that in the front end you can make use of your own APIs as well as any third party APIs. APIS : : You will using the Markup or HTML/css for the front end. In JAM stack markup is generally prebuilt at the deploy time. Markup Gatsby is static site generator, so what are the scenarios where I can use Gatsby? Gatsby is a static site generator. What this means is that the public folder that is created during the build time will function as a static website. So you can take that folder and deploy it on any server with Apache or nginx like server to serve the requests. For example if you use the default starter kit of Gatsby communicating with Drupal backend, you only need the backend to be up during the build time. Once your build is done, you can down your Drupal server and just deploy the public folder generated during the build. What this means is that your Gatsby site will communicate with your Drupal backend during the build time, fetch all the details necessary and create static files for all the paths that will be present on your site. My site has a lot of dynamic content, how do I make it work well with Gatsby. Think of Gatsby as a state-machine. The state consists of the code changes you make and also the data you save in your database. So whenever there is a change in the code or the data in the database, you need to rebuild it for the latest state. For code changes it is easier to trigger the build. For the changes in database it is a little bit difficult. But you can make use of web hooks to trigger these builds. But the question is how frequently do you trigger your builds? Let us say that you can stay with some stale data, then you can have builds every few hours. This might be suitable for blogs where if the content is refreshed every couple of hours/days it should be sufficient. But if your site has lot of user interactions then frequent rebuilds might be a pain and it will also add load on the server. In such scenarios it would make sense to keep the content that changes less frequently in a backend like Drupal and for the content that changes frequently(like comments) it might be better to off load to a third party service(like Disqus for comments). I have read that Gatsby is based on React, so how can I decide when to use Gatsby and when to use React only front end. If you would like to have control of all your data(both that changes less frequently and more frequently) then it might make sense to do the Gatsby build for content that changes less frequently, then add React components that interact with live API calls for content that changes more frequently. Since Gatsby is built on React, you can stick to your standard API calls in componenentDidMount and then render the results in render function. Can I use gatsby to progressive decouple my Drupal website? If you are not sure whether to use Gatsby for the whole of your site, then this approach can be good. For example let us say that you have a website that has many functionalities including blog, you can just transfer the client facing blog functionality of your website to Gatsby. In your apache/nginx config just make sure that only is handled by your Gatsby front end and all other parts are handled by Drupal. www.example.com/blog How can I configure Gatsby so that most of the configurations are read from the backend and a parity is maintained with current existing website. One simple approach I took was to make sure that I have Drupal backend at backend.example.com and front end at gatsby.example.com In my gatsby-node.js I added a rule like result.data.allNodePage.edges.forEach(({ node }) => {
                    createPage({
                        path: node.path.alias,
                        component: staticPageTemplate,
                        context: {
                            slug: node.path.alias,
                        },
                    })
                }) This makes sure that even in the gatsby front end I can use the path set in the backend. So for example backend.example.com/blog/firstblog to gatsby.example.com/blog/firstblog This way it was easier for me to map and check the missing pages from the sitemap.xml as well. What is server side rendering and how can I leverage it with my Gatsby and Drupal set up. Gatsby comes with a default SSR settings. What it means is that API calls to your data sources are made at the build time. So all the pages are server side rendered. Once you are done with a build you can pretty much shut down your backend unless you have any real time API calls that you need to do from your Gatsby app. Is Gatsby good only for the anonymous user or can I leverage it for authenticated flows as well? By default Gatsby plays well for read only kind of websites. But there is nothing that stops you from creating forms and authenticated user flows as it internally used React. Connecting Gatsby with Redux might be good for authenticated flows. Checkout for Gatsby with Reduct integration. If you want to add authentication to your Gatsby sites checkout https://medium.freecodecamp.org/how-to-get-started-with-gatsby-2-and-redux-ae1c543571ca https://www.gatsbyjs.org/blog/2019-03-21-add-auth0-to-gatsby-livestream/#authentication-in-gatsby What is the relationship between jsonapi and graphql. In Drupal jsonapi is a popular module. There is also a graphql module. But if you are using plugin you will observe that on the Drupal end you just enable the jsonapi module thats all. What is happening is that the the GatsbySourceDrupal plugin is taking care of the jsonapi output and converting it to graphql compliant structure so that you can actually use graphql queries to access your data in Drupal. You are generally asked to enable the jsonapi access to all user. Be careful as you might be exposing some confidential data that is not intended. Do check these out before deploying in live. https://www.gatsbyjs.org/packages/gatsby-source-drupal/ What are the other alternatives that play well with Drupal. Gatsby is a great tool. It makes starting with REACT and Drupal so easy. It is also highly configurable and customisable. But it can still be overwhelming for Drupal developers who have not worked with Frontend frameworks earlier. Thanks to open source we have other alternatives. Tome is another interesting project that generates static sites for Drupal 8 quickly and without needing to know the new frontend libraries and tools. In ’s own words https://twitter.com/DrupalSAM It’s important for me to make Tome more accessible to less technical users. Anyone that can make a Drupal site should be able to make a static site without learning a completely different toolset and programming language. I think this is a great goal and will help Drupal community immensely. Once people realise the advantage of JAM architecture I am sure most of the blog like sites will default to static sites :) If you don’t trust me, just checkout the video in below tweet to realise how easy it is to create static sites with Tome. body[data-twttr-rendered="true"] {background-color: transparent;}.twitter-tweet {margin: auto !important;} Working on an in-site preview feature for Tome - soon you'll be able to download, generate, and preview your Drupal static site without touching the command line. 🖱️✌️ https://t.co/qmXYMufgkG — @DrupalSAM function notifyResize(height) {height = height ? height : document.documentElement.offsetHeight; var resized = false; if (window.donkey && donkey.resize) {donkey.resize(height); resized = true;}if (parent && parent._resizeIframe) {var obj = {iframe: window.frameElement, height: height}; parent._resizeIframe(obj); resized = true;}if (window.location && window.location.hash === "#amp=1" && window.parent && window.parent.postMessage) {window.parent.postMessage({sentinel: "amp", type: "embed-size", height: height}, "*");}if (window.webkit && window.webkit.messageHandlers && window.webkit.messageHandlers.resize) {window.webkit.messageHandlers.resize.postMessage(height); resized = true;}return resized;}twttr.events.bind('rendered', function (event) {notifyResize();}); twttr.events.bind('resize', function (event) {notifyResize();});if (parent && parent._resizeIframe) {var maxWidth = parseInt(window.frameElement.getAttribute("width")); if ( 500 < maxWidth) {window.frameElement.setAttribute("width", "500");}} Should I use Gatsby or Tome? As with most answers to the technical questions, that answer is “It depends” :P But here is the thumb rule I use. If you are not comfortable with Frontend and you just want to use JAM stack or static sites for the performance sake Tome is a great choice. There is not much addition learning curve. If you want to leverage the fast pace of development in REACT eco-system but don’t want to build things from scratch, you can leverage Gatsby. Setting up Gatsby Before setting up Gatsby I would recommend going through the following two videos. They give you a high level insights and should be sufficient to get you started quickly. Understanding Gatsby Internals for Drupal If you would like to understand the details of how Gatsby works and how to customise it for Drupal, you can check out this blog by Ryan Bateman Tutorial: GatsbyJS for Drupalers; or, How to JAMStack-ify your Drupal Site with GatsbyJS Things I wish I knew in Gatsby before getting started Gatsby also uses the term node. This is different from the node used in Drupal. Any atomic data that Gatsby handles can be considered as node. So in Gatsby lingo Drupal nodes, categories, taxonomy terms and users can also be considered as nodes. There is no rule that each Gatsby node needs to have a page. You can decide for which nodes you need to create pages. Feedback I hope this answers some of your questions before shifting to Decoupled Drupal site or deciding on the front end frameworks for your Drupal API backend. If you have any questions that I am not able to answer just let me know about them in comments. We can explore them together and share our notes for mutual benefit. Let us make this as a repository for all the commonly asked Questions by Drupallers regarding headless option.

Gatsby and Drupal : Match made in heaven?

Hey Hackers, 2 years ago had been in HackerNoon's annual Startup of the Year awards in Santa Fe, USA. liveapp nominated As a result of voting, we took first place in the city’s nomination 🎉 In light of Startup of the Year 2023 I want to remember how it was and how we made it Meet liveapp liveapp is an app to watch videos nearby. Here people around you share what they are doing right at the moment and you can instantly see it On liveapp, people don't waste time on filters, masks and effects. They share only genuine records where you can catch undisguised emotions, exciting events, new places and local news from your city, country, or all around the globe My Role My name is Vitalii Vinogradov, I am a product manager and founder of the app I define the strategy and lead the team to make a better product for the users The idea didn’t come out of nowhere. I had experience working with a similar app before which helped people to find restaurants nearby, see the menu and book a table. When Tik Tok blew up we found out that video could be useful not only as entertainment but a source of useful information about local events and places. That’s how we started liveapp How We're Disrupting/Improving the video Industry In our studies, something like almost 40% of young people, when they’re looking for a place for lunch, they don’t go to Google Maps or Search, they go to TikTok or Instagram But it’s still too difficult to find a video post on something happening nearby because modern social apps don’t allow filtering content by geolocation effectively A nearby video app allows you to quickly watch videos captured at the current time and place with minimum retouching on the local news feed. Think about it as a video-only Google Maps or geosocial Tik Tok Standing Out from The Crowd That’s exactly what you can do on liveapp but cannot on the other apps watch unfiltered "lives" up to 30 seconds search videos by precise geolocation and exact areas explore geo-bound hashtags and unique "meetags" follow your favorite places as well as people like and comment on the best videos enjoy the unique minimalistic interface Today we have a rapidly growing user base, but that’s still only a beginning Our Predictions/Thoughts on the video Industry in 2023 Gen Z people rely more on video content to find places and things to do around rather than on classic search apps like Google Search Despite a short drop in interest in the last year due to the economic crisis the trend definitely will continue to grow making videos a more attractive source of findings Why we decided to participate in HackerNoon's Startup of the Year awards We were young and we need users) Just kidding, HackerNoon's Startup of the Year is a nice chance for a young startup team to tell about themselves and be heard Final Thoughts Thanks for reading and good luck in the 2023 competition 🏆 Take care!

Startups of the Year 2023: How Liveapp Got a Startup of the Year Award 2 Years Ago

213 Stories To Learn About Writing Prompts

311 Stories To Learn About Noonies

Hey HackerNoon Readers! I’m and I’m a Product Manager at and a Founder of . Vitalii Vinogradov Revolut liveapp I want to thank the HackerNoon community and staff for their nomination in the as well as for sponsoring the Emerging Tech category of the 2022 Noonies awards. Emerging Tech category, .tech domains I’ve been nominated for the following awards and if you like my writing, please do check out these award pages and vote for me: DEEP-LEARNING Learn more about my thoughts and opinions on the emerging technology of 2022 and my journey in the tech industry via the interview below. 1. Please tell us your story. What do you currently do in tech and how did you get started doing it? I started my path in tech more than 8 years ago by building an online shop with my college friends using a WordPress template) Many things have changed since that time and for the last 3 years I’m working mostly with biometrics and digital ID products, currently at where I’m working on digital onboarding experience for more than 20 million of Revolut’s users Revolut Also, I’m a founder of , the momentary video app which helps you to share what’s happening around you instantly liveapp 2. Tell us more! What are the things you make / write / manage / build? Launched one of the world's first Cloud Face Recognition platforms at (3.5 million users). Delivered the world's best biometrics products at (G20 best AI startup). Prepared hundreds of students as a product management teacher. Launched my own startup - . Building digital onboarding at (The World’s Top 10 Most Valuable Startups) Ivideon NtechLab liveapp Revolut 3. If we gave you 10 million dollars to invest in something today, what would you invest in and why? Basically, I would diversify my assets across my own startup (small communities social tech) and third-party shares if it is possible. In third-party ones, I would diversify my portfolio between AI, Web3 and VR/AR-related companies (see below why) 4. What do you think is the most exciting development in tech right now? And on a personal level, what tech are you most excited or passionate about? In my opinion, there are four most significant tech trends today First, - we can automate many operations now which seemed unsolvable by machines only 10 years ago. Starting from simple text recognition to full image generation. Machines can do a lot of things faster and more effective than humans and can help us to be more productive AI/ML Second, - with the blockchain we can create systems that require less hand control and can’t be changed by mistake or malicious intent in a non-transparent way. Combining with AI we can do software that will perform complex human-kind operations (like image recognition or text writing) but in a completely clear and trusted way without the need for control or verification of the true intentions of the operator Blockchain/Web3 Third, - the Web is becoming a more inclusive place. You can now use not only a classic mouse or touch screen to browse Web but also way more interesting devices like gloves that catch all your movements or glasses which add a layer of additional information into your view VR/AR And, finally, - here I want to tell a few words regarding social tech evolution. I think the small communities trend (by which I mean apps where people communicate in close circles of colleagues, neighbors, or by common interests"; see and as recent examples) is emerging which means that our Web life becoming more personal and mixed with the real one Small Communities Clubhouse BeReal 5. What are your concerns about how fast tech is evolving? Of course, we as humankind have some risks with such fast tech evolution mainly because it’s possible to skip some important ethical or security parts of what we are building (hello, Matrix and Machine war). But the results we get in terms of growing people's productivity are unbelievable in my opinion so the risks are covered enough and let’s be optimistic about the Machine war) 6. Which tech company do you trust the most? Which do you trust the least? Why? I trust independent companies first of all, which don’t rely heavily on the governments and focus on changing the world for the better rather than making money from people (money is important as a strict success indicator but not a mission or goal) 7. What are you currently learning? Recently was released by Los Alamos National Laboratory. That’s my first book on a queue to read on a weekend) the first quantum computers programming guide 8. What’s the best advice you’ve ever given someone? Don’t waste your time on things that don’t move you forward 9. What’s the best advice you’ve ever received? Don’t waste others’ time on things that don’t move them forward

The Future of Web3 With Vitalii Vinogradov, Noonies 2022 Nominee

Into Space

En el espacio

Dans l'espace

अंतरिक्ष में

宇宙へ

No espaço

Trong không gian

进入太空

Meet the Winners of Startups in North and Central America

Hey Hackers! I’m and I’m the co-founder of Vitalii Vinogradov liveapp is an app for instant information sharing in short video format, where thousands of users can instantly post what they see or thought right now at the current moment. liveapp First of all, I’d like to say a huge thank you to the community for recognizing us as the winner of: HackerNoon #1 Startup of The Year 2021 in Santa Fe, NM Reporter: What does it mean for you to win this title? HackerNoon That is a little step forward for all our team and a piece of recognition by the internet community. That’s very important for an early-stage internet company like we are. How do you or your company intend to embrace the responsibility of this title in 2022? We’ll just try to build the app that will help to connect people from different countries and cultures into a single worldwide community.) What are some of the goals you’re looking forward to accomplishing in 2022 (whether it be through company initiatives or your personal journey)? Our main goal for 2022 is to grow the community by a few times and finally get a strong market fit. I think the award will help us to reach more early users and great advisors to achieve this. I think the main trends that will drive the web for the next 10-20 years are as follows Artificial Intelligence and Machine Learning -  will help to automate many processes that might take years previously including very creative ones like content creation. VR and Metaverse - will extend and speed up our communication and experiences online. Blockchain - will make the whole web more decentralized and secure. After participating in this journey, is there anything that you would like the team to know? HackerNoon I think the initiative itself is great. It helps even the smallest team from the smallest city to get some recognition from the web. What is something special about you or your company that you’ve rarely had the chance to properly share but really wanted to? (Give this one a proper think, I’ll know if you didn’t!) It’s interesting that we started as a completely remote team and some of us even don't see each other in real life. But that doesn’t prevent us from being a great and effective team. That’s what I call the power of the web! What song or music album are you currently enjoying? I like Latina and Afro music. The last one I liked is . Goya Menor, Nektunez – Ameno Amapiano Remix I think you already saw many short videos with that on social media. What are some concepts you wish people knew more about? Just want to share some more info about . liveapp is an app for instant information sharing in short video format. Here thousands of users post their original momentary videos and you can become one of them liveapp In , we don't waste time on filters, masks and effects. Only live videos where you will see undisguised emotions, interesting events, useful information, and news from around the world. liveapp Learn interesting and useful things every day. Share moments from your life. Show your personality through instant videos. Join now! liveapp Is there anything else you’d like me to know? It can be anything. Thanks for the interview and for the fair competition! For more information: Read about the Noonies . To see all of the winners you may visit , pick any award, and view the name that appears at the top. Or follow for posts with the list of names released over the course of the week of February 14, 2022. here this site this account Read the FAQs about or . To see all of the winners you may visit , pick any city and view the name that appears at the top. Or follow for posts with the list of names released over the course of the week of February 14, 2022. Startups here here this site this account Thank you to all of our sponsors! We truly appreciate your contribution to the tech world and thus the future! : Startups , , , , , , , , , , Blockster Brex FreeCodeCamp LaunchDarkly Coil Udacity Udemy Vercel Asia Pacific Transgender Network Bybit Everscale Network : Noonies The Noonies Awards are sponsored by: , , and hosted by: Bybit Avast TECH Domains HackerNoon

Thrilled to be Recognized as the  #1 Startup in Santa Fe

The first time I laid eyes on a cold blockchain wallet, I was stunned. It wasn’t just the novelty of blockchain technology that shocked me, but the inherent vulnerabilities of wallets currently on the market. Where others saw secure pieces of equipment capable of safely storing millions of dollars in digital assets, I saw HSM formats, USB, ethernet cables, QR codes, and SD cards. In other words, I saw weak points that hackers could use to penetrate the wallets’ defenses, including WI-FI and bluetooth connections that made the notion of “air-gapped” computers laughable. I knew that each and every one of them was hackable. This was how millions of people around the world were holding their cryptocurrencies? There had to have been a better way. Sure enough, we discovered hardware and software vulnerabilities in several leading cold wallets, namely, the ability to manipulate the front-end application to show misleading data for the wallet holder during a transaction. Indeed, any device that has some kind of connection to the internet throughout the transaction process (all wallets do) is certain to experience a breach at some point. As such, referring to such wallets as “cold wallets” is misleading, because the modifier “cold” implies a lack of internet connection. Sure, many companies that claim to provide cold wallets make life harder for hackers, but at the end of the day, with enough research, resources, and will power, malicious actors will find and exploit the loopholes. With this knowledge, how is a wallet provider able to build a successful bounty program for its product that it truly believes to be unhackable? Part of the trick lies in providing the information hackers need in order to exploit vulnerabilities -- if the hacker never has a chance to hack, was it really a bounty program? At GK8 provide a step-by-step guidebook for building a truly convincing for wallet providers who seek to follow in our footsteps. bounty program STEP 1: Provide an incentive to lure in the big-league players Firstly, it’s important to lure in the top hackers in the world with a major incentive to prove your wallet can truly withstand any attack. We did this in two ways. , we offered a prize of $250,000, one of the largest bounties ever offered for such a program. The prize offered needs to be high enough to tempt the bad guys -- generally known as black-hat hackers -- who know what they’re doing. First Their hack attempts, and failures, only serve to bolster your company’s claim of providing a truly unhackable wallet. -- Bitcoin, Bitcoin, and Bitcoin. There is nothing that will attract state level actors more than offering the bounty prize in Bitcoin, because it’s a one-stroke, untraceable steal for the hacker if they are able to break into the wallet. We knew they wouldn’t be able to, but the idea is for them to try Second, third, and fourth We knew hackers would make the attempt, as the inherent weaknesses of blockchain protocols are well-known in cybersecurity circles. Blockchain is interactive in nature, as users must use wallet end-points to conduct transactions. In other words, any wallet can be compromised because it needs to have internet connectivity at some point, which opens many internet attack vectors. STEP 2: Build cyber defenses all around your company’s network Companies running a bounty program should consider the possible collateral damage, which can include DDOS attacks, ransomware attacks, and reputation damage. Companies should prepare for attacks on their networks, websites, or social media profiles. Failing to secure high-risk, sensitive data on company platforms that are hackable can single handedly ruin the entire program. From those platforms, hackers can potentially find their way to company workflows, as well as the way the product is designed. Companies must heighten their internal cybersecurity defenses before announcing. STEP 3: Prove the cold wallet is live Before a company goes live with its bounty, it needs to publicly declare that a live Bitcoin transaction will be executed at an expected time so hackers can forecast exactly when and to whom will the transaction be sent. Why is this important? Because it tells bounty participants when the device will be active and gives them the ability to exploit internet exposure and leverage a security breach within the wallet. A company can’t declare its device truly hack-proof if it doesn’t give hackers an actual shot at breaking into it. If I were an outside observer, watching our bounty program from the sidelines, I’d think we’re nuts, marching willingly to the gallows. Publicly disclosing the company’s physical address, as well as the sender and receiver address and the exact time the transaction is taking place a week in advance? Giving the hacker all that time to plant malicious code that can be activated when the transaction is executed? These guys can’t be serious, I’d think. And yet, we did it. There are many wallet providers who claim their solution is “offline.” While that kind of descriptor might appeal to marketing types, it induces skepticism among security experts. Blockchain is an interactive protocol: In order to sign transactions, you must communicate with the blockchain. That means that even if a wallet is generally not connected to the internet, it does connect the moment it needs to sign a transaction, exposing the risk-sensitive private key to the internet and opening the wallet to a vector attack. We, at GK8, were able to send a transaction without an internet connection through our technology. The idea of the bounty was to prove to skeptical security ears that we’re able to do so, for real. First, we needed to eliminate the possibility of such skeptics suggesting that we predefined the transaction details -- meaning the address, amount, and signature -- before the day of the event, because it is possible to do so. To ensure no one could accuse us of such meddling, as well as prove the liveability of the cold wallet, we built a formula so that no one on GK8’s staff could predict the address in advance, meaning we had to send it in real time, during the bounty. During the bounty, , we sent a real-time transaction to the address of the miner that closed the last block. In doing so, we proved that we had to generate the bitcoin addresses in addressee information in real time -- there was no manipulation of the results involved. every 24 blocks STEP 4: Protect against physical theft -- We kid you not A sum of $250,000 in Bitcoin is quite a trophy. As such, we had to take every step necessary to ensure this money would remain safe. It’s important to gather security experts -- the best the field has to offer -- in order to map out all the possible doomsday scenarios. Now, to be clear, I wasn’t at all worried about the product. When we say our cold wallet is hack-proof, we mean it. Still, we needed to act responsibly in order to protect our personnel and company infrastructure. Yes, that’s right. We even considered the possibility of an armed attacker physically entering the office and taking someone hostage for the money. In order to prevent that from happening, GK8 locked down its office, escorted staff to and from bus stations, and prepared a secure location with second-layer physical protection to lower the risk of physical attacks. There weren’t any physical attempts to rob the company on game day, although there was a minor-league hack attempt on GK8’s company network that was handily thwarted. Step 5: Bask in the success A couple hundred of hackers from all over the world -- including Russia, England, Israel, Ukraine, Turkey, and North America -- signed up for the bounty. This only accounts for “white hat” hackers who actually provided personal details to claim legal holding on the prize if they were to succeed. It doesn’t count for the black hat hackers that didn’t sign up, but still took a stab at hacking our cold wallet. GK8 is aware of thousands of users globally downloading, viewing, and going through company information. There were attempts to hack our business network, but nothing happened. None of the attempts to break into our wallet were successful. That’s because our cold wallet is the only true cold wallet on the market, as it is able to send a transaction without direct or indirect internet connection. That means it’s physically impossible to cyber-hack our wallet. But the bounty journey doesn’t quite end there… Step 6: PR, PR, PR! Once a company has shown that none of the thousands of hackers that tried to crack your wallet failed to do so, it needs to make it known to the world. In fact, public relations and marketing efforts should begin before the bounty even hits the ground running. That’s how you get attention. Companies shouldn’t be afraid to reach out to everyone, even directly. As long as you’re confident in your product, all PR is good PR. TechCrunch’s Zack Whittaker remains skeptical even after our successful bounty program. But, as he’ll eventually come to understand, our cold wallet is truly hack-proof. At GK8, we’re serious about changing the world of security and blockchain, making it possible for financial institutions to utilize blockchain technology’s benefits confidently and safely. We challenge others with the same aim to run equally successful bounty programs in the future, using the above steps as their guide. Good luck! About Lior: Lior Lamesh is the Co-founder and CEO of GK8, a cybersecurity company that offers high-security custodian technology for managing and safeguarding digital assets. Lior  gained his expertise in cybersecurity while serving in an elite team that answered directly to the Israeli Prime Minister’s Office on matters of strategic state assets protection.

Takeaways from Our $250,000 Bounty For Hacking Our Cold Wallet

Adoption of new innovative technologies coupled with a fresh look at current regulation offers an enormous opportunity for future progress and prosperity for the video industry. The complexities that exist in today’s video marketplace are screaming out for new solutions to meet the evolving consumer demands and technological requirements. However, rapid changes present many challenges and opportunities as the ecosystem continues to transform in the digital age. And while the problems are extremely complicated, I believe the overarching challenges in the current video distribution infrastructure can be analyzed in four distinct categories. Multiscreen and Multichannel Distribution Remember when people watched videos only on TV? Well, even though that simple reality was relatively recent, we’re living in an entirely different climate now. These days, media companies must make their content available over multiple video streaming devices and multiple platforms to remain competitive in the ever-changing market. Whether it is about watching video on demand or live streaming, most consumers name convenience of on-the-go consumption and easy access to their content anywhere and anytime as critical factors while making decisions about what media services provider to use. The tendency seems unlikely to change over the next few years. The new forecasts that from 2017 to 2022 mobile video traffic will grow 6 times only in the United States, while video will be 80% of the US mobile data. Visual Networking Index (VNI) by Cisco On the other hand, for video content owners and distributors the diversity of targeted devices and platforms means endless variety of encode formats, levels of video quality and different bit rates. This naturally results in complicated workflow and increased costs. Thus, developing and approving of standardized requirements would allow the creation of multiply versions of video content while saving highest possible quality should be definitely a high priority task for the industry leaders. Meanwhile, some platforms are already taking steps towards solving the growing issue of multiscreen and multichannel distribution and simplifying relationship between video production and delivery. has been launched to help creators meet Netflix’s technical and delivery specifications and make a more efficient workflow from production to post-production and distribution. offers a cost-effective method for the creation of broadcast quality video and audio with the ability to be streamed live to multiple platforms, including Facebook, Twitter, and YouTube. allows users to broadcast a single live video stream to all of a user’s social channels, while Telestream’s provides the creation of live or on-demand broadcasts for the web. The Netflix Post Technology Alliance Trickbox LivePin Wirecast Rights and License Management Exclusive content is another driving force in the video distribution business. While competition on this market is high with only about 200 SVOD options in the USA, companies invest money and efforts to fund original content. For instance, Ted Sarandos, chief content officer at Netflix, reported that around 85% of the company’ new spending is going to original content. In the lucrative world of , the changes are staggering as well. Twitter acquired specific NFL rights for the 2016 season, while Amazon Prime Video gained other rights for 2017 for its coverage of the ‘Thursday Night Football’ games. Although these acquisitions were for digital distribution with the idea of complimenting traditional broadcasts as opposed to being substitutions, the competition growth is certainly clear. sports broadcasting Obviously, in this situation content owners want to take full control over the rights usage. The market offers plenty of out of the box software for Digital Rights Management (DRM), however, most of them require customization to fit all users systems. Moreover, because of the variety of formats and platforms, content providers often have to support multiple DRMs. And even with all the latest technological advancements content protection and rights management still require a lot of manual work. This vital need is forcing major market players to look for technological partners and improve their current tools and develop new ones. Lack of Accurate Data How can viewership be tracked in the current ecosystem? Industry leaders are struggling to answer this question, as multiscreen and multichannel consumption creates massive gaps in data collection. Here’s the thing: although more video content is watched now than ever before, the ability to analyze viewership is diminished. As a growing number of consumers choose the “cord-cutting” of their traditional cable TV, the industry complexities increase. However, these same challenges offer incredible opportunities to leverage the plethora of available data to create better experiences for viewers through personalization and other means. I expect that we’ll see a range of tools emerging in the coming period to allow companies to effectively collect, analyze, and utilize the invaluable information derived from the viewing habits of consumers. Anyhow, effective data collection is not the key to success. To make the best possible decision it is important to understand what is the right data you need and learn how to understand it. Sebastian Wernicke, Chief Data Scientist at One Logic, illustrated how Netflix and Amazon used data when building their strategies. Data Security and User Privacy Protection Video content piracy is at an all-time high. Professional digital pirates pose a major risk to media companies, with the threat of posting content before its release date through hacking into computer systems. Many industry experts consider content piracy to be the most significant problem they face. From the growing need to effectively protect confidential data to the desire to maintain the originality and authenticity of content, media companies are in a difficult spot. The Online TV Piracy Forecasts report by that the US will remain the largest country by piracy losses, with $11.6 billion forecast by 2022, up by 30% on 2016. Digital TV Research predicts Video content distributors have to ensure content is secured wherever it is stored or whenever it is streamed. Encryption, GEO and IP restriction, password protection and watermarks are just some of the basic security technologies which are must have for any modern video platform and DRM systems to protect content from unauthorized access and usage. At the same time, companies should be concerned not only about their own security. While many consumers believe that sharing personal data is a part of the modern digital environment, different surveys indicate that many people do not trust companies and their ability to protect users personal data. Fortunately, industry leaders and their tech partners are working diligently to improve security measures across the board, with the goal of regaining full control of media content for its rights owners. To say that the video industry is facing challenges would be a wild understatement. Truly, the complexities are massive and the ecosystem is changing so rapidly that it’s difficult for companies to keep up. At the same time, the top minds in our industry are working with the best technology firms to not only solve the problems that exist, but to develop a new infrastructure that contains the necessary flexibility to grow along with the landscape changes, thereby shifting the challenges to become opportunities for us all. What are your thoughts about video distribution challenges and opportunities? Please share your opinions in the comments below. By Sergey Bludov , Senior Vice President of Media and Entertainment Practice at DataArt

Video Distribution: Challenges and Opportunities in an Unknown Future

The world is evolving at a rapid pace. Never before we have witnessed so many changes happening in a such short period of time. Economic, social, technological and political shifts are reshaping the world very quickly and new challenges arise for nations and particularly for cities . As governments are seeking to incorporate innovations within their cities, blockchain can offer something more. Photo credit : https://dailyhodl.com Global trends thus are deeply concerned by global shifts and have to take in consideration every changes in order to become a better place to live and offer a wide but successful range of innovative services to its citizens_._ One of these changes is that has been accelerating in the last 15 years. According to the World Bank, . Cities count for more than half of the world’s GDP urbanization more than 54% of the global population now lives in cities This phenomenon leads to a deep transformation: important . Indeed, population is continuously growing with a part of the elderly dangerously high. Today, . With 2030 at the corner : 1960’s baby-boomers will definitely make that elder category a matter. demographic and social changes 9% of the population is above 65 years old we’ll reach 8.5 billion people The way we live today has also a terrible . Climate change is a true concern and resource scarcity has been identified as a global trend by the World Economic Forum. impact on the environment But the most evident shift is the that impacts in every aspect the global economy. Corporations are spending more and more money in R&D ( ) as they identified innovation as an important key growth driver. Technologies have invaded our everyday life, making it more connected and faster than ever before. Every industry has been impacted by new production cycles, by new ways of consuming, by ubiquitous technologies and relentless communication between people, all of them triggered by disruptive innovations. technological breakthrough some of them almost 14% of their turnover Due to these global shifts (urbanization, demographic changes, environmental issues and technological progress) making cities smarter has been . Since cities are the primary environment for a nation’s development, making them smarter, more collaborative, more inclusive, more sustainable and more connected is an utmost goal in order to provide better services and a high quality of living for the citizens. a challenge for governments to offset the negative consequences and take the most out of these changes What is a smart city ? Techopedia.com defines a smart city in these words: “ ”. Put simply, it is a city using technology to enhance . A smart city is a designation given to a city that incorporates information and communication technologies (ICT) to enhance the quality and performance of urban services such as energy, transportation and utilities in order to reduce resource consumption, wastage and overall costs liveability “A smart city is an implementation of an advanced and modern urbanization vision” — Rakesh Kaul, Partner at PwC Smart cities allow operational efficiencies, maximize environmental sustainability efforts and create new citizen services. Smart cities can become an enabler to counter downsides of urbanization and demographic explosion that impact the environment. Indeed, the ever increasing population jeopardize the basic needs such as a clean habitat, water, energy and infrastructure. Pilots and projects undertook We have seen initiatives across the globe to develop such kind of cities. India, the second most populated country in the world, therefore very affected by this topic has launched in 2015 its Smart Cities Mission aiming to develop 100 smart cities in the country. and is intended to promote sustainable and inclusive urban development that would match and secure India’s cities expected growth rate. To make this happen, a lot of use cases have been thought of. From medical services to transportation, every cities will have to improve its services and face urban challenges such as shortage of power, insufficient water supply, unaffordable cost of living, inadequate public transportation, and pollution. This five years program will cost $29.9 billion USD Smart cities have to rethink the way they manage the ecosystem to provide better support and increase the quality of living. The following areas can be enhanced to make a city smarter: Citizen identities management and citizen participation Payment system between people and organization Employment Health Culture Transportation Environment and space Energy and waste Land management But concretely, if you had all the authority and unlimited resources to change your city, what would you do? Several pilots have been conducted in some places: 👶 — government partnered with Telenor to overcome laborious birth and death recordings by launching a civil registration system in which is integrated a platform updated by several authorized parties when birth and death of Myanmar’s citizens occur. Myanmar 🏥 — In general, the healthcare system is a very complex structure with lots of actors and sharp ethics issues, thus a field where innovation can trigger some changes and make considerable gains in time and efforts. Remote patient monitoring with sensors can, as one example, detect blood and glucose levels in patients, and send the resulting data to doctors for analysis and prognosis. Healthcare professionals can analyze all of the data collected by sensors to prescribe highly personalized treatments and medications for patients. You may even be able to 3D print your pill to take at home with no interference at all to your day-to-day routine. This example will soon be experienced is the new innovation-driven health hub currently being built in . Tons of medical services powered by AI or 3D printing will be provided to patients from all over the world. Dubai 🚋 — With huge amount of data collected on citizens’ movements and travel, the transportation system could be improved so that it would match the population needs in terms of routes and schedules. Powered with big data, transportation optimization is not so far-fetched. The Joint Transportation Management Center based in (the largest transportation center in North America) is using hundreds of cameras, vehicle detectors and advisory radios to manage congestion on expressways, predict incidents, strengthen on-the-spot intervention and reduce delays. New York Basically, thanks to innovation, anything can be transformed into a more efficient manner. With new enablers and tools like Internet of Things, 3D printing, big data, connected devices and wearables it is more easy for cities to transform themselves. But one enabler can bring out something more: . blockchain Blockchain goes hand in hand with the smart city concept Think about it, in the 3 aforementioned use cases realized in Myanmar, Dubai and New York, several common characteristics emerge: ✔️ Data is shared between multiple parties ✔️ Data is updated by multiple parties ✔️ Data is centralized ✔️ There are requirements for verification The internet has already brought a lot of possibilities for cities to become more efficient, moving manual services to digital services and storing information paperless. And although the use cases listed above are very effective for a smart city, one feature can be added to make it smarter. A feature which would bring decentralization and which would erase intermediaries. Above all, a feature that would bring security among the systems and interoperability between them. You have guessed, this feature is blockchain. Blockchain, as a reminder, is a secured, distributed and decentralized database that multiple parties share transparently to ensure accuracy of information with no risk of alteration or clearance of the database. Since blockchain is a pretty new technology, governments might not have seen yet the potential it can provide when building smart cities. It is indeed a new tool to help them secure and deal with information and transactions. Also, that will become more frequent in the digital age. blockchain can decrease the risk of systemic issues related to data security, inefficiency, lack of transparency and corruption Governments that have taken the lead Some governments have pioneered blockchain application and proved how powerful, cost-effective and timesaving they are. For this 1.3 million resident country, data integrity in medical records is critical. The Estonian E-Health foundation launched a project aiming to secure and store all patient health records and archive related activity logs. Estonia becomes the first country to implement blockchain for healthcare at a national scale. Estonia 💊 : 🏛: The UAE vice president announced that by 2021, 50% of the government’s transaction will be achieved through a blockchain, thus saving time and resources. The first step was taken by the Dubai Department of Finance who recently launched a blockchain-powered payment system intending to provide a more accurate and transparent governance process, as well as to enable real-time payments within and between government structures. United Arab Emirates After 2 years of testing, the Swedish land-ownership authority has conducted a successful test between individuals to buy and sell properties through a specific blockchain. They are now working on scaling this pilot to a national production. Sweden 🏡 : Smart Cities’ future looks bright As you can imagine, we are still in the very early-stages of blockchain applications development. As stated by , most organizations are in the discovery phase and experimentation with the aim of proving the viability of its use in their businesses. It goes the same for governments and smart cities. Gartner in their 2018 Market Guide for Blockchain While an extensive list of use cases can be thought of, a true implementation with a scaled, up-and-working application is quite hard to achieve. Thus before technological progress will help solve in the coming years the scalability and interoperability problems usually encountered to move from a Proof of Concept to a full live production, governments have been thinking to a lot of applications. Let’s review some of them. Voting system 📑 Citizen participation and more specifically the voting system is a perfect example of what blockchain can upgrade. The current system is too obsolete with numerous intermediaries and a high risk of corruption. By implementing a blockchain, thus a decentralized database, where each identifier would match a specific citizen, we could ensure that one person can vote securely, anonymously and uniquely. And we could think of it not only for big state decisions but for any development that can be brought to the city (for example a decision to build the local football stadium submitted only to the citizens of the concerned neighborhood). Moreover, we could encourage the citizen to participate in this civic life by incentivizing them thanks to a reward platform on which they could be offered specific services in return for their involvement. Traceability of medicines 🏳 One specific hurdle (among much others) in the health system can be enhance with blockchain: the supervision of medicines. Blockchain can achieve the control of an efficient drug supply chain that would reduce illegal use of certain medicines and monitor the pharmacies stocks in real-time. Education 🎖 Universities, colleges, high school and academies share a lot of information, sometimes inaccurate, on students. Processes are very centralized, increasing the risk that someone falsifies an information. A common ledger between all these stakeholders secured by a blockchain would ensure trustworthiness of the diploma and the educational cursus. As a sensitive information, it would only be shared to the authorized stakeholders thanks to a blockchain-based verification process. Food and agriculture 🍴 From the farmer’s field to the consumer’s mouth, food is travelling a long way. Consumers complain more and more about the lack of information on the provenance of the food they eat. The need for a more transparent producer-to-consumer system is amplifying. Blockchain could enable a food supply chain entirely verified by authorized entities that would update adequate information in the database. If a sanitary problem occurs, it would be easier to locate where it comes from. If there is a flaw in the process, supervisory authorities could distinguish quickly the reasons of this flaw. For instance Walmart, Nestlé, Unilever and seven other companies has partnered with IBM to develop The Food Trust blockchain to improve the companies’ ability to identify issues involved with food recalls, such as tracing outbreaks more quickly to limit customer risk. Energy ⚡️ The world’s energy grids get smarter and a new paradigm arises, where the passive user becomes an active producer and consumer. Distributed energy resources combined with smart microgrids provide an intelligent production and energy transfer within a community. Blockchain can play an active role in securing transactions between . Imagine an exchange platform where households and individuals could exchange their surplus of energy within a trading token-based system where the token serves as a mean of payment. The more electricity you produce and share, the more tokens you have. And as a sole consumer you get the advantage to pay only for what you consume. All of it being empowered by a secured, transparent and decentralized blockchain. That is exactly what . The system of this pilot allows 635 KW of energy to be traded through a platform between residents and across Bangkok’s grid. “prosumers” a neighborhood in Bangkok is currently achieving Employment ✒️ One good feature of the employment system nowadays is that it’s very decentralized. The problem is that there is a lack of confidence among job seekers and companies. The perfect match between them is often hard to carry out and there is a lot of business around head hunting and recruitment. Currently, when HR firms review resumes presented by prospective employees they have no way of knowing if its contents are valid. A secured and transparent registry containing all citizens’ past experiences and skills could facilitate the job search. Of course, information would be stored safely and provided to authorized employers. Blockchain technology is effective at establishing accurate records in a distributed system not controlled by any one entity. Insurance 🆘 How many papers are there to fill when you get sick? How many parties should you share your data with? The health system is quite a maze for patients, practicioners, insurances and pharmacies. Imagine that anybody could insure anything for any time. For instance, you could buy an insurance for any delay that your next flight may have. Blockchain-based smart contrats allow people to do just that. With an “IF/THEN” setting, smart contracts guarantee the outcome if the stated condition is met. If the flight is delayed, then you get a refund. is the realization of this example: «Fizzy is a fresh new genre in insurance. It offers direct, automatic compensation to policyholders whose flights are delayed. If your plane is more than two hours late, fizzy will reimburse you immediately.” Axa’s application Fizzy Conclusion Smart cities uses technology and infrastructure (systems, IT, physical, social and business) to provide higher quality of living for residents, good environment for businesses development, optimization of resource utilization and transparency for government. that acts as a tool for decentralized and distributed ecosystems. The security and transparency provided by blockchain empower all smart cities’ use cases relying on shared information, common database upgrading, information verification and decentralized feature. These goals can be outperformed with the use of blockchain We’re at the dawn of equipping our cities with smart devices and technologies that’ll improve services, amenities and life living. India, UAE and the US are probably the most advanced country on the topic but we’ll see further development across the globe as economic, political, social and environmental shifts are inevitable.

Make smarter cities with blockchain — Outlook and use-cases

Cities experience enormous challenges. As the world’s population continue to migrate to cities it raises questions about sustainability, liveability and the environmental impact. Enter smart cities, driven by technology, activated by the Internet of Things (IoT). At the most basic level, a smart city is a city that makes the most of the technology and the data available to it in order to address its challenges and to boost the wellbeing of its residents. Smart cities make better use of limited resources and operate more efficiently at every level. Today, smart city technology includes a co-ordinated mix of smart devices with IoT characteristics, riding on universal connectivity and enhanced by deep data analytics. Smart city technology is drawing huge investments: an suggests up to $80 billion will be invested towards smart cities in 2018 alone. IDC spending guide What makes smart cities possible? Yes, technology is what makes cities smarter. But more precisely, cities involve billions of data points and large data sets and it is, therefore, the ubiquitous nature of technology that enables smart cities. Technology is now so widely available that there is no restriction to the number of city facets that can be connected to a central data pool. This combination of omni-awareness and deep is what is making cities smarter than ever. Yet, there are individual technological components that complete the smart city puzzle: data analytics Smart cities thrive on data points and IoT-enabled sensors. Embedded devices can supply endless data points into the smart city data pool. The portability of sensors and devices has advanced so much that these can even be included in bicycles. In feeds data on pollution and traffic congestion back into the city’s systems. Sensors and embedded devices: Copenhagen, a hybrid e-bike The proliferation of sensors and embedded devices means that there is no region of a city and no aspect of a city that cannot be monitored to some degree. As a result, the information dark spots around cities are rapidly getting reduced: data can be collected anywhere, about anything. It is not only the proliferation of devices the enables smart cities. IoT devices will be of no use without connectivity and smart cities cannot be realised without pervasive wireless networks. The wide availability of 4G, easy-to-access Wi-Fi networks, and the upcoming 5G standard are all huge contributors. Telecoms and connectivity: 5G means cities can more easily handle the huge amounts of data and millions of devices that enable smart cities. Yinchuan, a city of 2 million in China, is arguably a pacesetter in terms of smart cities but getting there required and thousands of Wi-Fi access points. This level of connectivity will become more affordable to more cities as 5G rolls out. investing in an 8000GB fibre optic network The vast numbers of smart devices in cities deliver vast amounts of data, ideally suited to today’s big data capabilities. tunes into the technology essence of a smart city: processing and collecting billions of data points. Analytics, in turn, allows cities to use this data to make better decisions around city planning, transport, and sustainability. Big data and analytics: Big data Analytics platforms such as allows city halls to gain a bird’s eye view of different data sets including the use of parking, public safety concerns and waste management. In addition to the ability to control city parameters at a micro level, platforms such as these allow city managers to get a better grip on issues as they emerge. Cisco Kinetic for Cities How can city halls become smart? The presence of technology is not the defining characteristics of a smart city, but the optimal application of technology is essential to smart city operations. Yet city halls rarely function as centres of technological excellence. The solution lies in a partnership with a technology provider that understands how to get the best out of the latest technological advancements. Whether it is , or mining in-depth data analytics, your city’s technology partner can help you solve complex city-wide challenges by applying technological solutions. tapping IoT applications utilising wearables and mobile devices At ELEKS we know how to apply technology to drive innovation and growth in the public sector. to help you find solutions to the unique challenges that your city faces. Contact us Originally published at eleks.com on June 21, 2018.

Smart City Technology Enablers

The US immigration system is frustrating! After all the tax and tuition and a great job offer, you can still get kicked out of the country without a winning lottery in the H1-B process. Yes, we can have our fingers crossed for Donald Trump. But I will say, it is better if we hold our own future in hand. Here is a little bit about me… By now, I am sure you should be convinced that moving to another English-speaking immigration country is really a legitimate alternative if you do not get the H1-B (or if you up to an adventure). It can take a lot of work, but it will be worth it. In the end, remember to give it a clap if you find this article helpful. It will allow people to see it and use this resource. Best of luck! Sources US immigration visa category distribution yearly report https://travel.state.gov/content/travel/en/legal/visa-law0/visa-statistics/annual-reports.html International migrant population 1950–2015 https://www.migrationpolicy.org/programs/data-hub/international-migration-statistics United Kingdom immigration visa distribution, 2016 https://www.gov.uk/government/publications/immigration-statistics-july-to-september-2015/list-of-tables Canadian immigration visa distribution, 2016 https://www.canada.ca/en/immigration-refugees-citizenship/corporate/publications-manuals/annual-report-parliament-immigration-2016.html Australia immigration visa distribution https://www.homeaffairs.gov.au/ReportsandPublications/Documents/annual-reports/DIBP-Annual-Report-2014-15-optimised.pdf Liveability ranking and overview January 2008 https://store.eiu.com/article.aspx?productid=475217632 Average vacation days by country http://www.vpcalendar.net/average-vacation-days-by-country.html Average monthly disposable salary by country/Averge rent for inside city center one bedroom http://www.nationmaster.com/country-info/stats Cost of living index by country https://www.numbeo.com/cost-of-living/rankings_by_country.jsp United States work visa qualifications and waiting time http://www.immi-usa.com/immigration/employment-green-card/ https://travel.state.gov/content/travel/en/us-visas/immigrate/employment-based-immigrant-visas.html United Kingdom work visa qualifications and waiting time http://www.ukpermits.com/visa-types/tier-2 Canada work visa qualifications and waiting time https://www.canadianvisaexpert.com/visa-types/ https://www.canadavisa.com/canadian-immigration-application-processing-times.html https://www.canadavisa.com/canada-immigration-processing-times.html Australia work visa qualifications and waiting time http://visacorp.com.au/australian-skilled-visas.html http://www.thamesmigration.com/australia-visa-application-processing-times-2017/

If not a US H1-B, how about one in Canada, Australia or the UK?

“Anything that could give rise to smarter-than-human intelligence — in the form of artificial intelligence, brain-computer interfaces, or neuroscience-based human intelligence enhancement — wins hands down beyond contest as doing the most to change the world. Nothing else is even in the same league.” — Eliezer Yudkowsky, artificial intelligence theorist. Once relegated to science fiction, artificial intelligence is now becoming a part of our daily lives, with the potential to immensely improve virtually every aspect of our cities and communities. As the global population increases, efficient transportation becomes much harder to achieve. However, AI holds the promise to revolutionize this essential aspect of our lives. Many experts agree that self-driving vehicles will be widely adopted by 2020, and this shift will go far beyond cars to include driverless delivery trucks and autonomous delivery drones. Additionally, it’s expected that a significant percentage of personal car ownership will be replaced by “cars as a service” models, with the likelihood of either removing the need for public transportation or at least having it transition to an on-demand service. These changes will reduce congestion and increase efficiency, thereby allowing commuting time to be used for relaxing or working instead of being a source of stress and a waste of valuable hours. The effect on cities will be enormous, improving liveability while massively reducing the need for parking. AI’s potential to improve healthcare is enormous. For example, artificial intelligence could be used to automate the majority of diagnostics work via patient data and scientific research. Our overburdened doctors and nurses could be relieved from many of their current duties, thereby allowing them to focus on the human dimensions of care and the more advanced analytical requirements. Additionally, elderly patients will benefit from intelligent walkers and wheelchairs combined with smart home monitoring technologies to provide them with increased independence as they age. Home and service robots are expected to become much more common in the coming years. Perhaps one day in the not-too-distant future you will have a robot cleaning your house while another delivers a package for you. Advancements in cloud-connected robots will increase the ability to share data to accelerate learning, while perceptual technology growth will improve robots’ interactions with humans. We are on the cusp of a new world where AI combines with human intelligence to improve our cities and lives in a multitude of ways, some of which are hard to even imagine at this point, while other utilizations are already making the world a better place to live for us all. Written by Igor Ilunin, head of IoT at DataArt .

Using AI to Improve Our Cities and Lives

A messaging guide for talking about our autonomous car future Ask your mother-in-law over coffee. Bring it up over wine with friends. Talk to a bunch of kids when it’s your turn to carpool. Call your dad. “When do you think we’ll have self-driving cars? Will *you* ride in them?!” For all the rapid progress we’re making toward a self-driving future, the answers you get are probably, “A long time from now,” and somewhere between “maybe ¯\_(ツ)_/¯?” and “when they pry the steering wheel from my hands.” Thing is, they coming—fast. But we’re lousy at explaining what that will mean for people’s lives. We’re too technical, too in awe of the technology, too pie-in-the-sky, too quick to assume that culture will shift without a fight. are And that’s a problem, because we have big decisions to make right now—about insurance, regulation, taxes, labor, our —that will determine whether we get it right or muck it all up. cities So, I’ve assembled a messaging guide to explain self-driving cars, why they matter, and how we can use this big transformative moment to improve the places we live and make people’s lives better. It builds on the amazing work of and many others, and it’s just a first draft. Robin Chase So with that, start your engines! Err.. unlock your phone? Onward! Top-line Messages Self-driving electric vehicles will provide affordable transportation to everyone. Available anywhere, anytime with a tap on a smartphone, a fleet of shared self-driving cars will offer convenient door-to-door transportation that’s as fast as driving yourself for the price of taking the bus — all in the safest cars ever to hit the road. Autonomous vehicles have arrived. They’ve proven themselves in millions of miles of testing, they’re in commercial use in a handful of cities, and they’ll be available for purchase within two years. They have the potential to save tens of thousands of lives in the U.S. every year. But not all AVs are the same. Simply replacing our personal cars with AVs will do nothing to address climate change, reduce congestion, or make transportation more affordable. We should focus on FAVES, or Fleets of Autonomous Vehicles that are Electric and Shared. With always available FAVES (coined by Chase), we will only need 10% of the vehicles on the road today. And the benefits to this market-driven solution are huge: fast, convenient, low-cost transportation for everyone. Now is our chance… To make getting around fast, easy and affordable for all; to eliminate CO2, air pollution and fossil fuels from our transportation system; and to transform our communities into sustainable, affordable and healthy places to live. 🚙 🚘 🚌 🚙 🚘 🚌 🚙 🚘 🚌 🚙 Supporting Messages & Data 1. Fast, Convenient and Affordable FAVES (Fleets of Autonomous Vehicles that are Electric and Shared) will offer convenient door-to-door transportation that’s as fast as driving yourself for the price of taking the bus. Easy as a Lyft, as cheap as a bus, as seamless as walking. Personal car ownership will become an increasingly unnecessary burden, shifting our focus from owning a vehicle to taking efficient trips. The average family spends $9,000 per year to keep a car in the garage. Using shared electric autonomous vehicles will be cheaper by 2/3rds. People love low-cost shared rides: 30% of all US Lyft rides and 50% of Uber/Lyft rides in San Francisco are shared. Ever improving algorithms and a robust fleet of cars will mean sharing gets more convenient and even cheaper. 2. Revolution in Road Safety AVs will make our streets and highways dramatically safer — for riders commuting home, pedestrians, bikers, kids playing and families enjoying their neighborhoods. No longer will we have to accept the real risk of injury or death just to live our lives and get where we’re going. Each year in the US, we lose 38,000 lives in car accidents — 90% due to poor decisions made by people behind the wheel. An additional 4.4 million are injured, their lives changed forever in an instant. There is not a family in America that has not been touched in some way by tragedy on the road. And the situation is getting worse: In 2015, we saw the highest one-year percentage increase in traffic deaths in half a century. AVs have the potential to save more lives than airbags, seatbelts, stability control and crumple zones — combined. Never before has a technology offered this potential to completely transform road safety. AVs have superhuman senses, seeing hundreds of yards around the vehicle in all directions, at all times. They use radar to see a slowdown three cars in front of you, infrared to detect a pedestrian on a dark night, and lasers to map and incredibly detailed view around you. And, they learn from one another in real time, which means that when one car sees a new pothole or construction detour, the whole fleet knows — immediately. In cities, streets will be safer for pedestrians, air will be cleaner (saving lives and reducing disease), and day-to-day liveability will rise. Outside of cities, where single car crashes, high speeds and traffic fatalities are common, the shift to AVs will be a lifesaver. For communities that have seen too many young lives end behind the wheel because of a moment of bad judgment, the change will be dramatically positive. 3. Mobility and Access to Opportunity for All Shared autonomous rides will be available to all people, in all neighborhoods, any day, anytime. That means easier access to jobs, affordable transportation options in places where none existed before, and freedom of mobility for people who are left out of our current system. More transportation freedom. We think of the automobile as the ultimate personal freedom machine. But what’s more free than one-button on-demand transportation to anywhere at anytime, paying only for the distance traveled, without worry of maintenance, parking, insurance, or gas? For too many workers, affordable transportation is a major barrier to quality jobs and an escape from poverty. Parents race to catch a once-an-hour bus so they can pick up their kids on time, service workers commute for hours, and car trouble can mean lost wages and lost jobs. Reliable, inexpensive transportation will transform access to opportunity, which means more employment, more stability, and higher quality of life. Young folks, older adults, and disabled people will now have dramatically expanded mobility — on their terms. With the baby boomer generation moving into active retirement, this new freedom will profoundly change their experience and shape their decisions about where and how to live. Transportation deserts, where personal car ownership is the only way to get around, will be transformed by FAVES — from disadvantaged urban neighborhoods where bus service is poor and taxis won’t go, to suburban and rural regions where a car is required to live a productive life. 4. Reducing CO2 and Moving Toward Renewable Energy As our climate continues to warm, FAVES offer a lifeline to dramatically reduce the CO2 and harmful emissions from our day-to-day transportation. The fleet will use electric vehicles, and any increased energy demand will come from renewable sources. Our planet continues to set the wrong kind of records, with the unprecedented warmth of 2015 eclipsed by an even warmer 2016. Arctic sea ice during its summer melting now covers 40% less area than it did in the early 1980s. The decades long trends show us that a changing climate is our new normal. With FAVES, we will need 10% of the vehicles we have now — and they’ll be electric. Together, the sheer reduction in the number of vehicles plus the move toward electric cars powered by renewable sources will put us on the path to a zero emissions future. 5. Streets for People, Not Cars We have a once-in-a-generation chance to boldly rethink our streetscapes and our communities. Unencumbered by the fossil fuel-driven personal automobile, and needing only 10% of the vehicles we have now, we can remake all of our streets into beautiful creative spaces focused on people, not cars. It’s time to rethink our public space priorities. In the words of NYC transportation visionary Janette Sadik-Khan: “You can store glass & steel or win hearts & minds.” Our city streets are like the “Blackberry fixed keyboards” of infrastructure. Bespoke with extra-wide lanes, parking lining both sides, high car throughput, an occasional bike lane, and zero flexibility. They were engineered for texting, eating, distracted humans and, critically, for moving cars and people into, through, and out of places as quickly as possible. Autonomous vehicles don’t need wide lanes, they don’t need to park in urban areas, they don’t require much signage or signals, they won’t hit bikers or pedestrians, they don’t get impatient, and they don’t text and drive. It’s time to throw out the “keyboard” and start thinking of our streets as a blank iPhone screen. Many cities are already taking the lead, from Vancouver and Portland to New York and Denver. Local communities are taking back public space, bringing walkability, friendly streetscapes and new life into urban spaces that had previously been dedicated to fast throughput and storing cars. We need to be asking ourselves new questions about our urban, suburban and rural spaces: How should people feel when they walk through the space? What do local businesses need from the street? How should the space change throughout the day, or day of the week? Should AVs access the street? At all times? Some of the time? How does public transit support the community’s transportation needs? How should trees, sidewalks, bike paths, park-lets, seating, wifi, art, and gardens fit in? What changes can we make to rural/less-urban communities so they share in these benefits? We have two choices that will determine whether the world we leave for our children is livable and sustainable: Let the autonomous vehicle levee break and spill and transform our cities to its will. Or start planning now for the flood, using this transformative moment to fix our cities’ problems and design the places we want to live in for the next generation. _The Great Urban Do-Over Is Coming. Let’s Not Eff It Up._medium.com What if every street looked like the High Line? 6. Fix Our Broken Tax Structure & Our Infrastructure Transportation tax revenues are endangered and headed for a fiscal cliff. Every source, from gas taxes and tolls to vehicle registration and tickets, will dry up as the inevitable move to shared electric autonomous vehicles occurs. We need to do the work *now* to focus on miles travelled, congestion, and vehicle weight to secure funding to maintain and improve our transportation infrastructure in the future. We can’t improve our complex streets that were built for human piloted cars incrementally. Transportation planners, state and local governments, and federal DOT must consider shared AVs as a coming reality in all transportation projects. 7. Time to Seize the Moment AVs are happening now. You can summon an autonomous Uber in multiple cities, your Tesla has continuously learning autopilot, every major automaker is acquiring companies and building its autonomous team, updated regulations are opening roads to AVs, and retail sales are expected by 2019. Since we know they’re inevitable, the time to be proactive about the rollout of FAVES is now. How do we do that? We can develop smart regulations of AVs to avoid crippling congestion from empty cars circling the block, update how we tax transportation before our gas tax revenues fall off a cliff, plan to integrate AVs into our cities to make them safer and more liveable, and insist on electric AVs powered by renewables to wean ourselves from CO2-producing fuels. We are in the middle of a big shift, on the same scale as the industrial revolution, or the post-WWII boom of cars, highways and suburbs. AV technology has arrived and our climate needs urgent attention. If we take a “wait and see” approach, we’re doomed; if we’re proactive, we can use this moment as an opportunity to tackle our world’s biggest challenges. “By 2025, private car ownership will all-but end in major U.S. cities” — John Zimmer, President of Lyft. _Ten years ago I found myself standing outside the Automotive Hall of Fame in Dearborn, Michigan. I looked out over…_www.wired.com Self-Driving Cars Will Improve Our Cities. If They Don't Ruin Them. 🚘 🚘 🚘 🚘 This document is directly based on the pioneering ideas, writing, and research by on FAVES and our autonomous future. My goal was to start with her smart thinking on this topic (see ), combine it with ideas and inspiration from all the good folks working to make autonomous vehicles a positive force for global change, and create a useful draft working document for the field. Ed Note: Robin Chase Osmosys Next up: Part 2 will cover the tough questions we still need to answer about labor, policy, data & security, culture shifts and more. Please contact me with any ideas, edits, or additions. Thx for reading, recommending, and sharing with folks who might find it useful! Cheers,Alex Fieldtwitter: @alexfield

Stories, News & Mentions about LIVEAPI

Liveapi’s Stories on HackerNoon